Build a Model Evaluation Script with Scikit-learn

1. 1
   Prepare Your Environment and Data

   Import necessary libraries and create a sample dataset. A standard practice is to split your data into training and testing sets to evaluate the model on unseen data, preventing overfitting.
2. 2
   Train a Simple Classifier

   Train a basic model, like Logistic Regression, on your training data. This model will serve as the subject for our Scikit-learn Model Evaluation Script.
3. 3
   Generate a Classification Report

   Use the trained model to make predictions on the test set. The `classification_report` function provides a comprehensive text summary of precision, recall, and F1-score for each class.
4. 4
   Visualize a Confusion Matrix

   Create a confusion matrix to visually analyze model errors. The matrix shows the counts of true positives, true negatives, false positives, and false negatives, which is key for error analysis.
5. 5
   Plot the ROC Curve and Calculate AUC

   Generate a Receiver Operating Characteristic (ROC) curve and calculate the Area Under the Curve (AUC). This visualization helps assess the model's ability to distinguish between classes across different thresholds.

import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, ConfusionMatrixDisplay, RocCurveDisplay

# This script requires scikit-learn and matplotlib
# You can install them with: pip install scikit-learn matplotlib

def run_model_evaluation():
    """A complete script to train a model and evaluate its performance."""
    # 1. Generate and prepare data
    X, y = make_classification(n_samples=1000, n_features=10, n_informative=5, n_redundant=0, random_state=42)
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
    print(f"Data prepared: {len(X_train)} training samples, {len(X_test)} test samples.")

    # 2. Train a model
    model = LogisticRegression(random_state=42)
    model.fit(X_train, y_train)
    print("Model trained successfully.")

    # 3. Make predictions
    y_pred = model.predict(X_test)

    # 4. Generate text-based report
    print("\n--- Classification Report ---")
    print(classification_report(y_test, y_pred))

    # 5. Generate and display visualizations
    print("\n--- Generating Visualizations ---")
    # Confusion Matrix
    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5))
    fig.suptitle('Model Performance Visualizations')
    ConfusionMatrixDisplay.from_estimator(model, X_test, y_test, ax=ax1)
    ax1.set_title('Confusion Matrix')

    # ROC Curve
    RocCurveDisplay.from_estimator(model, X_test, y_test, ax=ax2)
    ax2.set_title('ROC Curve')

    plt.tight_layout()
    plt.show()

if __name__ == '__main__':
    run_model_evaluation()