3.10 Using Common Classification Metrics

3.10 Using Common Classification Metrics

Model evaluation is the objective mathematical discipline of assessing how effectively a trained algorithm generalizes to unseen data, meets business operational requirements, and balances structural errors. Selecting inappropriate metrics or misinterpreting learning curve diagnostics can lead to deploying poorly performing models into mission-critical production systems. This section details the complete catalog of classification and regression evaluation metrics, the nuances of evaluating models with transformed targets, and systematic protocols for diagnosing and remediating bias and variance.

---

Classification Evaluation Metric Framework

Binary and multiclass classification models output discrete class assignments or continuous probability vectors that must be evaluated against ground-truth labels using a confusion matrix:

+-----------------------------------------------------------------------------+
|                        CONFUSION MATRIX TAXONOMY                            |
|                                                                             |
|                              ACTUAL POSITIVE (1)      ACTUAL NEGATIVE (0)   |
|   PREDICTED POSITIVE (1)     True Positive (TP)       False Positive (FP)   |
|                              Correct Detection        Type I Error (Alarm)  |
|                                                                             |
|   PREDICTED NEGATIVE (0)     False Negative (FN)      True Negative (TN)    |
|                              Type II Error (Missed)   Correct Rejection     |
+-----------------------------------------------------------------------------+

Mathematical Metric Formulations

1. Accuracy: Measures overall proportion of correct classifications: $\text{Accuracy} = \frac{\text{TP} + \text{TN}}{\text{TP} + \text{TN} + \text{FP} + \text{FN}}$

   • Limitation: Highly misleading under class imbalance.

2. Precision (Positive Predictive Value): Measures the accuracy of positive predictions: $\text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}$

   • Optimization Goal: Maximize when False Positives are costly (e.g., spam filtering, VIP transaction freezing).

3. Recall / Sensitivity (True Positive Rate): Measures the proportion of actual positives successfully identified: $\text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}}$

   • Optimization Goal: Maximize when False Negatives are costly (e.g., medical pathology diagnosis, critical equipment failure).

4. Specificity (True Negative Rate): Measures the proportion of actual negatives correctly rejected: $\text{Specificity} = \frac{\text{TN}}{\text{TN} + \text{FP}}$

5. F1-Score: Harmonic mean of precision and recall, balancing both dimensions: $F_1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}} = \frac{2\text{TP}}{2\text{TP} + \text{FP} + \text{FN}}$

6. $F_\beta$-Score: Weighted harmonic mean that assigns $\beta$ times as much importance to recall as to precision: $F_\beta = (1 + \beta^2) \cdot \frac{\text{Precision} \cdot \text{Recall}}{\beta^2 \text{Precision} + \text{Recall}}$

   • $F_2$ Score ($\beta=2$): Weights recall twice as heavily as precision (ideal for fraud and medical detection).
   • $F_{0.5}$ Score ($\beta=0.5$): Weights precision twice as heavily as recall (ideal for automated trading or spam filters).

7. Log Loss / Cross-Entropy: Evaluates calibrated probability outputs rather than hard thresholded labels: $\text{LogLoss} = -\frac{1}{N} \sum_{i=1}^N \left[ y_i \ln(\hat{p}_i) + (1-y_i) \ln(1-\hat{p}_i) \right]$

   • Property: Heavily penalizes confident incorrect predictions (e.g., predicting $\hat{p}=0.99$ when actual $y=0$).

---

Threshold Metrics vs. Ranking Metrics

Classification metrics fall into two families, and mixing them up is a frequent error.

Two practical consequences:

• A model can have excellent ROC-AUC and terrible F1 simply because the default 0.5 threshold is wrong for its calibration. The remedy is threshold tuning (Section 3.2), not retraining.
• Reporting accuracy on an imbalanced problem is meaningless: predicting the majority class for everything already achieves the prevalence rate.

Computing Them on Databricks

from pyspark.ml.evaluation import BinaryClassificationEvaluator, MulticlassClassificationEvaluator

roc = BinaryClassificationEvaluator(labelCol="label", rawPredictionCol="rawPrediction",
                                    metricName="areaUnderROC").evaluate(preds)
pr  = BinaryClassificationEvaluator(labelCol="label", rawPredictionCol="rawPrediction",
                                    metricName="areaUnderPR").evaluate(preds)

f1  = MulticlassClassificationEvaluator(labelCol="label", predictionCol="prediction",
                                        metricName="f1").evaluate(preds)

Note which column each evaluator consumes. BinaryClassificationEvaluator needs rawPredictionCol (scores or probabilities) because it sweeps thresholds; MulticlassClassificationEvaluator needs predictionCol (hard labels) because its metrics are threshold metrics. Passing the wrong column is a classic snippet bug.

Multiclass averaging

For more than two classes, precision and recall must be averaged across classes:

• Macro — unweighted mean over classes; every class counts equally, so rare classes matter as much as common ones.
• Weighted — mean weighted by class support; dominated by the common classes. This is what Spark's weightedPrecision and weightedRecall compute.
• Micro — pooled counts across classes; for single-label multiclass this equals accuracy.

---

Worked Implementation: Classification Metrics in scikit-learn

import numpy as np
from sklearn.metrics import (
    confusion_matrix,
    precision_score,
    recall_score,
    f1_score,
    fbeta_score,
    roc_auc_score,
    average_precision_score
)

y_true = np.array([0, 1, 0, 0, 1, 1, 0, 1, 0, 1])
y_pred = np.array([0, 1, 0, 0, 0, 1, 0, 1, 1, 1])
y_prob = np.array([0.1, 0.9, 0.2, 0.3, 0.45, 0.85, 0.15, 0.78, 0.65, 0.92])

cm = confusion_matrix(y_true, y_pred)
prec = precision_score(y_true, y_pred)
rec = recall_score(y_true, y_pred)
f1 = f1_score(y_true, y_pred)
f2 = fbeta_score(y_true, y_pred, beta=2.0)  # Recall weighted 2x over precision
roc_auc = roc_auc_score(y_true, y_prob)
pr_auc = average_precision_score(y_true, y_prob)

print("Confusion Matrix:\n", cm)
print(f"Precision: {prec:.3f}, Recall: {rec:.3f}, F1: {f1:.3f}, F2: {f2:.3f}")
print(f"ROC-AUC: {roc_auc:.3f}, PR-AUC: {pr_auc:.3f}")