3.11 Using Common Regression Metrics

3.11 Using Common Regression Metrics

Regression Evaluation Metric Framework

Regression models estimate continuous targets $y \in \mathbb{R}$. Evaluation metrics quantify the distribution and magnitude of residuals $e_i = y_i - \hat{y}_i$:

+-----------------------------------------------------------------------------+
|                        REGRESSION METRIC COMPARISON                         |
|                                                                             |
|   METRIC   FORMULA                               PROPERTIES / USAGE         |
|   ------   -----------------------------------   -------------------------- |
|   MSE      (1/n) * sum( (y_i - y_hat_i)^2 )      Penalizes large outlier    |
|                                                  errors quadratically; units|
|                                                  are squared.               |
|                                                                             |
|   RMSE     sqrt( MSE )                           Same units as target;      |
|                                                  sensitive to large outliers|
|                                                  Standard ML benchmark.     |
|                                                                             |
|   MAE      (1/n) * sum( |y_i - y_hat_i| )        Same units as target;      |
|                                                  linear penalty; highly     |
|                                                  robust to extreme outliers.|
|                                                                             |
|   R^2      1 - [ SS_res / SS_tot ]               Fraction of variance       |
|            1 - [ sum((y-y_hat)^2)/sum((y-y_bar)^2)] explained by model;     |
|                                                  <= 1.0 (1.0 = perfect,     |
|                                                  0.0 = baseline mean, <0 bad|
|                                                                             |
|   MAPE     (100%/n) * sum( |(y-y_hat)/y| )       Scale-independent relative |
|                                                  percentage error.          |
+-----------------------------------------------------------------------------+

---

What Each Metric Actually Penalises

RMSE versus MAE, concretely

Take two models on ten predictions. Model A misses by 10 units on every prediction. Model B is perfect on nine and misses by 100 on the tenth.

• MAE: A = 10.0, B = 10.0. The two look identical.
• RMSE: A = 10.0, B = $\sqrt{10{,}000/10} = 31.6$. B is punished three times harder.

Because $\text{RMSE} \ge \text{MAE}$ always, the gap between them is itself a diagnostic: a large gap means the error distribution has a heavy tail. Which model is better is a business decision — if one catastrophic miss is far worse than ten small ones, RMSE captures it; if cost accrues linearly with error, MAE is the honest score.

$R^2$ pitfalls

$R^2 = 1 - \frac{\sum (y_i - \hat{y}_i)^2}{\sum (y_i - \bar{y})^2}$

• $R^2 = 0$ means the model does no better than predicting the training mean.
• $R^2 < 0$ is possible on a test set and means the model is worse than that baseline.
• $R^2$ rises mechanically as features are added, which is why adjusted $R^2$ exists for model comparison at differing feature counts.
• Because the denominator is the variance of the actuals, $R^2$ is not comparable across datasets with different target spreads even though it is unitless.

MAPE's failure mode

$\text{MAPE} = \frac{100}{n}\sum \left| \frac{y_i - \hat{y}_i}{y_i} \right|$

Every term divides by the actual value, so a single row with $y_i$ near zero can send MAPE to thousands of percent, and $y_i = 0$ makes it undefined. For demand data with frequent zeros, prefer SMAPE or a weighted absolute-percentage error — which is why Databricks AutoML uses SMAPE as its forecasting default.

Computing Them

from pyspark.ml.evaluation import RegressionEvaluator

for metric in ["rmse", "mae", "r2", "mse", "mape"]:
    value = RegressionEvaluator(labelCol="price", predictionCol="prediction",
                                metricName=metric).evaluate(preds)
    print(f"{metric}: {value:.4f}")

RegressionEvaluator defaults to rmse when metricName is omitted — worth remembering when a code snippet leaves it out.

In scikit-learn the equivalents live in sklearn.metrics:

from sklearn.metrics import (root_mean_squared_error, mean_squared_error,
                             mean_absolute_error, r2_score,
                             mean_absolute_percentage_error)

rmse = root_mean_squared_error(y_test, y_pred)   # current API
mse  = mean_squared_error(y_test, y_pred)        # returns MSE, not RMSE
mae  = mean_absolute_error(y_test, y_pred)
r2   = r2_score(y_test, y_pred)
mape = mean_absolute_percentage_error(y_test, y_pred)   # a fraction, not a percent

Two API details matter. mean_squared_error returns MSE, not RMSE — older code reached for squared=False, but that argument was deprecated in favour of the dedicated root_mean_squared_error function, so take the square root explicitly or use the newer function. And mean_absolute_percentage_error returns a fraction (0.12), not a percentage (12%), so reports must scale it.

Reporting Regression Results Honestly

• Always state the units. "RMSE 42,000" is meaningless without "dollars", and it is meaningless in a different way if the model was trained on a log target (Section 3.13).
• Quote a baseline. The mean predictor's RMSE — the standard deviation of the target — is the number a model must beat, and it is what $R^2 = 0$ corresponds to.
• Report more than one metric. RMSE and MAE together reveal the shape of the error distribution, and $R^2$ adds a scale-free view. A single metric hides the tail.
• Evaluate on the untouched holdout. Cross-validated scores are used for model selection; the final number reported to stakeholders comes from the test split scored exactly once.