7.1 Evaluation Metrics for Predictive Models
Key Takeaways
- Precision is TP / (TP + FP), recall is TP / (TP + FN), and F1 is their harmonic mean, which balances both on imbalanced data.
- AUC PR is more informative than AUC ROC when the positive class is rare, because ROC curves can look strong when negatives dominate.
- RMSE penalizes large errors more heavily than MAE, and MAPE is undefined when any actual value is zero.
- Agent Platform model evaluation compares batch inference results against ground-truth data and can run from Model Registry or as a pipeline component.
- Raising a classifier's confidence threshold generally increases precision and lowers recall.
Section 2.3 of the exam guide includes evaluating predictive and gen AI solutions (for example, model evaluation metrics and LLM-as-a-judge). This section covers predictive models. Section 7.2 covers gen AI. On the exam, the right metric is the one that matches the business cost of errors and the data distribution, not the one with the most familiar name.
Classification: The Confusion Matrix First
| Predicted positive | Predicted negative | |
|---|---|---|
| Actual positive | True positive (TP) | False negative (FN) |
| Actual negative | False positive (FP) | True negative (TN) |
| Metric | Formula | Answers |
|---|---|---|
| Accuracy | (TP + TN) / total | How often is the model right overall? Misleading with imbalanced classes |
| Precision | TP / (TP + FP) | When the model flags something, how often is it correct? |
| Recall (true positive rate) | TP / (TP + FN) | Of all real positives, how many did the model catch? |
| F1 | 2 × P × R / (P + R) | A balance of precision and recall |
| False positive rate | FP / (FP + TN) | How often are negatives wrongly flagged? |
Worked example: why accuracy misleads
A fraud model scores 10,000 transactions, and 100 are truly fraudulent. It flags 150, of which 80 are real fraud.
- TP = 80, FP = 70, FN = 20, TN = 9,830
- Accuracy = (80 + 9,830) / 10,000 = 99.1%
- Precision = 80 / 150 = 53.3%
- Recall = 80 / 100 = 80.0%
- F1 = 2 × 0.533 × 0.800 / (0.533 + 0.800) = 0.64
A model that flags nothing reaches 99.0% accuracy and catches zero fraud. With rare positives, look at precision, recall, F1, and AUC PR, not accuracy.
Threshold-Independent Metrics
| Metric | What it summarizes | Use when |
|---|---|---|
| AUC ROC | True positive rate vs. false positive rate across all thresholds | Classes are reasonably balanced, or you care about ranking quality overall |
| AUC PR (average precision) | Precision vs. recall across thresholds | The positive class is rare (fraud, defects, churn). Less inflated by many true negatives |
| Log loss | Penalty for confident wrong probabilities | Probabilities themselves are used, as in pricing and risk scores |
Choosing the threshold
Most classifiers output a score. The confidence threshold turns it into a decision. Raising it usually raises precision and lowers recall. Pick it from business costs:
- Missed fraud costs $500 and a false alarm costs $5 in review time: favor recall and accept a lower threshold.
- An email marketing model where every false positive annoys a customer: favor precision.
- Regulated screening with a required minimum recall (for example, 95%): choose the highest precision that still meets the recall floor. That is exactly what AutoML's "precision at recall" objective optimizes.
Multi-class problems report per-class precision and recall plus micro averages (pooled counts, dominated by large classes) and macro averages (average of per-class scores, which treats rare classes equally). Agent Platform also reports "at 1" metrics, which consider only the top-scoring label.
Regression Metrics
| Metric | Behavior | Choose when |
|---|---|---|
| MAE | Average absolute error, in the target's units | Errors should count linearly. Robust to outliers |
| RMSE | Square root of mean squared error | Large errors are especially costly |
| RMSLE | RMSE on log(1 + value), penalizing under-prediction more | Relative error matters and targets span orders of magnitude. Needs non-negative values |
| R² | Share of variance explained (0-1 in Agent Platform's definition) | Comparing fit quality across models |
| MAPE | Average absolute percentage error | Stakeholders think in percentages. Undefined when any actual value is 0 |
Worked example: errors of +2, -2, and +10 give MAE = (2 + 2 + 10) / 3 = 4.67 and RMSE = √((4 + 4 + 100) / 3) = √36 = 6.0. The one large error pulls RMSE well above MAE.
Forecasting, Ranking, and Clustering
- Forecasting adds WAPE (total absolute error divided by total actuals, stable when many values are small or intermittent), RMSPE, and quantile metrics for probabilistic forecasts.
- Recommendation and ranking: BigQuery ML matrix factorization with implicit feedback reports recall, mean squared error, normalized discounted cumulative gain (NDCG), and average rank. NDCG rewards putting relevant items near the top of a list.
- Clustering: BigQuery ML k-means reports the Davies-Bouldin index (lower is better) and mean squared distance to centroids. Clusters still need a business check to confirm they're meaningful segments.
Beyond Aggregate Metrics
- Slice metrics: evaluate by region, device, language, or demographic group. An overall AUC of 0.90 can hide a segment at 0.70 (fairness, Chapter 18).
- Baseline comparison: always compare with a simple model and the current production model on the same test data.
- Calibration: check that predicted probabilities match observed rates when scores drive pricing or risk.
- Offline vs. online: offline metrics decide whether a model is ready to test. Online A/B tests measure the business KPI (Chapter 13).
Common Metric Traps on the Exam
| Trap | Better choice |
|---|---|
| Reporting accuracy for a 1%-positive fraud model | Precision, recall, F1, AUC PR |
| Using AUC ROC to compare models on a very rare class | AUC PR |
| Optimizing MAE when a few huge misses cause outages | RMSE |
| Reporting MAPE on intermittent demand with zero days | WAPE or MAE |
| Tuning the threshold on the test set, then reporting test metrics | Choose the threshold on validation data, then report on untouched test data |
| Comparing two models evaluated on different test windows | Same test data for every candidate |
Model Evaluation on Agent Platform
Agent Platform's predictive model evaluation takes a trained model, batch inference output, and ground truth, and computes metrics such as precision, recall, AuPRC, AuROC, log loss, confusion matrices, MAE, RMSE, and MAPE. You can:
- Create evaluations from Model Registry in the console and compare them across models or versions.
- Add the model evaluation component to an Agent Platform Pipeline, usually after a batch inference component, so every retraining run is evaluated the same way (Chapter 15).
- Run evaluations periodically on fresh labeled production data. This is continuous evaluation, which signals when to retrain.
AutoML also produces its own evaluation metrics during training. That is separate from the model evaluation service described here.
A defect-detection model runs on 5,000 parts, and 50 are truly defective. It flags 60 parts, and 45 of them are real defects. What are its precision and recall?
A medical screening model must catch at least 95% of true cases, and among models that meet that bar the team wants the fewest false alarms. Which evaluation approach fits best?
A retailer forecasts daily demand for thousands of SKUs, and many SKUs have days with zero sales. Which metric should the team avoid as its primary accuracy measure?