3.12 Choosing the Most Appropriate Metric for the Objective
Key Takeaways
- Start from the asymmetry of the business costs: whichever error type is more expensive determines whether precision, recall, or a balance is optimised.
- When false negatives dominate the cost — disease screening, fraud, equipment failure — maximise recall or $F_2$.
- When false positives dominate — spam filtering, automated account closure, costly interventions — maximise precision or $F_{0.5}$.
- For rare events, prefer PR-AUC over ROC-AUC and never report raw accuracy; for well-calibrated probabilities used in expected-value decisions, use log loss.
- For regression, pick RMSE when large errors are disproportionately costly, MAE when cost is proportional to error, and $R^2$ only for communicating explanatory power.
3.12 Choosing the Most Appropriate Metric for the Objective
Metric selection is not a statistical preference — it is a translation of the business cost structure into a single number the tuning loop can optimise. The exam poses this as a scenario, and the scenario always contains the decisive clue.
The Decision Procedure
- Classification or regression? Determined by the target.
- What does each error type cost? Write down the cost of a false positive and of a false negative. Whichever is larger is what the metric must punish.
- How rare is the positive class? Below roughly 5–10% prevalence, accuracy and ROC-AUC both flatter the model; use PR-AUC and $F_\beta$.
- Is a hard decision made, or is the score consumed downstream? A hard decision calls for a threshold metric; a score feeding an expected-value calculation calls for a calibration metric such as log loss.
- Is the metric being reported to stakeholders? Prefer one expressed in units they already reason about — dollars, minutes, percentage points.
Scenario to Metric
| Scenario | Dominant cost | Metric |
|---|---|---|
| Cancer screening: a missed tumour is fatal, a false alarm means one more blood test | False negatives | Recall, or $F_2$ |
| Spam filter: a lost legitimate email is far worse than a spam message getting through | False positives | Precision, or $F_{0.5}$ |
| Fraud detection at 0.2% prevalence, blocking a card costs customer friction | Both, asymmetric | PR-AUC to compare models, $F_\beta$ at the chosen threshold |
| Credit limit decisions where the score feeds an expected-loss calculation | Probability quality | Log loss |
| Ranking a recommendation slate — only the ordering is used | Ordering quality | ROC-AUC or ranking metrics |
| Balanced classes, symmetric costs | Neither | Accuracy or F1 |
| Predicting house prices where a large miss is disproportionately costly | Large errors | RMSE |
| Predicting delivery times where cost accrues per minute late | Proportional | MAE |
| Explaining to executives how much of price variation the model captures | Communication | $R^2$ |
| Demand forecasting across products at very different scales | Relative error | SMAPE (MAPE breaks near zero) |
Traps
- Accuracy under imbalance. At 99.8% negatives, "predict negative always" scores 99.8%. Any answer choosing accuracy on a rare-event problem is wrong.
- ROC-AUC under extreme imbalance. The false-positive-rate denominator is $FP + TN$; with an enormous $TN$, hundreds of false positives barely move it. PR-AUC ignores true negatives entirely and exposes the problem.
- Optimising a metric you do not act on. If the deployed system applies a fixed threshold, tuning to ROC-AUC optimises a property the system never uses.
- Metrics on a transformed target. RMSE on $\ln(y+1)$ is not RMSE in dollars — see Section 3.13.
- MAPE with near-zero actuals. Mathematically undefined at zero and explosive near it.
- Comparing $R^2$ across datasets. The denominator is dataset-specific, so a higher $R^2$ on a different dataset does not mean a better model.
Connecting the Metric to the Tuning Loop
Whatever you choose has to become the objective the search minimises:
# Hyperopt minimises, so a metric to maximise is negated
def objective(params):
model = fit(params)
f2 = fbeta_score(y_val, model.predict(X_val), beta=2.0) # recall-weighted
return {"loss": -f2, "status": STATUS_OK}
# Spark ML: the Evaluator passed to CrossValidator IS the selection criterion
evaluator = BinaryClassificationEvaluator(labelCol="label",
rawPredictionCol="rawPrediction",
metricName="areaUnderPR")
A frequent real-world failure — and a plausible exam distractor — is selecting the best
model by one metric while reporting another. The evaluator handed to CrossValidator
is what decides bestModel, so it must be the metric the business actually cares
about.
One Metric to Optimise, Several to Watch
Tuning requires a single scalar, but reporting should not be reduced to one number. A workable convention is to nominate one optimisation metric and a set of guardrail metrics that must stay within agreed bounds:
| Role | Example on a fraud model | Behaviour |
|---|---|---|
| Optimisation metric | PR-AUC | Maximised by the search |
| Guardrail | False positives per 10,000 transactions | Must stay under the operations team's review capacity |
| Guardrail | p95 inference latency | Must stay under the payment authorisation budget |
| Guardrail | Recall on the highest-value segment | Must not regress against the incumbent |
A candidate that wins on the optimisation metric but breaches a guardrail is rejected. This is exactly the shape of the automated validation gate described in Section 1.1, and it is why metric choice is an MLOps decision as much as a statistical one.
Translating a stated cost into $\beta$
$F_\beta$ treats recall as $\beta$ times as important as precision, so the direction of the cost asymmetry picks $\beta$ directly:
| Stated cost relationship | Choose |
|---|---|
| False negatives are much more expensive | $\beta > 1$ — typically $F_2$ |
| Costs are roughly symmetric | $\beta = 1$ — the ordinary $F_1$ |
| False positives are much more expensive | $\beta < 1$ — typically $F_{0.5}$ |
The exam does not ask for a numeric $\beta$; it asks you to read the asymmetry in the scenario and pick the metric that punishes the expensive error.
A hospital deploys a screening model for a serious disease. Missing a case can be fatal; a false alarm results in one additional low-cost confirmatory test. Which metric should the tuning loop optimise?
A fraud model operates at 0.2% positive prevalence. Which pair of choices is appropriate for comparing candidate models and for reporting performance at the deployed threshold?
A model's scores feed a downstream expected-loss calculation that multiplies the predicted probability by a dollar exposure. Which metric best reflects what this system needs from the model?