Define Success Criteria and KPIs
Key Takeaways
- Success criteria come in three layers — model performance, business impact, and adoption or experience — and the project counts as successful only when the business layer moves.
- On imbalanced data accuracy misleads: on 100,000 claims of which 500 are fraudulent, a model that flags nothing scores 99.5% accuracy, so precision and recall carry the meaning.
- Precision is true positives divided by everything flagged; recall is true positives divided by everything that was truly positive; which one you favor is set by the relative cost of a false positive versus a false negative.
- A model must beat the documented incumbent baseline — today's rules engine or human performance measured before Phase 4 — not merely post a respectable score in isolation.
- The classification threshold is a business decision owned by whoever carries each error cost and the review capacity, and success criteria must be signed before model development, never written after seeing results.
Three Layers, and Only One of Them Decides Success
The exam expects a three-layer metric stack. Model metrics say how well the model predicts. Business impact metrics say whether the organization is measurably better off. Adoption and experience metrics say whether people are actually using it. All three go into the success criteria, but only one of them decides: a project with excellent model metrics and no movement in the business layer has failed, however good the demo was. Define all three in CPMAI Phase 1 (Business Understanding) so the Phase 5 (Model Evaluation) go/no-go has something real to test against.
Model Metrics at Management Depth
You do not need to build a classifier, but you must be able to read one. A confusion matrix compares what the model said with what was true. A fraud model scores 100,000 claims for a quarter; 500 of them — 0.5% — are truly fraudulent. The model flags 800:
| Actually fraudulent | Actually not fraudulent | |
|---|---|---|
| Model flagged | 400 (true positives) | 400 (false positives) |
| Model did not flag | 100 (false negatives) | 99,100 (true negatives) |
- Precision = true positives / everything flagged = 400 / 800 = 50%. Half of what the investigators open is a waste of their day.
- Recall (sensitivity) = true positives / everything truly fraudulent = 400 / 500 = 80%. Four fraud cases in five are caught.
- F1 score = the harmonic mean of precision and recall = about 0.62. One number for ranking candidate models; useless as a decision.
- Accuracy = (400 + 99,100) / 100,000 = 99.5%. Here is the trap: a model that simply answers "not fraudulent" every single time also scores 99.5% accuracy and catches nothing at all. On imbalanced data, accuracy measures the imbalance, not the model — which is why a headline accuracy figure on a rare-event problem should always make you ask for the confusion matrix.
Precision and recall trade against each other. Lower the score threshold and you catch more fraud (recall rises) while opening more innocent claims (precision falls). Which way you lean is decided by the relative cost of the two errors. A missed sepsis case can kill a patient, so recall dominates and false alarms are tolerated up to the point of alarm fatigue. A false decline on an e-commerce payment loses a good customer permanently, so precision matters more. For continuous targets — demand units, remaining useful life, claim cost — the equivalents are MAE (mean absolute error), RMSE (root mean squared error, which punishes large misses harder) and MAPE (mean absolute percentage error, readable by business people but unstable near zero).
Beat the Incumbent, Not an Abstract Standard
Whatever does the job today is your baseline: a rules engine, a scoring sheet, a vendor tool, or an experienced human. Measure it before Phase 4 (Model Development) begins, on the same data and the same definitions the model will be judged on. If the current rules engine catches 65% of fraud at 40% precision, then a model at 80% recall and 50% precision is a genuine gain, and a model at 62% and 38% is an expensive way to get worse — even though 62% sounds respectable in a slide. For human baselines, measure a sample of real decisions rather than asking people how accurate they think they are. Without a documented baseline, every later comparison is a comparison to zero, and the Phase 5 go/no-go becomes a matter of opinion.
The Threshold Is a Business Decision
A classifier outputs a score, not a verdict. Where you cut that score decides how many false positives and false negatives you buy. That cut is not a data-science optimum — the F1-maximizing threshold is a mathematical convenience, not a policy. It is set by whoever owns the cost of each error type, constrained by review capacity. If the special investigations unit can work 200 referrals a day, the threshold is capacity-bound before it is anything else. The project manager's job is to convene the business owner, the risk or compliance owner, and the operations manager; then document the chosen threshold, the expected daily volumes at it, and who approved it, in the scope statement and business case.
Business Impact Metrics and Thresholds
A business metric needs four things: a number, a source system, a measurement window, and an attribution method. "Reduce unplanned downtime" is a direction, not a criterion. "Reduce unplanned downtime hours on the packaging line by 15% within two quarters of go-live, measured in the maintenance management system against the prior four quarters, using the two untreated lines as a control" is a criterion. Attribution matters because business numbers move for many reasons; a holdout group, an A/B split, or a phased rollout with a control site is what lets you claim the improvement was the model's.
Adoption, Experience, and Technical Benchmarks
Carry the Task 6 measures — coverage, override rate, time-to-decision, follow-through — into the success criteria with targets set before go-live, and add a short user-satisfaction instrument with a stated cadence. Set technical benchmarks in the same document: 95th-percentile inference latency inside the workflow's budget, availability target, and cost per thousand predictions. A model that meets its recall target but misses the latency budget has still failed the workflow.
Align Every Criterion to an Organizational Objective
| Metric layer | Example metric | Who owns it | Threshold set by whom |
|---|---|---|---|
| Model performance | Recall and precision at the operating threshold | Data science lead | Business owner of the error costs, advised by data science |
| Technical performance | 95th-percentile latency, availability, cost per 1,000 predictions | Platform and MLOps engineering | IT architecture, against the workflow's latency budget |
| Business impact | Fraud losses avoided per quarter versus baseline | Executive sponsor or process owner | Executive sponsor, recorded in the business case |
| Adoption and experience | Coverage, override rate, user satisfaction | Operations manager for the affected team | Change sponsor with the operations manager |
| Strategic alignment | Contribution to the named corporate objective | Portfolio or PMO lead | Executive leadership |
If no named executive owns the business metric you have chosen, it is not a business metric — it is a project vanity metric, and it will not defend the project at the next budget review.
The Trap: Defining Success After You See the Results
When a team finishes Phase 4 with 0.71 recall and then proposes 0.70 as the success criterion, success has been reverse-engineered from the result. Criteria set that way can never fail, teach nothing, and collapse the moment finance asks what improved. Lock model, business, technical, and adoption criteria — each with a threshold and an owner — before development starts, and treat any later change as a documented, sponsor-approved change to the business case.
A pilot model that detects fraudulent claims reports 99.4% accuracy on a portfolio in which 0.6% of claims are fraudulent. The sponsor wants to declare the pilot a success and fund production. What should the project manager point out?
The data science team has tuned a sepsis-risk classifier to the threshold that maximizes its F1 score and asks the project manager to confirm that this threshold should go into the success criteria. What should the project manager do?
Phase 4 Model Development ends with a claims model at 0.71 recall. No success criteria were documented beforehand, and the team proposes adopting 0.70 recall as the target since the model exceeds it. The existing rules engine catches 0.65 of cases at noticeably higher precision. What should the project manager do?