Verify the Model Is Ready for Operationalization

The Last Gate Before Production

CPMAI Phase 5, Model Evaluation, decides whether there is anything worth deploying. Phase 6, Model Operationalization, and ECO Domain V then execute the deployment; this task is the approval that precedes it. By now the model has passed the quality and control work of Domain IV Tasks 1 to 3 and the data readiness gate of Task 5. Task 6 asks a harder question: is this fit to be run by an operations team, in front of real customers, for years?

Evaluate Against the Criteria You Already Agreed

The success criteria were fixed in Domain II, Task 8, during Business Understanding, before anyone had seen a result. This gate measures against those:

• The agreed technical metric and its threshold — the one written down, not the one that looks best today.
• The business-impact threshold — the operational or financial effect the project promised.
• The baseline the model must beat — the current process, the existing rules engine, or a simple heuristic. A model that beats random but not the rules it replaces has produced no value at greater cost and lower transparency.

Metric shopping is the trap the exam sets. The team agreed to recall at a fixed false-positive rate, arrives with a strong area-under-curve figure instead, and explains that the original metric "was not really the right one." Your response is neither to accept the better-looking number nor to accuse anyone: require the model to be reported against the agreed criterion and the baseline, and treat any change of metric as a formal change to the success criteria with the sponsor's approval.

Two translations matter. Convert the metric into business terms at production volume — "94 percent accurate" means nothing until you say what the other 6 percent costs per year. And be explicit about the operating threshold: performance is always reported at a chosen cut-off, and where that cut-off sits is a business trade between false positives and false negatives. The process owner sets it, not the data scientist, and reported results must be the results at that setting.

Robustness and Generalization

• The untouched test set. It counts only if it was genuinely held back. If it was used to compare candidates or tune a threshold, it is a second validation set and no honest generalization estimate remains.
• Recent data. Report performance on the most recent period, not the average across five years. A model can look fine overall while having quietly stopped working last quarter.
• Subgroups. Break results down by site, region, channel, product, customer tenure, and any protected attribute in scope. An aggregate number routinely hides a segment where the model performs worse than the process it replaces.
• Degraded and shifted inputs. What happens when a field arrives empty, a feed is late, an unseen category appears, or volume doubles? Whether the system degrades gracefully or fails belongs in the runbook.

Then set expectations honestly: offline performance overstates production performance, essentially always. Prepared data is cleaner than live data, the test set shares its era with the training data, and people behave differently once they know a model is scoring them. Never approve a business case that only works at the offline number.

Deployment Readiness

• Infrastructure provisioned and sized, with the cost per prediction known at forecast volume rather than at pilot volume.
• Latency and throughput load-tested at production peak. A model that answers in 40 milliseconds on a workstation and four seconds behind a real queue misses the operational window it was funded to hit.
• End-to-end integration testing through the upstream feed, the scoring service, the downstream system that consumes the score, and the human workflow that acts on it. Most unpleasant surprises live here, not in the model.
• Security review covering access control, endpoint exposure, data in transit and at rest, adversarial or prompt-injection testing where relevant, and the provenance of third-party models and libraries.
• A tested rollback path with a known execution time, a named person authorized to trigger it, and written criteria for when they should.

Documentation and Operational Procedures

• Model card — intended use, explicitly out-of-scope uses, a description of the training data and its date range, evaluation results overall and by segment, and known limitations.
• Monitoring plan — named metrics, thresholds, frequency, and an owner for each alert, covering input drift and, where feasible, actual outcomes rather than predictions alone.
• Retraining triggers — what condition initiates retraining, who authorizes it, and how a retrained model is re-validated before it replaces the incumbent.
• Runbook — the support team's document: how to tell it is working, what to do at each alert, whom to call, how to fall back to manual processing, how to roll back.
• Human oversight procedures — review, override, and appeal paths wherever the model affects people, plus completed training for the staff who will use it. A tool nobody trusts has been installed, not deployed.

The Approval Decision

You assemble the evidence pack: the evaluation report against the original criteria, test-set and subgroup results, robustness testing, load and integration results, the security sign-off, the bias and fairness assessment, the model card, the monitoring plan, the runbook, and the rollback plan. Signatures come from the people who carry the consequences.

You then have four ways to close the gate. You can approve outright. You can approve conditionally, releasing the model into a limited rollout — shadow operation, one region, a capped share of volume, or human review on every output — with the conditions named and a review date set. You can defer, holding approval until specific remediation is complete. Or you can reject and send the work back to an earlier phase. Nothing obliges you to pick the first one because the metric cleared.

The Exam Behavior to Internalize

A model that clears its technical metric but has no monitoring plan and no rollback path is not ready, and the project manager withholds approval. AI systems degrade as the world moves away from their training data; deploying without the ability to see that degradation or to reverse the release is the one-and-done deployment failure the entire methodology exists to prevent. Narrowing the rollout is a legitimate conditional approval — but only when the conditions include observability and reversibility. Scoping to a single region without monitoring merely makes the blind spot smaller.