Oversee Model Governance in Production

The Model Is an Asset with a Lifecycle

Ask an organization that has been running models in production for two years how many are currently serving live traffic, who owns each one, and what each was approved to do. The answer is very often a guess, and closing that gap is what ECO Domain V, Task 3 exists to do. The task asks you to govern each deployed model as a living asset with a named owner and a controlled lifecycle rather than as a finished deliverable — necessary because a model's accuracy moves while its code sits untouched, so nobody is prompted to look. Every production model runs through four stages:

1. Registration. The model is entered into the model inventory with its version, purpose, owner, risk tier, training data reference, approved use, documented limitations, and the approvals that let it into production.
2. Active service. The model runs under monitoring, with defined performance expectations and a review cadence proportionate to its risk tier.
3. Revision. Retraining, re-thresholding, feature changes and scope changes all pass through change control.
4. Retirement. The model is withdrawn, its endpoint decommissioned, and its records retained for the required period. Undocumented zombie models still serving traffic are a genuine audit finding.

Every entry has a named model owner who is accountable for its performance and for calling the decision to retrain, restrict or retire. This is operational stewardship; it is distinct from the accountability documentation and audit trail of Domain I, Task 5, which records who was answerable and what evidence exists, while this task controls how the asset is managed over time.

Versioning and Change Control in Production

The most common governance failure is treating a retrained model as a routine refresh. It is not. A retrained model is new software with new behavior, produced from different data, and it must move through the same controls the original did: a new version identifier, a registry entry, updated documentation, validation against a held-out set, a comparison against the model currently in production, and a recorded approval before promotion.

Build the change process to answer three questions for any production change: what changed (data, features, algorithm, thresholds, or serving configuration), who approved it, and how it can be reversed. Keep an emergency path for urgent fixes, with a documented retrospective approval, so that teams do not bypass governance when something is genuinely on fire.

Monitoring Drift

Accuracy can fall while the code is untouched. That is the property of AI that distinguishes this domain from ordinary IT operations, and it is examined through the drift distinction.

A worked contrast makes the difference stick. A credit model trained mainly on applicants aged 35 to 50 starts receiving a wave of applications from 18 to 24 year olds: the inputs moved, but income and debt still predict default the same way, so this is data drift. During a sharp downturn, borrowers with historically strong scores begin defaulting at twice their usual rate while the applicant mix is unchanged: what counts as a safe borrower has itself changed, so this is concept drift. The responses differ, and so does the urgency.

Coordinating Retraining

Retraining strategy is a management decision, and there are two defensible forms:

• Scheduled retraining on a fixed cadence, such as monthly or quarterly. It is predictable, easy to resource and easy to audit, but it can arrive too late for a fast-moving concept shift and can waste effort when nothing has changed.
• Trigger-based retraining driven by monitoring thresholds: a drift index breach sustained over a defined window, an accuracy floor crossed once ground truth arrives, or a known external event such as a product launch, a pricing change or a regulatory change. It responds faster but needs carefully set thresholds, or it fires constantly.

Most mature programs run both: a baseline cadence with trigger-based escalation. Whichever you choose, agree the thresholds and the approval path before you need them.

Two traps recur. The first is the feedback loop: retraining on data the model itself shaped. A fraud model that blocks a transaction means you never learn whether it would have been fraudulent; a lending model that rejects an applicant generates no repayment history; a recommender only sees clicks on items it chose to show. Retraining on that record narrows the model onto its own past decisions and hides its errors. Mitigations are managerial as much as technical: hold out a small randomized control slice, keep sampling human review on cases the model would have handled, and deliberately capture outcomes for overridden decisions.

The second trap is promoting a retrained model because it is newer. Validate every retrained candidate against the incumbent on the same evaluation set, including performance across the segments your bias checks cover, and promote only on evidence.

Ongoing Compliance and Periodic Review

Schedule a periodic model review with a cadence set by risk tier, at which the owner presents performance since the last review, drift history, incidents and overrides, changes to the data supply, whether the documented intended use still matches actual use, and a recommendation to continue, retrain, restrict or retire. Confirm at the same time that the model still complies with the privacy, transparency and regulatory obligations set in Domain I, since laws and internal policy move independently of the model.

The exam behavior to internalize: when accuracy declines and no code has changed, investigate drift in the inputs and in the outcome relationship before rebuilding the model. Rebuilding first is expensive and frequently solves nothing.