5.4 Lifecycle of a Machine Learning Solution

Microsoft's AB-731 bullet is describe the lifecycle of a machine learning solution. On a 45-minute business-leader exam this is not an MLOps engineering course. It is the story you use to stop a vendor who says "we trained it, so we are done." A model that is not monitored is a policy that expires in silence. Microsoft's machine learning operations (MLOps) maturity model makes the same point: level 0 is manual training and opaque production; level 4 emits metrics, detects drift, and can trigger retraining. You will not be asked to quote every level. You will be asked to know the stages and why skipping evaluation or monitoring is a business failure.

The ML lifecycle, in leader language

1. Problem framing

Start with a decision, not a dataset. Who will use the score? What action changes if the score is high versus low? What is the success metric in business units (fewer stockouts, faster ticket handle time, lower fraud loss) and the model metric you will not game (precision at a recall floor, forecast error, calibration)? Confirm that ML is even the right tool using the previous section. If you cannot name the owner of false positives, you do not have a problem statement.

2. Data

ML is downstream of data quality. You need representative history: the same population, seasonality, and edge cases the model will see in production. Label definitions must match the business (what counts as "churn"?). Privacy and security still apply: minimize personal data, restrict who can see training extracts, and document sources. Garbage in is not a slogan; it is how a claims model learns last year's workaround codes and then fails when operations cleans them up.

3. Train and validate

Training fits the model to examples. Validation (and a held-out test set) checks that you did not memorize noise. Leaders should hear data leakage here in a second sense: using information at train time that would not exist at prediction time (for example, a "closed date" when scoring an open claim). You do not need to run the split. You do need to ask whether the data scientist can explain the split and why production features will be available in real time.

4. Evaluate

Evaluation is more than a single accuracy number. Classification needs error types (missed fraud versus false alarms). Forecasts need error in units the CFO understands. Fairness asks whether error rates differ across groups that matter legally and ethically — regions, languages, tenure bands — using the same Privacy and security / fairness conversation the later governance chapter will deepen. If a rural clinic's patients are systematically scored worse because they were under-represented in training, "95% accurate" is not a responsible result.

5. Deploy

Deployment puts a versioned model behind an endpoint with the identity, network, and logging bar from section 5.2. Approved-registry and change-control policies belong here: production should not silently point at an unreviewed file. Microsoft's MLOps guidance promotes models through registries and gated environments (development, test, production) rather than emailing an unversioned model file to an app team.

6. Monitor

Production is where models die. Data drift is when incoming features no longer look like training (new product mix, new form fields). Concept drift is when the meaning of the target changes (fraudsters adapt; "high severity" is redefined). Monitor prediction volume, latency, error rates, and business outcomes, not only server uptime. If nobody owns a dashboard, you are at MLOps level 0 regardless of how expensive the training cluster was.

7. Retrain

Retraining is a controlled loop: new labeled data, same evaluation gates, new version, rollback plan. Microsoft's highest MLOps level uses drift or regression signals to trigger retraining. Leaders still decide cadence versus event-driven, and they still require a human to accept a model that affects credit, hiring, or safety.

Contrast: generative AI application lifecycle

Microsoft's GenAIOps guidance is explicit: classical MLOps focuses on training new models for specific tasks; many generative solutions consume pretrained models and treat the prompt plus orchestrator plus grounding data as the workload. Fine-tuning a foundation model can reuse MLOps (data prep, train, evaluate, deploy). Prompting and retrieval-augmented generation (RAG) add different assets: system prompts, chunking and index pipelines, and evaluation metrics such as groundedness and relevance rather than only precision and recall.

For AB-731, remember a compact genAI loop:

• Prompt — instructions, techniques, versioned system prompts (the previous prompt-engineering chapter).
• Ground — attach enterprise data through Graph, Search, or RAG so answers are about your business.
• Evaluate — quality, groundedness, safety (content filters, red teaming), cost in tokens.
• Govern — who may use it, what data it may see, logging, and human fallback.

Inner-loop GenAIOps still has DataOps and experimentation; outer-loop still has deployment, inferencing, monitoring, and feedback. What changes is what you version: not only model weights, but prompts, indexes, and orchestrator logic. Token usage, latency, and throttling errors become operational metrics a CFO can see on a Foundry resource.

Responsible evaluation: fairness, drift, human fallback

Fairness. Measure error across groups that the business and law care about. Do not ship a hiring ranker or a claims severity model on average accuracy alone. Involve HR, legal, and the people who live with the decisions.

Drift. Schedule reviews. Tie retraining to signals, not to "when someone has time." If input forms change, assume the model changed even if the file name did not.

Human fallback. Microsoft's AI security guidance calls for human-in-the-loop on high-risk actions (external transfers, configuration changes). The same idea applies to ML scores: low-confidence or high-impact cases go to a person with authority and an audit trail. Generative agents that can spend money or send mail need the same brake. Fallback is part of the lifecycle, not an embarrassment when automation is "not ready."

Scenario. A claims vice president launches a severity model. Three months later the mix of weather-related claims shifts and a rural region shows worse error. Responsible operation is not "the data science team already presented a notebook." It is monitor for drift, slice fairness, keep adjusters in the loop above a dollar threshold, and retrain only through the same evaluation gates used at launch.

What you fund as a transformation leader

• A named product owner for the model after go-live, not only during the pilot.
• Data-labeling and privacy review as first-class cost, not a volunteer side task.
• Evaluation that includes slices and safety, not a single demo accuracy.
• Monitoring and a retrain trigger, mapped to MLOps maturity you actually intend to reach.
• For genAI apps, prompt and index operations with the same seriousness as model weights.

If you remember one sentence for AB-731: ML is framed, trained, evaluated, deployed, watched, and retrained; generative apps are prompted, grounded, evaluated, and governed — and both still need humans when the decision can hurt people or the brand.