Oversee Model QA/QC and Configuration Management

Quality Assurance for a Probabilistic Artifact

Traditional software quality assurance asks a deterministic question: given this input, did the system produce the specified output? A machine learning model has no such specification. The same customer scored by two versions can receive two different answers, and "correct" is a statement about a population rather than about one record. So model quality assurance and quality control (QA/QC) tests a distribution of behavior and controls the conditions under which a number was produced. Domain IV Task 2 of the Examination Content Outline names both halves: model performance and configuration management.

The Test Set Is a Single-Use Instrument

The team splits the prepared data three ways.

• Training set fits the model.
• Validation set is used repeatedly, to tune settings and choose between candidate models.
• Test set is held out and touched once, to produce the performance number quoted in the go/no-go memo.

The rule that matters to you is the third one. Every time the team scores the test set, sees a disappointing result, adjusts the model and scores it again, information about that set leaks into the model through the team's own choices. The number drifts upward, the model looks better each round, and none of that improvement will appear in production. This is quiet and undetectable from the outside, which is why the exam expects you to ask a blunt question at the gate: how many times has this test set been evaluated, and on what date was it frozen?

Cross-validation is the tool for the legitimate version of this work. In k-fold cross-validation the training data is split into k parts; the model trains on k-1 and scores on the held-out one, rotating through all k. You get a performance range instead of a single lucky number, and the test set stays untouched for the final confirmation.

Testing Beyond the Headline Metric

A single accuracy figure is the least informative artifact in Phase 4. The testing protocol you establish should require five more things.

1. Subgroup performance. Break the metric down by segment: geography, product line, channel, language, age band, and any protected characteristic you may lawfully measure. Overall accuracy of 92 percent can conceal 61 percent for one branch or one dialect. This is also the evidence base for the bias checks in Domain I.
2. Edge and rare cases. The high-value, low-frequency records the average hides: the catastrophic claim, the newly onboarded customer with three weeks of history, the seasonal spike.
3. Adversarial and malformed inputs. Empty fields, wrong encodings, out-of-range values, duplicated records, and deliberately manipulated inputs. Production will supply all of them. You want a documented behavior for each, not a stack trace.
4. Stability across reruns. Retrain with a different random seed on the same data. If the metric moves several points, the result is fragile and any single reported number is partly luck.
5. Behavior under realistic load. Latency and throughput at the volume the deployment plan assumes, because a model that meets its accuracy target but misses its latency ceiling has failed a constraint.

Configuration Management: What Must Be Pinned

This is the reproducibility rule for the whole methodology, and Domain IV Task 2 is where you enforce it. A machine learning result is a function of code, data, settings and environment together, so a result is reproducible only when five things are pinned as a set and recorded against the run:

1. Code at a specific commit — the pipeline, the training script and the configuration files, not just the notebook.
2. Data at a specific version or snapshot identifier, with a hash so the snapshot can be proved unchanged.
3. Configuration — the complete hyperparameter set actually used for that run, not the defaults sitting in the script.
4. Random seed, because the same code on the same data produces different results without one.
5. Environment — pinned library, framework and runtime versions, since a minor dependency upgrade can move a metric.

Pin four of the five and the result is an anecdote. The version of this failure you will actually meet is versioning the code alone, the habit carried over from conventional software delivery, which leaves the three inputs that move the number most — the data, the settings and the environment — entirely unrecorded.

Two systems hold that record, and confusing them is a common error because both are described as "versioning":

• Experiment tracking is the run-level record. Every training run is logged automatically with its code commit, data version, hyperparameters, seed, environment and resulting metrics. It answers "which run produced that number, and can we rebuild it?" It captures everything the team tried, including the runs that lost.
• The model registry is the promotion-level record. It holds the model artifacts that were actually selected, each with a version identifier, a lifecycle stage such as candidate, staging, production or archived, lineage back to the run that produced it, and a record of who approved each promotion and when.

Experiment tracking tells you what the team tried; the registry tells you what the organization deployed and on whose authority. The model card required by Domain I links straight to the registry entry, and the registry is what lets you answer the question auditors and regulators actually ask months later: which data and which code produced the model that made this decision?

Your job is to confirm both systems exist and are being used before Phase 4 opens, not to build them.

Watching Performance During Development

You do not read learning curves for a living, but you must recognize three patterns when the team shows them.

• Training performance high, validation performance much lower and widening. Classic overfitting: the model has memorized the training data rather than learned a general rule. Remedies are the team's problem; noticing and asking is yours.
• Both training and validation performance low. Underfitting, or more often a data problem that Phase 3 did not resolve.
• Performance near perfect. Treat this as an alarm. In a real business problem, 99-plus percent almost always means data leakage: a field present in the training data that would not exist at the moment of prediction. A collections flag inside a default model, a discharge code inside a readmission model, a claim-closure reason inside a fraud model. Duplicate records straddling the train and test split, or scaling the data before splitting it, produce the same illusion. The correct managerial reflex is to order a leakage investigation before the number reaches the sponsor, because a retracted celebration is far more expensive than a delayed one.

Peer Review and Independent Validation

The ECO asks you to coordinate peer reviews and technical validation of model designs. Schedule the design review before the expensive training runs, not after, and give it a scope: is the target variable the right one, is the metric the one the business agreed, are the features available at prediction time, is the evaluation design sound. Minutes with named reviewers and tracked actions are the artifact.

In regulated environments, peer review is not enough. Banking, insurance and healthcare organizations operate an independent model validation function, staffed outside the build team, that re-derives results and challenges assumptions before a model may be used. Treat that review as a gate with a lead time in the schedule. Teams routinely discover it exists three days before the planned launch.

Engineering Standards for Pipeline Code

The notebook that produced the winning experiment is not a production asset. The ECO enabler on coding standards and best practices means: the pipeline is in version control, changes are code-reviewed, the data transformation steps have tests, credentials and file paths are not hard-coded, dependencies are pinned, and a second engineer can run the whole thing end to end on a clean machine. If only one person can rebuild the model, you have a single point of failure disguised as a deliverable.

What the Exam Wants From You

The signature item in this task shows a model scoring far better than anyone expected. The tempting answers are to report the good news, to move to Phase 5, or to raise the success threshold. The correct answer is to ask for a leakage investigation first. Extraordinary results demand ordinary skepticism, and the project manager who verifies before announcing is the one the exam rewards.