5.1 Validating Data and Models Inside Pipelines
Key Takeaways
- A pipeline should infer a schema once from a trusted baseline, version it, and validate every subsequent batch against it rather than re-inferring each run.
- Data validation catches four distinct anomaly classes: schema violations, distribution shift, missing or out-of-range values, and training-serving skew between a training batch and served traffic.
- A validation failure needs an explicit policy — halt, quarantine to a dead-letter destination, or proceed with a warning — decided per anomaly class rather than left to default behaviour.
- Model validation compares a candidate against the current champion on a fixed evaluation set and on business-critical data slices, not on the aggregate metric alone.
- Validation gates belong inside the pipeline so an automated retrain cannot silently promote a degraded model.
5.1 Validating Data and Models Inside Pipelines
Blueprint reference: Section 5.1, "Validating data and models."
An automated retraining pipeline without validation gates is a machine for shipping degraded models quickly. This section covers the two gates; the promotion mechanics that consume their verdict are covered in Section 5.5, MLOps CI/CD Automation, Retraining Triggers and Skew Prevention.
Schema: Infer Once, Version, Then Enforce
The standard mistake is inferring a schema from each incoming batch. If today's batch is broken, the inferred schema describes the breakage and validation passes.
The correct pattern:
- Infer a schema from a trusted baseline dataset — one that has been reviewed and is known good.
- Review and curate it. Inferred schemas need human adjustment: mark features required or optional, set valid domains for categoricals, set numeric ranges, declare expected value counts.
- Version it in source control alongside the pipeline code, so a schema change is a reviewed commit rather than a silent drift.
- Validate every subsequent batch against the versioned schema, and treat a needed schema change as a deliberate update.
Baseline data ──► infer schema ──► human review ──► schema v3 (in Git)
│
New batch ─────────────────► validate against v3 ───────┘
│
anomalies? ──► policy per anomaly class
The Four Anomaly Classes
| Class | Example | Usual cause |
|---|---|---|
| Schema violation | New categorical value; a column disappears; type changes | Upstream producer changed |
| Distribution shift | A feature's mean moves sharply between batches | Real-world change, or an upstream bug |
| Missing / out-of-range values | Null rate jumps from 1% to 40%; ages of 300 | Broken join, sensor failure, unit change |
| Training-serving skew | Serving traffic distribution differs from the training batch | Different preprocessing paths, or the world moved |
Distinguishing them matters because the responses differ. A new categorical value may be legitimate growth that requires a schema update; a null rate jumping to 40% is almost always an upstream break; a distribution shift may be genuine and call for retraining rather than blocking.
Failure Policy Per Class
A validation step must be told what to do, and "fail the pipeline" is not always right.
| Anomaly | Reasonable default |
|---|---|
| Missing required feature, type change | Halt. Training on this batch produces a broken model |
| Null rate or range violation beyond threshold | Halt and alert; this is nearly always an upstream defect |
| New categorical value below a small fraction of rows | Warn and proceed, routing affected rows to a dead-letter destination |
| Distribution shift beyond threshold | Warn, and treat as a retraining signal rather than a blocker |
| Individual malformed records | Quarantine to a dead-letter table with the raw payload and the error |
The anti-pattern the exam probes is a pipeline that silently drops bad records. A job that discards 30% of its input and reports success is indistinguishable from a healthy one until a model degrades for untraceable reasons.
Model Validation Beyond the Aggregate Metric
Once a candidate is trained, validation asks whether it should be allowed anywhere near production. Four checks, in order:
1. Absolute threshold. Does the candidate clear the minimum acceptable metric at all? A retraining run on corrupted data can produce a model that fails this outright.
2. Comparison to the champion. Evaluate the candidate and the current production model on the same fixed evaluation set. A candidate that is worse than the model already serving should not be promoted regardless of how it compares to last month's run.
3. Slice-level checks. This is the one teams omit and the exam rewards. An aggregate improvement can conceal a serious regression on a subgroup — a candidate that gains one point overall while losing eight points on a minority segment or on the highest-value customer tier is not an improvement. Define the slices that matter (region, device type, customer tier, protected attributes where fairness is in scope) and require no material regression on any of them.
4. Operational checks. Does the artifact load? Does it respond within the latency budget on the target machine type? Does the input schema still match what serving sends? A model that is statistically better and 400 ms slower fails the deployment.
candidate ──► absolute threshold ──► beats champion? ──► no slice regression? ──► loads & meets latency?
│ │ │ │
fail fail fail fail
└──────────────► do not promote; alert; keep champion serving ────────┘
Evaluation Set Discipline
- Fixed and versioned. A metric comparison across model versions is only meaningful on identical data.
- Held out from tuning. An evaluation set used to select hyperparameters has become a training set.
- Refreshed deliberately. As the world changes, the evaluation set eventually needs updating — but that is a reviewed change, and the champion should be re-scored on the new set so comparisons remain like for like.
Exam Traps
- Inferring the schema from each incoming batch. It legitimizes the breakage.
- Silently dropping invalid records. Dead-letter them instead.
- Promoting on an aggregate metric alone. Check slices.
- Comparing a candidate to a stale number rather than re-scoring the champion on the same set.
- Treating every anomaly as a blocker. Distribution shift is often a retraining signal, not a defect.
A retraining pipeline infers its data schema from each incoming batch and validates the batch against that inferred schema. Validation has never failed. What is wrong with this design?
A candidate model improves aggregate AUC from 0.812 to 0.821 and the pipeline promotes it automatically. Two weeks later, complaints surface from customers in one region where the model performs far worse than before. What validation step was missing?
A daily ingestion step begins receiving a small number of previously unseen values in a categorical feature, representing under 1% of rows. Which failure policy is most appropriate?
A team compares each newly trained candidate against the metric recorded for the champion when it was originally trained six months ago. Why is this comparison unreliable?