15.1 Validating Data & Models in Pipelines
Key Takeaways
- BigQuery ML's ML.VALIDATE_DATA_SKEW compares serving data with statistics saved at training, and ML.VALIDATE_DATA_DRIFT compares two sets of serving data.
- BigQuery ML data validation uses L-infinity distance by default for categorical features, and both categorical and numerical thresholds default to 0.3.
- TFX pipelines validate data with StatisticsGen, SchemaGen, and ExampleValidator, and they gate models with Evaluator blessings and InfraValidator checks.
- Google Cloud Pipeline Components include model evaluation components for classification, regression, and forecasting that can feed conditional deployment steps.
- A validation gate should compare the candidate model with the current production model on the same held-out data before any deployment step runs.
The first consideration in the exam guide's pipeline section is validating data and models. In an automated pipeline, no human looks at every run, so validation steps decide whether training proceeds and whether a model deploys.
Why Pipelines Need Validation Gates
| Failure | Without a gate | With a gate |
|---|---|---|
| Upstream schema change (a column renamed or a type changed) | Training crashes or silently uses nulls | Pipeline stops at data validation with a clear error |
| Distribution shift in new data | The model trains on corrupted or unrepresentative data | Drift check alerts. The run pauses for review |
| New model worse than production | A worse model deploys automatically | Evaluation gate blocks deployment |
| Model artifact can't load in the serving container | Deployment fails in production | Infrastructure validation catches it first |
Data Validation
What to check
- Schema: expected columns, types, required fields, allowed categorical values.
- Statistics: row counts, missing-value rates, min, max, mean, and distinct counts within expected ranges.
- Skew: do new training or serving data differ from the data the current model was trained on?
- Drift: did this period's data change significantly from the previous period?
- Freshness and completeness: did today's partition arrive, and is it complete?
BigQuery ML validation functions
| Function | Purpose |
|---|---|
ML.DESCRIBE_DATA | Descriptive statistics for training or serving data |
ML.VALIDATE_DATA_SKEW | Compares serving data with the training statistics saved when the BigQuery ML model was created, so the original training data isn't needed |
ML.VALIDATE_DATA_DRIFT | Compares two sets of serving data, such as yesterday vs. today |
ML.TFDV_DESCRIBE | Fine-grained statistics equivalent to TensorFlow Data Validation's statistics generation |
ML.TFDV_VALIDATE | Compares statistics sets to find anomalies, like TFDV validate_statistics |
Thresholds: categorical features use L-infinity distance by default (Jensen-Shannon divergence is an option), numerical features use Jensen-Shannon divergence, and both default thresholds are 0.3. You can set per-feature thresholds.
SELECT * FROM ML.VALIDATE_DATA_SKEW(
MODEL `risk.credit_model`,
TABLE `risk.applications_2026_09_16`,
STRUCT(0.2 AS categorical_default_threshold,
0.2 AS numerical_default_threshold));
Run a query like this as a pipeline step (BigqueryQueryJobOp). If it reports anomalies, a condition stops the run or sends a notification.
TensorFlow Data Validation in TFX pipelines
Agent Platform Pipelines runs TFX pipelines as well as KFP pipelines. Google suggests TFX for workflows that process terabytes of structured or text data with TensorFlow. The TFX data validation chain:
- ExampleGen ingests data.
- StatisticsGen computes statistics.
- SchemaGen infers a schema, which you curate and version.
- ExampleValidator flags anomalies such as missing features, unexpected values, and skew or drift against the schema and previous statistics.
Other validation options
- Knowledge Catalog data quality scans on BigQuery tables for rule-based checks.
- Custom Python components (for example, pandas or Great Expectations style assertions) for domain rules such as "claim_amount must be positive."
Model Validation
Evaluation gates
- Evaluate the candidate on a held-out test set with the pipeline evaluation components (
ModelEvaluationClassificationOp,ModelEvaluationRegressionOp,ModelEvaluationForecastingOp), usually afterModelBatchPredictOp. - Compare with the champion on the same data. Absolute thresholds alone can pass a model that's worse than production.
- Check slices such as region and demographic groups, so an average gain doesn't hide a regression for one group. Preview GCPC components can detect data and model bias (Chapter 18).
- Gate deployment with a pipeline condition:
# A small custom component reads the evaluation artifact and returns True or False
gate = compare_to_champion(candidate_metrics=eval_task.outputs["evaluation_metrics"],
champion_auc_pr=champion_auc_pr, min_gain=0.005)
with dsl.If(gate.output == True):
ModelDeployOp(model=upload_task.outputs["model"], endpoint=endpoint, ...)
In TFX, the Evaluator computes metrics, often per slice, and blesses the model only when thresholds and comparison with the baseline pass. The Pusher deploys only blessed models.
Infrastructure validation
A model can be accurate but unservable: wrong artifact name, missing dependency, too slow to load. Infrastructure validation loads the model in a serving-like environment and sends test requests before deployment. Examples include TFX's InfraValidator, the AutoML Tabular workflow's InfraValidatorOp, or a staging endpoint smoke test.
Gen AI validation gates
For prompt or model changes in gen AI pipelines, run Gen AI evals (Chapter 7) against a fixed evaluation set, and gate on rubric pass rates, grounding, and safety scores.
Choosing What Happens on Failure
Not every anomaly should stop a pipeline. Decide per check:
| Check result | Typical policy |
|---|---|
| Schema broken (missing column, wrong type) | Hard stop. Training on broken data is never useful |
| Row count far below normal | Hard stop. A partial load is likely |
| Moderate drift on a few features | Soft gate: continue training but require human approval before deployment |
| Candidate slightly worse than champion | Register the version without the champion alias, and don't deploy |
| Fairness slice regression | Block deployment and escalate to the responsible AI review |
Record every gate decision as pipeline metadata so audits can show why a model was or wasn't deployed.
Putting It Together
| Stage | Gate | Action on failure |
|---|---|---|
| After ingestion | Schema and statistics validation | Stop the run, notify data owners |
| Before training | Skew and drift vs. training baseline | Stop, or continue with an alert, depending on policy |
| After training | Evaluation thresholds + champion comparison + slices | Don't upload as default and don't deploy |
| Before deployment | Infrastructure validation | Don't deploy |
| After deployment | Canary guardrails (Chapter 13) | Roll back traffic |
A BigQuery ML pipeline must check each day whether new serving data differs from the data the model was trained on, but the original training table was archived. Which function fits?
An automated retraining pipeline deploys any new model whose AUC exceeds 0.80. Last week it deployed a model with AUC 0.81 that replaced a production model with AUC 0.86. What is the best fix?
A model passes all accuracy gates, but deployment fails because the serving container can't load the exported artifact. Which pipeline step would have caught this before deployment?