15.3 Ensuring Consistent Preprocessing Between Training & Serving
Key Takeaways
- Training-serving skew happens when features are computed differently, or from different data, at training time and at serving time.
- Embedding preprocessing in the model, through BigQuery ML TRANSFORM, Keras preprocessing layers, or an exported TensorFlow Transform graph, makes serving apply identical logic.
- Apache Beam MLTransform can write transformation artifacts during training and read the same artifacts during inference.
- A feature store serves the same governed feature definitions for training (offline, point in time) and for online inference.
- Logging served feature values and comparing them with training distributions detects skew that code reviews miss.
The exam guide lists ensuring consistent data preprocessing between training and serving. The consequence of getting it wrong is training-serving skew: the model sees inputs in production that differ systematically from what it learned on, so accuracy drops even though nothing looks broken.
Sources of Training-Serving Skew
| Source | Example |
|---|---|
| Duplicated logic | Standardization in a SQL training query, re-implemented in Java for serving with a different mean |
| Different statistics | Serving code computes normalization statistics from the current request batch instead of the training set |
| Vocabulary mismatch | Categories encoded with a training vocabulary, while serving builds a new mapping |
| Different data sources | Training uses a cleaned warehouse table, while serving reads raw operational data with different nulls or units |
| Time leakage | Training uses end-of-day aggregates, while serving only has partial-day data |
| Feature availability | A feature populated in historical data is missing or delayed at request time |
| Library or version differences | A different tokenizer version in the serving container |
Pattern 1: Put Preprocessing Inside the Model
If the model artifact contains the transformations, every consumer gets the same logic.
| Tool | How |
|---|---|
BigQuery ML TRANSFORM | Preprocessing and its training statistics are saved with the model and applied by ML.PREDICT. Exported models include supported transformations (Chapter 2) |
| Keras preprocessing layers | Normalization, lookup, and text vectorization layers are part of the saved model |
| TensorFlow Transform (tf.Transform) | Full-pass analyzers compute statistics over training data (on Dataflow at scale) and export a transform_fn graph that is attached to the serving signature |
Best when transformations can be expressed in the framework and latency allows running them in the model graph.
Pattern 2: Share Transformation Artifacts
When preprocessing runs outside the model, fit once and reuse the fitted artifacts:
- Apache Beam
MLTransformwrites artifacts (vocabularies, scaling statistics) to a location during training and reads the same artifacts at inference in Dataflow pipelines. - scikit-learn Pipelines (scaler plus model saved together as
model.joblib) served by the prebuilt scikit-learn container. - A custom prediction routine that loads saved scalers or tokenizers in
load()and applies them inpreprocess()(Chapter 13).
Pattern 3: One Feature Definition for Offline and Online
For aggregated features, such as 30-day spend, compute them once in a governed pipeline and serve them from Feature Store (Chapters 5 and 14):
- Training reads point-in-time values (offline serving or
ML.FEATURES_AT_TIME) as of each label's timestamp. - Serving reads the latest synced values from the online store.
- No application team re-implements the aggregation.
Pattern 4: Same Code Path for Batch and Streaming
Apache Beam lets the same transformation code backfill historical training data in batch and compute real-time features in streaming on Dataflow. That removes the classic "Spark for training, custom Java service for serving" split.
Pattern 5: Package and Version Together
- Put shared preprocessing code in one versioned library imported by the training job and the serving container.
- Build training and serving images from the same base image and dependency lock file.
- Record preprocessing versions in ML Metadata, and register the model version with them.
Verifying Consistency
| Check | How |
|---|---|
| Parity tests | Run training-time and serving-time transformations on the same raw records and require identical outputs (within float tolerance) |
| Golden requests | Fixed inputs with known predictions, checked after every deployment |
| Served-feature logging | Log post-preprocessing features for sampled requests (request-response logging to BigQuery) |
| Skew monitoring | Model Monitoring compares serving feature distributions with training data (Chapter 19). BigQuery ML ML.VALIDATE_DATA_SKEW does the same for BigQuery ML models |
Gen AI Consistency
Consistency matters for gen AI too:
- Prompt templates used in tuning data must match production templates (Chapter 2).
- Chunking and embedding models used to index a RAG corpus must match those used to embed queries. Changing the embedding model means re-embedding the corpus.
- System instructions and tool definitions should be versioned with evaluation results.
Where Skew Checks Belong in the Lifecycle
| Stage | Consistency control |
|---|---|
| Design | Choose one feature definition and one owner per feature |
| Development | Shared preprocessing library, parity unit tests |
| Pipeline | Transformation artifacts versioned and passed between training and deployment steps |
| Deployment | Golden-request tests against the staging endpoint |
| Production | Request-response logging plus training-serving skew monitoring with alerts |
Worked Scenario
A delivery-time model trained on BigQuery data with distance_km computed by a SQL UDF performs well offline. In production, the mobile backend computes distance with a different formula, and MAE doubles.
Fix options, from strongest to weakest:
- Move the distance computation into a shared feature pipeline that feeds Feature Store, or into the model's TRANSFORM clause if the model is BigQuery ML, so clients send raw coordinates.
- Put the calculation in a CPR
preprocess()that uses the same library as training. - As an interim step, add parity tests and skew monitoring on
distance_kmto catch divergence immediately.
A team normalizes numeric features in training with statistics from the full training set, but the serving container computes the mean and standard deviation from each incoming request batch. What problem does this create, and what is the fix?
A Dataflow pipeline preprocesses data for training with MLTransform, computing vocabularies and scaling. A separate Dataflow streaming pipeline runs inference. How should inference apply the same transformations?
Several services compute a 30-day purchase count differently, and models trained on warehouse data perform worse online. Which architecture best removes this inconsistency?