13.3 Inference Preprocessing & Postprocessing
Key Takeaways
- Preprocessing can live in the model graph, in the serving container (for example, a custom prediction routine), or upstream in the application, and the choice affects training-serving consistency.
- Batch inference instanceConfig can include or exclude input fields and send instances as JSON objects or arrays without custom container code.
- Postprocessing turns raw model outputs into decisions: thresholds, label mapping, top-k filtering, calibration, and business rules.
- Gen AI postprocessing validates structured outputs against a schema, checks grounding or citations, and filters unsafe or sensitive content before responses reach users.
- Putting shared preprocessing inside the exported model or serving container avoids re-implementing it in each client application.
The exam guide lists developing solutions for inference preprocessing and postprocessing. The design question is where each transformation runs, because that decides latency, consistency with training, and how many teams must maintain the logic.
Where Preprocessing Can Run
| Location | Examples | Pros | Cons |
|---|---|---|---|
| Inside the model | Keras preprocessing layers, a TensorFlow Transform graph exported with the model, BigQuery ML TRANSFORM | Training and serving use identical logic. Clients send raw features | Limited to operations the framework can express |
| Serving container | Custom prediction routine preprocess(), custom container code, Triton ensembles | Arbitrary Python logic (tokenizers, lookups) next to the model. One implementation for all clients | Adds latency on every request. Code must match training |
| Upstream in the application or gateway | API service validates and enriches the request, and fetches features from Feature Store | Keeps the model server simple. Reuses app data | Risk of skew if clients implement it differently |
| Data pipeline (batch/streaming) | Dataflow MLTransform then RunInference, BigQuery SQL before batch scoring | Scales for large volumes. Full-pass statistics computed once | Not for synchronous low-latency calls |
Rule: logic that must match training exactly should be inside the model or shared code used by both training and serving (Chapter 15). Business logic that changes often (display labels, promotion rules) belongs after the model, where it can change without retraining.
Preprocessing Patterns
Online inference
- Validate inputs: types, ranges, and required fields. Reject bad requests with clear errors instead of producing garbage predictions.
- Enrich with features the client doesn't have. For example, fetch
customer_90d_spendfrom Feature Store online serving (Chapter 14). - Transform: scale, encode, and tokenize, using the same fitted artifacts (vocabularies, scalers) saved at training time.
- Batch small requests on the server when the framework supports it (for example, TensorFlow Serving's batching configuration) to use GPUs efficiently.
Custom prediction routines (CPR)
A CPR Predictor separates stages cleanly:
| Method | Typical work |
|---|---|
load(artifacts_uri) | Load the model plus preprocessing artifacts, such as a scaler or tokenizer |
preprocess(request) | Parse instances, validate, and transform |
predict(instances) | Run the model |
postprocess(outputs) | Map outputs to response JSON with labels, scores, and reasons |
A custom Handler covers raw HTTP concerns such as unusual payload formats or headers.
Batch inference without custom code
instanceConfig on a batch inference job can:
excludedFieldsorincludedFields: drop keys or extra columns (such ascustomerId) from what the model sees, while keeping them in the output for joins.instanceType: send each instance as a JSON object or array.
For heavier preprocessing, run a BigQuery or Dataflow step before the batch job, or use Dataflow RunInference so transformation and scoring run in one pipeline.
Postprocessing for Predictive Models
| Step | Example |
|---|---|
| Thresholding | Convert probability 0.73 to "flag for review" using the business-chosen threshold (Chapter 7) |
| Label mapping | Class index 4 → "Water damage" |
| Top-k and filtering | Return the top 10 recommendations, excluding items already purchased or out of stock |
| Calibration | Adjust scores so a 0.8 score really means about 80% likelihood |
| Business rules | Never auto-decline a loan. Send declines to human review |
| Explanations formatting | Turn feature attributions into human-readable reason codes |
| Output validation | Clamp impossible values, such as negative delivery times |
Pre- and Postprocessing for Gen AI
| Stage | Examples |
|---|---|
| Pre | Prompt templating with system instructions, retrieving grounding context (RAG), redacting PII from user input, screening prompts for injection (Model Armor, Chapter 18), trimming context to control tokens |
| Post | Validating JSON against the response schema, checking that citations exist, filtering unsafe content, redacting sensitive data in responses, retry or fallback when validation fails |
Example: An insurance claims assistant sends a claim note with retrieved policy clauses and a JSON schema. Postprocessing validates the JSON, confirms each cited clause ID exists, and redacts any account numbers before returning the result.
Testing Pre- and Postprocessing
- Unit tests for each transformation, with fixed inputs and expected outputs.
- Parity tests that run the training-time transformation and the serving-time transformation on the same records and require identical results.
- Contract tests that send real client payloads, including malformed ones, to a staging endpoint.
- Golden prediction sets: fixed inputs whose expected outputs are checked after every container or model update.
Performance and Reliability Considerations
- Measure latency by stage. Tokenization or feature lookups can exceed model inference time.
- Cache static lookups, such as vocabularies and reference tables, in memory at
load(). - Keep containers stateless so autoscaling replicas behave identically.
- Version preprocessing artifacts with the model version. A new vocabulary means a new model version.
- Log inputs after preprocessing (where privacy allows), so skew monitoring compares what the model actually saw.
Worked Scenario
A mobile app sends raw text reviews for sentiment scoring, and several app teams call the endpoint.
- Put tokenization and truncation in a CPR
preprocess(), using the tokenizer saved at training time, so no app team re-implements it. - Return
{"sentiment": "negative", "score": 0.91}frompostprocess(), and keep the threshold configurable. - Fetch nothing upstream. Clients send only the review text and a request ID.
Several client applications call an online endpoint, and each implements its own text normalization before sending requests. Predictions differ across apps for identical reviews. What is the best fix?
A BigQuery input table for batch inference has a customerId column that the model must not receive as a feature, but results must be joinable back to customers. What is the simplest solution?
A gen AI extraction service must return valid JSON with required fields, and downstream systems crash on malformed responses. Which postprocessing step addresses this directly?