17.1 Choosing a Retraining Policy
Key Takeaways
Google's MLOps guidance lists retraining triggers on demand, on a schedule, on availability of new training data, on model performance degradation, and on significant changes in data distributions.
Schedule-based retraining fits when new labeled data arrives predictably, while degradation-based retraining needs ground truth or reliable drift signals.
Continuous training (CT) automatically retrains and validates models in production, which Google describes as MLOps level 1.
Label delay limits performance-triggered retraining: fraud or churn outcomes can take weeks, so drift on input features often serves as an earlier proxy signal.
Every retraining policy still needs validation gates, because automatic retraining on bad data can deploy a worse model.
The exam guide lists determining an appropriate retraining policy. The policy answers three questions: when to retrain, on what data, and what must pass before the new model replaces the old one.
Why Models Need Retraining
| Change | Example | Effect |
|---|---|---|
| Data drift (input distribution shifts) | A new customer segment from a marketing campaign | The model sees unfamiliar inputs |
| Concept drift (the relationship between inputs and label changes) | Fraudsters change tactics, so the same features now mean something different | Accuracy drops even if inputs look similar |
| Seasonality and events | Holiday demand, pandemics, price changes | Past patterns stop holding |
| Upstream changes | A new data source or changed feature definition | Silent skew or breakage |
| Business changes | New products, policies, or markets | Model outputs no longer match needs |
Retraining Triggers
Google's MLOps guidance lists these pipeline triggers:
| Trigger | How it works on Google Cloud | Best when | Watch out for |
|---|---|---|---|
| On demand | Manually run the pipeline or template | Early stages, investigations, one-off fixes | Depends on people remembering |
| On a schedule | Scheduler API cron (daily, weekly, monthly) or a Managed Airflow DAG | New labeled data arrives on a predictable cadence, such as weekly sales | Wastes compute when nothing changed, and misses sudden shifts between runs |
| On availability of new training data | Eventarc trigger on BigQuery inserts, Pub/Sub message from ingestion, or an Airflow sensor | Data arrives irregularly, such as a monthly label file from a partner | Tiny data increments may not justify retraining |
| On performance degradation | Continuous evaluation with ground truth detects metric drops → alert → Pub/Sub → pipeline | Labels arrive quickly enough to measure accuracy | Needs labels, and label delay slows detection |
| On significant distribution change | Model Monitoring drift or skew alert → Pub/Sub → pipeline | Labels are slow, and input drift is a useful early signal | Not all drift hurts accuracy, so combine with evaluation gates |
Factors That Shape the Policy
Label availability and delay
- Fast labels (clicks, next-day demand): monitor accuracy directly and retrain on degradation or a short schedule.
- Slow labels (loan defaults after months, churn after a billing cycle): use input drift and prediction drift as early warnings, and retrain on a schedule timed to label arrival.
Rate of change
| Domain | Typical cadence |
|---|---|
| Ad click-through, news recommendations | Daily or continuous |
| Retail demand | Weekly, plus event-driven retraining before major promotions |
| Credit risk | Quarterly or semiannual, with strict validation and governance |
| Image defect detection on a stable line | Retrain when the line, product, or camera changes |
Cost and risk
- Each retraining run costs compute (training, evaluation, tuning) and review time.
- Each deployment carries regression risk, which is why validation gates and canaries are part of the policy (Chapters 13 and 15).
- Regulated models often need human approval after automated validation.
What Data to Retrain On
| Strategy | Description | Use when |
|---|---|---|
| Sliding window | Most recent N months only | Old behavior is no longer relevant |
| Expanding window | All history to date | Patterns are stable and rare events matter |
| Weighted recency | All data, with recent rows weighted more | Balance between stability and adaptation |
| Incremental / warm start | Continue training from the previous model on new data | Large models where full retraining is expensive. Watch for forgetting older patterns |
| Continuous tuning (Gemini) | Tune further from a tuned model or checkpoint with new examples | Gen AI tasks with new labeled examples (Chapter 10) |
Keep a fixed, time-appropriate evaluation set, plus the most recent period, so successive models are compared fairly.
Full Retrain vs. Re-tune vs. Redesign
| Situation | Action |
|---|---|
| Same features, fresh data, mild drift | Retrain with the existing pipeline and hyperparameters |
| Larger shifts or noticeably lower validation performance | Retrain plus hyperparameter tuning |
| New signals needed, or the concept fundamentally changed | Redesign features or model. This is a new experiment cycle, not automatic CT |
Designing a Retraining Policy: Worked Example
An online retailer's demand forecasting model:
- Labels: actual sales arrive daily, so accuracy (WAPE) can be computed weekly.
- Baseline trigger: a weekly schedule every Monday after the previous week's sales load, with a data-freshness sensor.
- Event trigger: WAPE on the last 7 days more than 15% worse than the validation benchmark → alert → Pub/Sub → an ad hoc pipeline run.
- Early warning: Model Monitoring drift alerts on price and promotion features notify the team but don't retrain automatically.
- Data: 2-year sliding window, with holiday periods kept from prior years.
- Gates: schema and statistics validation, champion comparison on the last 8 weeks, per-category checks, then a canary rollout.
Continuous Training in the MLOps Maturity Model
| Level | Characteristics |
|---|---|
| Level 0: Manual | Data scientists train and hand off models manually. Releases are infrequent. There's no CT and no production monitoring feedback |
| Level 1: ML pipeline automation | An automated training pipeline is deployed to production. Continuous training runs on triggers, with data and model validation, feature store (optional), and metadata management |
| Level 2: CI/CD pipeline automation | Adds automated build, test, and deployment of the pipeline itself, so new ML ideas (features, architectures) reach production quickly (Section 17.2) |
A lender's loan-default model gets true labels only 6 to 12 months after origination. Leadership wants early warning when the model may be going stale. Which signal is the most practical trigger for investigation?
Daily accuracy computed against ground-truth default outcomes
Input and prediction drift alerts from Model Monitoring
The number of predictions the endpoint serves each day
Average CPU utilization across the serving endpoint
A partner delivers labeled data at irregular intervals, sometimes twice in a week and sometimes once in two months. Which retraining trigger best matches this pattern?
A fixed daily retraining schedule using the scheduler API
Manual retraining only when someone happens to notice
Retraining whenever the endpoint's latency starts rising
An event trigger that fires when new training data lands
An automated weekly retraining pipeline retrained on a week of corrupted data and deployed a much worse model. What should the retraining policy have included?
Data validation and model validation gates before deployment
A shorter retraining interval
Removing the evaluation step to speed up retraining
A larger training cluster
Sections you finish are checked off in the contents.