17.1 Choosing a Retraining Policy
Key Takeaways
- Google's MLOps guidance lists retraining triggers on demand, on a schedule, on availability of new training data, on model performance degradation, and on significant changes in data distributions.
- Schedule-based retraining fits when new labeled data arrives predictably, while degradation-based retraining needs ground truth or reliable drift signals.
- Continuous training (CT) automatically retrains and validates models in production, which Google describes as MLOps level 1.
- Label delay limits performance-triggered retraining: fraud or churn outcomes can take weeks, so drift on input features often serves as an earlier proxy signal.
- Every retraining policy still needs validation gates, because automatic retraining on bad data can deploy a worse model.
Last updated: September 2026
The exam guide lists determining an appropriate retraining policy. The policy answers three questions: when to retrain, on what data, and what must pass before the new model replaces the old one.
Why Models Need Retraining
| Change | Example | Effect |
|---|---|---|
| Data drift (input distribution shifts) | A new customer segment from a marketing campaign | The model sees unfamiliar inputs |
| Concept drift (the relationship between inputs and label changes) | Fraudsters change tactics, so the same features now mean something different | Accuracy drops even if inputs look similar |
| Seasonality and events | Holiday demand, pandemics, price changes | Past patterns stop holding |
| Upstream changes | A new data source or changed feature definition | Silent skew or breakage |
| Business changes | New products, policies, or markets | Model outputs no longer match needs |
Retraining Triggers
Google's MLOps guidance lists these pipeline triggers:
| Trigger | How it works on Google Cloud | Best when | Watch out for |
|---|---|---|---|
| On demand | Manually run the pipeline or template | Early stages, investigations, one-off fixes | Depends on people remembering |
| On a schedule | Scheduler API cron (daily, weekly, monthly) or a Managed Airflow DAG | New labeled data arrives on a predictable cadence, such as weekly sales | Wastes compute when nothing changed, and misses sudden shifts between runs |
| On availability of new training data | Eventarc trigger on BigQuery inserts, Pub/Sub message from ingestion, or an Airflow sensor | Data arrives irregularly, such as a monthly label file from a partner | Tiny data increments may not justify retraining |
| On performance degradation | Continuous evaluation with ground truth detects metric drops → alert → Pub/Sub → pipeline | Labels arrive quickly enough to measure accuracy | Needs labels, and label delay slows detection |
| On significant distribution change | Model Monitoring drift or skew alert → Pub/Sub → pipeline | Labels are slow, and input drift is a useful early signal | Not all drift hurts accuracy, so combine with evaluation gates |
Factors That Shape the Policy
Label availability and delay
- Fast labels (clicks, next-day demand): monitor accuracy directly and retrain on degradation or a short schedule.
- Slow labels (loan defaults after months, churn after a billing cycle): use input drift and prediction drift as early warnings, and retrain on a schedule timed to label arrival.
Rate of change
| Domain | Typical cadence |
|---|---|
| Ad click-through, news recommendations | Daily or continuous |
| Retail demand | Weekly, plus event-driven retraining before major promotions |
| Credit risk | Quarterly or semiannual, with strict validation and governance |
| Image defect detection on a stable line | Retrain when the line, product, or camera changes |
Cost and risk
- Each retraining run costs compute (training, evaluation, tuning) and review time.
- Each deployment carries regression risk, which is why validation gates and canaries are part of the policy (Chapters 13 and 15).
- Regulated models often need human approval after automated validation.
What Data to Retrain On
| Strategy | Description | Use when |
|---|---|---|
| Sliding window | Most recent N months only | Old behavior is no longer relevant |
| Expanding window | All history to date | Patterns are stable and rare events matter |
| Weighted recency | All data, with recent rows weighted more | Balance between stability and adaptation |
| Incremental / warm start | Continue training from the previous model on new data | Large models where full retraining is expensive. Watch for forgetting older patterns |
| Continuous tuning (Gemini) | Tune further from a tuned model or checkpoint with new examples | Gen AI tasks with new labeled examples (Chapter 10) |
Keep a fixed, time-appropriate evaluation set, plus the most recent period, so successive models are compared fairly.
Full Retrain vs. Re-tune vs. Redesign
| Situation | Action |
|---|---|
| Same features, fresh data, mild drift | Retrain with the existing pipeline and hyperparameters |
| Larger shifts or noticeably lower validation performance | Retrain plus hyperparameter tuning |
| New signals needed, or the concept fundamentally changed | Redesign features or model. This is a new experiment cycle, not automatic CT |
Designing a Retraining Policy: Worked Example
An online retailer's demand forecasting model:
- Labels: actual sales arrive daily, so accuracy (WAPE) can be computed weekly.
- Baseline trigger: a weekly schedule every Monday after the previous week's sales load, with a data-freshness sensor.
- Event trigger: WAPE on the last 7 days more than 15% worse than the validation benchmark → alert → Pub/Sub → an ad hoc pipeline run.
- Early warning: Model Monitoring drift alerts on price and promotion features notify the team but don't retrain automatically.
- Data: 2-year sliding window, with holiday periods kept from prior years.
- Gates: schema and statistics validation, champion comparison on the last 8 weeks, per-category checks, then a canary rollout.
Continuous Training in the MLOps Maturity Model
| Level | Characteristics |
|---|---|
| Level 0: Manual | Data scientists train and hand off models manually. Releases are infrequent. There's no CT and no production monitoring feedback |
| Level 1: ML pipeline automation | An automated training pipeline is deployed to production. Continuous training runs on triggers, with data and model validation, feature store (optional), and metadata management |
| Level 2: CI/CD pipeline automation | Adds automated build, test, and deployment of the pipeline itself, so new ML ideas (features, architectures) reach production quickly (Section 17.2) |
Test Your Knowledge
A lender's loan-default model gets true labels only 6 to 12 months after origination. Leadership wants early warning when the model may be going stale. Which signal is the most practical trigger for investigation?
A
B
C
D
Test Your Knowledge
A partner delivers labeled data at irregular intervals, sometimes twice in a week and sometimes once in two months. Which retraining trigger best matches this pattern?
A
B
C
D
Test Your Knowledge
An automated weekly retraining pipeline retrained on a week of corrupted data and deployed a much worse model. What should the retraining policy have included?
A
B
C
D