19.1 Configuring Model Monitoring on Agent Platform
Key Takeaways
- Model Monitoring v1 is generally available and configured on Agent Platform endpoints, while Model Monitoring v2 (preview) attaches a model monitor to a model version in Model Registry.
- Model Monitoring v2 supports tabular models served on Agent Platform or elsewhere, including GKE and BigQuery, through reference models.
- Model Monitoring v1 monitors every feature with a default alert threshold of 0.3 and runs every 24 hours by default, with a minimum granularity of 1 hour.
- Endpoint-based monitoring needs request-response logging, so private endpoints using VPC Network Peering can't be monitored this way.
- Monitoring v2 compares a target dataset with a baseline dataset using a time window and optional offset, and it can run on demand or on a schedule.
The exam guide lists configuring and using Model Monitoring on Gemini Enterprise Agent Platform to establish continuous evaluation metrics for production models. Monitoring catches silent failures: the endpoint is healthy and fast, but predictions are getting worse because the data changed.
Two Offerings
| Aspect | Model Monitoring v1 (generally available) | Model Monitoring v2 (preview) |
|---|---|---|
| Attached to | An Agent Platform endpoint (monitoring job per endpoint) | A model version in Model Registry (a model monitor per version) |
| Where the model runs | Agent Platform endpoints | Agent Platform or other infrastructure (GKE, BigQuery, and others) through reference models |
| Model types | Tabular AutoML and imported tabular custom models | Tabular models |
| Detection | Training-serving skew (needs training data) and prediction drift | Input feature drift, output inference drift, feature attribution changes, against any baseline |
| When to choose | Production-level support for an endpoint-deployed model | All other cases. Google says v2 has all v1 capabilities and more |
v1 users don't have to migrate. Both versions can run side by side during a transition.
Model Monitoring v1 on an Endpoint
- Ensure all models on the endpoint are tabular AutoML or imported tabular custom models.
- For skew detection, provide the training dataset (Cloud Storage or BigQuery) and the target field. For drift detection, training data isn't required.
- For custom models, provide an analysis instance schema (the job waits in pending until it has one).
- Configure:
- Alert thresholds per feature. The default is 0.3 for every feature. For example,
Age=0.4alerts when Age's distribution distance exceeds 0.4. - Sampling rate of logged prediction requests (all requests by default).
- Monitoring frequency: 24 hours by default, 1 hour minimum.
- Notifications: email and Cloud Monitoring channels (the console supports PagerDuty, Slack, and Pub/Sub).
- Alert thresholds per feature. The default is 0.3 for every feature. For example,
Model Monitoring v2 Building Blocks
1. Register the model
Register it in Model Registry. For models served elsewhere, register a reference model (a placeholder name without artifacts). Reference models can't use feature attribution monitoring.
2. Create a model monitor (one per model version)
- Schema: feature fields, prediction fields, and optional ground-truth fields, with data types and
repeatedfor arrays. AutoML Tables schemas are fetched automatically. - Defaults: objectives, training dataset, output location, and notification settings. Jobs use these unless overridden.
- Output: a Cloud Storage location. The monitoring service agent needs write access, or the service creates a default bucket.
3. Choose data sources
| Source | Notes |
|---|---|
| BigQuery table or query | Needs a timestamp column (timestamp_field) for time windows and continuous monitoring. Partition by timestamp for cost and speed |
| Cloud Storage | CSV (with header) or JSONL |
| Batch inference job | Monitoring runs soon after the batch job completes |
| Endpoint request-response logging | Requires logging enabled. Tables include logging_time. Private (VPC peering) endpoints aren't supported because they don't log requests |
| Managed dataset | Tabular dataset format |
Nested features aren't supported, so flatten them first. Array columns are flattened into individual values.
4. Define baseline and target
- Baseline: a reference, usually the training dataset or an earlier production window.
- Target: recent data to compare.
- Time specification: a start and end time range, or a window plus offset. For example, the target is the last 24 hours, and the baseline is the same 24 hours one week earlier.
- Larger windows reduce noisy alerts caused by small sample sizes.
5. Select objectives and thresholds
| Objective | Metrics |
|---|---|
| Input feature data drift | Categorical: L-infinity or Jensen-Shannon divergence. Numerical: Jensen-Shannon divergence |
| Output inference data drift | Same metrics, applied to predictions |
| Feature attribution | SHAP values, tracking changes in each feature's contribution |
6. Run and schedule
Run on-demand jobs or scheduled runs for continuous monitoring. Scheduled runs with time specifications support BigQuery and endpoint logging sources. Model Monitoring v2 itself isn't charged while in preview, but you still pay for the BigQuery, Cloud Storage, and batch inference it uses.
Monitoring When Ground Truth Arrives
Distribution monitoring is a proxy. When labels arrive, measure real performance:
- Store predictions with a request ID and timestamp (endpoint logging or batch output).
- Join them with ground truth in BigQuery when outcomes are known.
- Run the model evaluation component on recent windows in a scheduled pipeline, and compare with the validation baseline (Chapter 7).
- Alert on performance drops, sliced by key segments.
From Alerts to Action
| Alert | Typical response |
|---|---|
| Skew on key features right after launch | Check preprocessing parity and feature pipelines (Chapter 15) |
| Gradual drift over weeks | Investigate the cause. Retrain on recent data if accuracy is affected |
| Prediction drift with stable inputs | Possible concept change or an upstream label or business change. Check outcomes |
| Attribution shift for an important feature | Look for broken or changed feature definitions |
Route alerts to Pub/Sub so a Cloud Run function can open incidents or trigger a retraining pipeline where the policy allows (Chapter 17).
A company serves a tabular model on GKE, not on Agent Platform, and wants managed drift monitoring of its logged predictions in BigQuery. What should it use?
A team enables Model Monitoring v1 on an endpoint without changing any settings. How often do monitoring jobs analyze logged inputs, and what alert threshold applies to each feature?
A bank wants Model Monitoring based on endpoint request-response logging, but its model is on a private endpoint that uses VPC Network Peering. What is the issue?