13.2 Rollout Strategies: A/B Tests, Canaries & Traffic Splits

The exam guide lists implementing model rollout strategies (for example, A/B testing and canary deployments) to compare model versions. On Agent Platform, the main mechanism is the traffic split on an endpoint.

Traffic Splitting on Endpoints

An endpoint can have several deployed models (different models or versions) at once. The trafficSplit map assigns a percentage of requests to each deployed model ID, and the percentages must add up to 100. When deploying a new model, the key "0" refers to the model being deployed in that request:

"trafficSplit": {
  "0": 10,
  "DEPLOYED_MODEL_ID_CURRENT": 90
}

Each deployed model has its own replicas and machine configuration, so the new version gets its own capacity. Traffic splits work on dedicated public, shared public, and Private Service Connect endpoints. Private endpoints that use VPC Network Peering don't support traffic splitting.

Rollout Strategies Compared

Rolling deployment details

A rolling deployment (set with rolloutOptions) creates a new DeployedModel with a new ID and revision number. It reuses the previous deployment's traffic split and dedicated resources, and it replaces replicas as surge capacity allows. Traffic moves in proportion to ready replicas. If new replicas never pass health checks, no traffic goes to them, and the deployment fails and reverts to the previous model.

Designing a Canary

1. Define guardrail metrics before starting: error rate, p95 latency, prediction distribution (for example, share flagged as fraud), and business proxies.
2. Start small (1-10%) with enough traffic to see problems quickly.
3. Compare canary and control on the same time window, because traffic patterns change by hour and day.
4. Step up only when the guardrails hold for a set period.
5. Automate rollback: alert policies on endpoint metrics trigger a pipeline or script that sets the canary split to 0%.

Designing an A/B Test

• Hypothesis and KPI: "The new ranking model increases add-to-cart rate by at least 2%."
• Randomization unit: a split per request can put one user in both arms. For user-facing experiences, assign users consistently in the application layer (for example, by hashing the user ID) and call the matching deployed model, or use separate endpoints.
• Sample size and duration: run long enough to reach significance and cover weekly cycles.
• Logging: enable request-response logging, record the model version with each prediction, and join with outcome data in BigQuery.
• Delayed labels: fraud or churn outcomes may arrive weeks later, so plan the analysis window.

What to Monitor During Rollout

Cost of Safe Rollouts

Each deployed model on an endpoint has its own replicas, so a canary or blue/green rollout temporarily pays for two deployments. Keep this cost down by sizing the canary's minimum replicas for its traffic share, setting a clear end date for A/B tests, and undeploying the old version once rollback is no longer needed. A rolling deployment avoids double capacity by reusing the same compute, at the cost of less side-by-side comparison.

Gen AI Rollouts

The same patterns apply to prompt or model changes in gen AI applications:

• Canary a new prompt template or model version to a small share of users.
• Run Gen AI evals on sampled production traffic for both variants (Chapter 7).
• Watch safety filter block rates, cost per request, and latency, along with quality.

Worked Scenario

A bank deploys fraud model v8 next to v7 on a Private Service Connect endpoint.

1. Deploy v8 with trafficSplit {"0": 5, "v7": 95} and minimum replicas sized for 5% of peak.
2. For 48 hours, compare decline rate, manual-review volume, p95 latency, and error rate between the versions.
3. Raise to 25%, then 50%, then 100% as guardrails hold.
4. Keep v7 deployed at 0% for one week as an instant rollback path, then undeploy it to save cost.
5. Move the champion alias in Model Registry to v8.