3.3 Choosing the Deployment Strategy at Design Time
Key Takeaways
- The deployment strategy is a design-time decision because it constrains model size, framework, feature availability, and preprocessing before any training begins.
- The first question is online versus batch: if consumers can tolerate stale scores, batch prediction removes an entire class of serving cost and complexity.
- Features that will not exist at request time cannot be used in an online model, no matter how predictive they are offline.
- Serving surface choice — Agent Platform Inference, Cloud Run, GKE, or embedded/edge — follows from latency, scale, isolation, and operational ownership.
- Rollout mechanics (canary, traffic split, shadow) belong to the same design decision, because they determine whether two model versions must be servable simultaneously.
3.3 Choosing the Deployment Strategy at Design Time
Blueprint reference: Section 3.1, "Choosing the deployment strategy."
Google lists this bullet in Section 3 — scaling prototypes into models — not in Section 4, where serving lives. That placement is the lesson: deployment strategy is decided before training, because it constrains what you are allowed to build.
Why This Is a Design-Time Decision
Four things are fixed by the deployment choice and cannot be retrofitted cheaply:
- Which features are legal. A feature computed from a 30-day aggregate is available in a nightly batch job. It is available online only if something maintains it in a low-latency store. If neither is true, the model cannot use it — and discovering this after training wastes the whole cycle.
- How large the model may be. A model that must run on a mobile device has a hard size and memory budget. One served from an endpoint does not.
- Which frameworks and versions are viable. Prebuilt inference containers cover specific framework and version pairs; anything else requires a custom container.
- How preprocessing is packaged. Online serving usually requires preprocessing to travel with the model; batch scoring can apply it upstream in the pipeline.
Decision One: Online or Batch
| Question | Batch | Online |
|---|---|---|
| Are scores needed for a known population, on a schedule? | Yes | — |
| Do inputs only exist at request time? | — | Yes |
| Is a few hours of staleness acceptable? | Yes | — |
| Is per-request latency budgeted in milliseconds? | — | Yes |
| Cost profile | Resources provisioned for the run, then released | Endpoint bills continuously |
Batch prediction is the underused answer. Propensity scores, next-best-offer rankings, lead scores, churn risk, and content recommendations for a known user base can all be computed nightly and read from a serving table. The application then does a key lookup rather than an inference call, which is faster, cheaper, and more reliable than a live endpoint.
Online is required when the input is only knowable at request time — the contents of the current basket, the text the user just typed, the image just uploaded — or when the decision must reflect the current instant.
A common production shape is both: batch-precompute the expensive user-level features nightly, and combine them online with request-time context.
Decision Two: The Serving Surface
| Surface | Choose when |
|---|---|
| Agent Platform Inference endpoint | Standard managed serving: autoscaling, traffic splitting, monitoring integration, accelerator support |
| Batch prediction job | Large offline scoring; no endpoint to maintain |
| Cloud Run | Lightweight CPU models, scale-to-zero, spiky traffic, an existing container-first team |
| GKE | Existing Kubernetes estate, complex multi-container inference graphs, fine-grained control |
| BigQuery ML in place | Scores consumed analytically inside the warehouse |
| Edge / on-device | Offline operation, strict privacy, or network-independent latency |
Scale-to-zero is the distinguishing property of Cloud Run in this list. A model serving a handful of requests an hour costs nearly nothing on Cloud Run and costs a full node-hour rate on an always-on endpoint. The trade-off is cold starts, which matters if the p95 must stay low even for the first request after idleness.
Edge deployment imposes the tightest design constraints: the model must be quantized or distilled to fit, the feature set must be computable on-device, and updates require a distribution mechanism. If a scenario says a device operates without connectivity, or that images may not leave the device, edge is decided at design time and everything else follows.
Decision Three: Rollout Requirements
Rollout mechanics feed back into design because they determine whether two versions must run at once.
- Traffic splitting / canary — two model versions deployed to one endpoint with a percentage split. Requires that both versions accept the same request schema.
- Shadow deployment — the candidate receives a copy of production traffic and its predictions are logged but not returned. Requires the ability to mirror requests and a place to log predictions for comparison.
- Blue-green — full switch between two ready deployments. Requires double capacity during the transition.
- A/B test — traffic split with an outcome metric measured per arm. Requires request-level assignment and outcome tracking, which is a data-pipeline requirement, not just a serving one.
If the organization requires canary rollouts, the request and response schema must be stable across versions from the first design. A model whose input schema changes with every retrain cannot be canaried.
Feature Availability: The Most Common Design Failure
Worth stating on its own because it appears repeatedly on the exam. A model trained with a feature such as "customer's total spend over the last 90 days" will fail in online serving unless that aggregate is maintained somewhere it can be read in milliseconds — which is what Feature Store exists for. The three legal outcomes are:
- Maintain the feature in an online store (Feature Store, Bigtable, Memorystore).
- Compute it cheaply at request time from data already in the request.
- Drop the feature and accept the accuracy cost.
Training with a feature and hoping to solve serving later is not one of them.
Exam Traps
- Choosing an endpoint for a workload with no real-time requirement. Batch is cheaper.
- Using an offline-only feature in an online model. Feature Store or drop it.
- Ignoring cold starts when choosing scale-to-zero for latency-sensitive traffic.
- Planning canary rollouts with a schema that changes per version.
- Deferring the deployment decision until after training. The blueprint deliberately places it before.
A model uses a feature representing a customer's rolling 90-day transaction total. It performs well offline. The serving requirement is a 60 ms online endpoint call during checkout. What must be designed before training completes?
A content platform ranks recommendations for its 4 million registered users. Rankings refresh once daily and are shown on the home screen at login. The team is debating an always-on endpoint versus batch prediction. Which is more appropriate and why?
A field inspection app must classify equipment defects from photographs on handsets that frequently operate with no connectivity, and images may not leave the device for privacy reasons. What does this determine at design time?
An organization mandates that every model release go out as a canary with a gradual traffic increase. During design, which constraint must be honoured?