4.6 Scaling the Serving Backend for Throughput
Key Takeaways
- Autoscaling on an inference endpoint is driven by a target utilization metric between a minimum and maximum replica count, and the minimum determines both floor cost and cold-start exposure.
- Load testing at the target QPS is the only way to establish replicas-per-QPS; capacity planning from assumption is the standard failure.
- Scaling out adds replicas for throughput; scaling up adds resources per replica for models that are too slow or too large on the current machine type.
- Endpoints do not scale to zero, so genuinely intermittent workloads belong on Cloud Run or in batch prediction rather than on an always-on endpoint.
- Feature lookups, preprocessing, and downstream calls are part of the latency budget and are frequently the real bottleneck rather than model compute.
4.6 Scaling the Serving Backend for Throughput
Blueprint reference: Section 4.2, "Scaling the serving backend based on the throughput (e.g., Gemini Enterprise Agent Platform Inference and containerized serving)."
Scaling questions are capacity-planning questions. The exam supplies a traffic shape and a latency budget and expects a configuration that meets both without paying for idle capacity.
How Endpoint Autoscaling Works
A deployed model on an inference endpoint is configured with:
- Minimum replica count — the floor. Always running, always billed. Determines resilience to sudden spikes and how often a request meets a cold replica.
- Maximum replica count — the ceiling. Caps both capacity and spend.
- A target metric — typically target CPU utilization, or accelerator duty cycle for accelerator-backed deployments. The service adds replicas when observed utilization exceeds the target and removes them when it falls below.
Two properties follow that questions like to test:
- A normal deployment has a floor of one replica. For a standard deployment
min_replica_countmust be at least 1 (and defaults to 1), so the endpoint always carries a baseline cost. There is a separate Scale To Zero feature, enrolled by settingmin_replica_count = 0, which lets a deployment drop to zero replicas and stop billing when idle — but the trade is explicit: a request arriving at a scaled-down endpoint is rejected with429 - Model is not yet ready for inferenceand is what triggers the scale-up back toinitial_replica_count. It therefore only suits callers that retry and tolerate that first failure, and it is not available for co-hosted deployment resource pools. - Scaling is reactive and takes time. A replica must be provisioned and the model loaded before it serves. For a large model this is not instant, so a spike arriving faster than replicas can start will queue or fail regardless of the maximum.
Scaling Out Versus Scaling Up
| Scale out (more replicas) | Scale up (bigger machine per replica) | |
|---|---|---|
| Fixes | Insufficient throughput at acceptable per-request latency | Per-request latency too high, or the model does not fit |
| Symptom | Latency rises with QPS; each request is individually fast at low load | Latency is high even at one request per second |
| Limit | Maximum replica count, quota | Largest available machine type |
The diagnostic is simple: measure latency at a single request per second. If it already exceeds the budget, more replicas will not help — the model or the machine is the problem. If it is fast alone and degrades under load, the answer is more replicas.
Capacity Planning by Load Test
The only defensible way to size an endpoint:
- Deploy one replica of the exact model version on the exact machine type.
- Drive load at increasing QPS, measuring p50, p95, and p99 latency and error rate.
- Find the QPS at which p95 crosses the budget — that is one replica's usable capacity.
- Divide the target QPS by that figure, add headroom for the autoscaler's reaction time, and set the minimum from the traffic floor and the maximum from the expected peak plus margin.
Include realistic payloads. A load test with a tiny synthetic input measures nothing useful if production requests carry large images or long documents.
Handling Spikes and Cold Starts
- Raise the minimum replica count to the level that absorbs a spike while the autoscaler reacts. This is the direct trade: predictable spike handling in exchange for continuous cost.
- Pre-warm before known events. For a campaign launch or a scheduled batch of traffic, raise the minimum ahead of time rather than relying on reactive scaling.
- Shrink the model so replicas start faster; load time is dominated by model size.
- Smooth the traffic with a queue where the workload tolerates asynchrony — a Pub/Sub buffer converts a spike into a longer, flatter processing period.
- Set a lower target utilization so scaling begins earlier, at the cost of running more replicas on average.
When an Endpoint Is the Wrong Shape
| Traffic shape | Better fit |
|---|---|
| Steady, high volume, latency-sensitive | Inference endpoint with tuned autoscaling |
| Very spiky, low average, tolerant of cold starts | Cloud Run — scales to zero, pay only when serving; or an endpoint enrolled in Scale To Zero if the caller retries the initial 429 |
| Known population, results tolerate staleness | Batch prediction — no serving infrastructure at all |
| Extremely high sustained throughput with custom routing or multi-container graphs | GKE — full control of the topology |
| Analytical consumption inside the warehouse | BigQuery ML in place |
The recurring cost trap is an endpoint serving a handful of requests per hour: the minimum replica bills continuously to answer almost nothing, and either Cloud Run or a batch job would cost a fraction.
The Latency Budget Is More Than the Model
When measured latency exceeds expectations, decompose the request before touching the endpoint configuration:
Total p95 = network + auth + feature lookup + preprocessing
+ model compute + postprocessing + response
Feature lookups are the usual surprise. A model that infers in 8 ms but waits 60 ms for an online feature read is not a serving-capacity problem, and adding replicas will not fix it. The remedies are on the feature side: co-locate the online store, batch the lookups, cache hot entities, or reduce the number of features fetched per request.
Exam Traps
- Assuming a normal endpoint scales to zero. A standard deployment has a floor of one replica; scaling to zero requires explicit enrolment in Scale To Zero and costs the first request a 429.
- Adding replicas when single-request latency already breaches the budget. Scale up or compress instead.
- Sizing capacity from assumption instead of a load test with realistic payloads.
- Ignoring cold start for large models during spikes.
- Blaming the endpoint when the feature lookup dominates the budget.
An endpoint meets its 80 ms p95 budget at 50 requests per second but degrades badly at 400. At one request per second, p95 is 22 ms. What is the correct scaling response?
A model receives roughly 40 requests per hour, arriving unpredictably, and callers tolerate an occasional slower first response. It currently runs on an inference endpoint with one minimum replica, and cost is under scrutiny. What is the better fit?
An endpoint's p95 latency is 78 ms against a 60 ms budget. Tracing shows model compute at 9 ms and an online feature lookup at 61 ms. What is the correct remediation?
A team is preparing for a product launch expected to drive a tenfold traffic spike within minutes of the announcement. The model is large and takes noticeable time to load into a new replica. What should they do?