14.3 Serving Hardware & Autoscaling for Throughput
Key Takeaways
- By default, Agent Platform autoscales online inference so CPU usage, or the higher of CPU and GPU usage on GPU deployments, matches a 60% target.
- Autoscaling can also use request count per minute, DCGM GPU utilization, vLLM KV-cache usage, vLLM waiting requests, or Pub/Sub queue size.
- Replica targets are computed as ceil(current replicas × current utilization / target), evaluated every 15 seconds against the highest target from the last 5 minutes.
- Scale To Zero (preview) lets minReplicaCount be 0 on endpoint types other than shared public; requests to a scaled-down model get a 429 while it scales up.
- Model co-hosting in a DeploymentResourcePool shares one VM across several sparse-traffic models, but the models aren't isolated from each other.
The exam guide lists choosing appropriate hardware (for example, CPU, GPU, TPU, and edge) and scaling the serving backend based on throughput (for example, Agent Platform Inference and containerized serving).
Choosing Inference Hardware
| Hardware | Best for | Examples |
|---|---|---|
| CPU | Tree ensembles, linear models, small neural networks, low-to-moderate QPS, latency tolerance in tens of ms | XGBoost fraud scoring, scikit-learn pipelines |
| GPU | Deep networks with heavy matrix math, large batches, vision and speech models, LLM serving | NVIDIA L4 for cost-efficient inference, A100, H100, and H200 for large LLMs |
| TPU | Large TensorFlow, JAX, or PyTorch/XLA models at high throughput | Agent Platform supports single-host Cloud TPU v5e, v6e, and TPU7x for online inference (multi-host in preview). Use the optimized TensorFlow runtime (2.15+) or PyTorch TPU container (2.1+) |
| Edge | Offline or intermittent connectivity, on-device latency, data that must stay local | AutoML Edge exports (TF Lite, Edge TPU TF Lite, Core ML, TensorFlow.js, container). Agent Platform on Google Distributed Cloud for on-premises or disconnected sites |
Right-size by measurement. Load-test candidate machine types at target QPS, and compare cost per 1,000 predictions at your p95 latency target. A GPU replica that serves 20× a CPU replica's throughput can be cheaper overall.
How Agent Platform Autoscaling Works
Set dedicatedResources.minReplicaCount and maxReplicaCount. If max is higher than min, the deployment autoscales.
Default targets
- CPU-only deployments: scale so CPU utilization ≈ 60%.
- GPU deployments: scale so the higher of CPU or GPU utilization ≈ 60%. A container with a busy unrelated CPU process can trigger unnecessary scale-up. If you configure scaling on CPU only, it won't scale up for high GPU usage.
Custom metrics (autoscalingMetricSpecs)
| Metric | Unit | Default |
|---|---|---|
| CPU utilization | % per replica | 60% target |
| GPU duty cycle | % per replica | 60% target |
| DCGM GPU utilization (preview) | % per replica | Disabled |
| vLLM GPU KV-cache usage (preview) | % per replica | Disabled |
| vLLM requests waiting (preview) | Requests per replica | Disabled |
| Request count | Requests per minute per replica | Disabled |
| Pub/Sub undelivered messages (preview) | Messages per replica | Disabled |
For LLM serving, KV-cache usage or waiting requests reflect load better than CPU. For uniform, predictable requests, request count per replica gives a direct throughput target.
The formula and timing
Target replicas = ceil(current replicas × current utilization ÷ target utilization)
- 2 replicas at 100% with a 60% target → ceil(2 × 100/60) = ceil(3.33) = 4
- 10 replicas at 1% → ceil(10 × 1/60) = ceil(0.167) = 1
Every 15 seconds, the system sets replicas to the highest target from the previous 5 minutes. Scale-up happens quickly, and scale-down waits until a 5-minute window stays low. New replicas still need time to provision VMs, pull the container, and load the model, so set minimum replicas to absorb sudden bursts.
Deployment Options That Affect Scaling
| Option | Behavior |
|---|---|
| minReplicaCount ≥ 1 (standard) | Always-on capacity. The deployment succeeds only when min replicas are ready, unless you set requiredReplicaCount |
| requiredReplicaCount | The deployment counts as successful once this lower number of replicas is ready. The rest keep provisioning |
| Scale To Zero (preview) | minReplicaCount = 0. Not available on shared public endpoints. Requests while scaled down get 429 "Model is not yet ready" and trigger scale-up to initial_replica_count (default 1). idle_scaledown_period and min_scaleup_period default to 1 hour (range 5 minutes to 8 hours) |
| Co-hosting (DeploymentResourcePool) | Several models share one VM's resources. Good for many models with sparse traffic. Models aren't isolated and can compete for CPU and memory. Up to 20 concurrent deployment requests. An empty pool uses no quota |
| Spot VMs | Cheaper inference capacity that can be preempted. Suitable for fault-tolerant or non-critical serving |
| Reservations | Guaranteed accelerator capacity for predictable peaks |
Quota
Serving quota counts CPUs and GPUs used by active replicas. For example, each a2-highgpu-2g replica counts as 24 CPUs and 2 GPUs. If the sum of maxReplicaCount across deployments exceeds quota, some deployments may fail to autoscale during peaks. Request quota for expected peak, not average.
Containerized Serving Alternatives
| Platform | Scaling mechanism |
|---|---|
| Cloud Run | Request concurrency autoscaling, scale to zero, one GPU per instance |
| GKE | Horizontal Pod Autoscaler on CPU, GPU, or custom metrics such as queue depth. Cluster autoscaler for nodes. Inference Gateway for load-aware routing |
Monitoring the Serving Backend
Watch these endpoint metrics in Cloud Monitoring, with alerts:
- Replica count against
maxReplicaCount. Sitting at max means you need more capacity or quota. - Latency percentiles (p50, p95, p99) and error rates (429 and 5xx).
- CPU, GPU duty cycle, and memory per replica.
- For LLMs, KV-cache usage and queued requests.
Worked Scenario
A retailer's recommendation endpoint gets 50 QPS at night and 2,000 QPS during a daily 6 p.m. flash sale that starts instantly. Each L4 replica handles about 250 QPS at the latency target.
- Peak need: 2,000 / 250 = 8 replicas at 100%. At a 60% utilization target, about 14 replicas.
- Set maxReplicaCount to at least 14, and confirm GPU quota covers it.
- Because the spike is instant and new replicas take minutes to be ready, use a scheduled job to raise minReplicaCount before 6 p.m. and lower it afterward, instead of relying only on reactive autoscaling.
- At night, a minimum of 2 replicas keeps the service redundant.
An endpoint has 3 replicas running at 90% CPU with the default 60% target. How many replicas will autoscaling target?
An LLM served with vLLM on GPU replicas shows low CPU usage even when requests queue up, and the endpoint doesn't scale out. Which change best fixes the scaling signal?
A company has 60 small regional models, each receiving a few requests per hour. Deploying each to its own VM is expensive. What Agent Platform feature reduces cost while keeping online serving?