4.5 Choosing Serving Hardware: CPU, GPU, TPU and Edge
Key Takeaways
Serving hardware is chosen from latency at the required throughput and cost per request, not from what the model was trained on.
Most tabular and small models serve fastest and cheapest on CPU; accelerators only pay off when the model is large enough to keep them busy.
Batching raises accelerator throughput but adds queueing latency, so the batch window is a direct latency-versus-cost dial.
Edge deployment is chosen for offline operation, privacy, or network-independent latency, and forces compression through quantization, pruning, or distillation.
Serving accelerators should never run on Spot or preemptible capacity, because reclamation removes replicas from a live endpoint.
4.5 Choosing Serving Hardware: CPU, GPU, TPU and Edge
Blueprint reference: Section 4.2, "Choosing appropriate hardware (e.g., CPU, GPU, TPU, and edge)."
Training hardware selection was covered in Section 3.11. Serving is a different problem with a different cost function, and the most common error is assuming a model trained on GPUs must be served on GPUs.
Inference Economics Differ from Training
| Training | Serving | |
|---|---|---|
| Workload shape | One long job, high utilization by design | Continuous, often bursty, frequently underutilized |
| Batch size | Large, chosen for throughput | Often 1, dictated by the request |
| Cost driver | Job duration × device rate | Replica-hours × device rate, plus idle time |
| Optimization goal | Time to trained model | Latency at a target QPS, per-request cost |
| Interruptible? | Yes, with checkpointing | No — a reclaimed replica is a dropped request |
That last row is a hard rule: Spot and preemptible capacity is correct for training and wrong for a live endpoint, because reclamation removes serving capacity without warning.
When an Accelerator Pays Off at Inference
An accelerator only helps if the model is large enough to occupy it. A gradient boosted tree or a small MLP responds in microseconds on CPU, and putting it on a GPU adds transfer overhead and a much higher hourly rate for no latency gain — the utilization graph will show a device that is idle almost all the time.
| Model | Typical serving choice | Why |
|---|---|---|
| Tree ensembles, linear, small MLPs | CPU | Sub-millisecond on CPU; accelerator adds cost, not speed |
| Mid-size CNNs, embedding models, small transformers | CPU with enough cores, or an entry-level inference GPU | Depends on QPS; measure before deciding |
| Large vision models, large language models | GPU | Model size and compute make CPU serving impractical |
| Very large models at very high sustained throughput, XLA-compatible | TPU | Cost per unit throughput at scale |
| Anything on a device, offline, or privacy-bound | Edge | Network independence is the requirement |
The practical procedure: serve on CPU first, load-test to the required QPS, and move to an accelerator only when CPU cannot meet the latency budget at acceptable replica count. Many teams discover that ten CPU replicas cost less than one GPU replica and meet the target comfortably.
Batching: The Latency-Throughput Dial
Accelerators are efficient on batches and wasteful on single requests. Server-side dynamic batching collects arriving requests for a short window and runs them as one batch.
Batch window 0 ms → lowest latency, lowest accelerator efficiency, highest cost/request
Batch window 10 ms → modest added latency, substantially better throughput
Batch window 50 ms → high throughput, may breach an interactive latency budget
The window is a design parameter, not a default. Set it from the latency budget: if p95 must stay under 100 ms and inference itself takes 60 ms, a 10 ms batching window is affordable and a 50 ms one is not.
For offline scoring, batching is unambiguously right and the window can be as large as convenient — which is one reason batch prediction is so much cheaper per record than online serving.
Other Levers Before Buying Hardware
- Model compression. Quantization, pruning, and distillation (covered in Section 4.3) often move a model from "needs a GPU" to "fits comfortably on CPU."
- Optimized runtimes. Compiling the model for the target hardware frequently yields large speedups without changing the device.
- Right-sizing the machine type. Serving replicas are often over-provisioned on memory and under-provisioned on CPU, or vice versa; measure.
- Caching. If the same inputs recur, a cache in front of the model removes inference entirely for those requests.
Edge Deployment
Edge is chosen for one of three reasons, and the reason determines the constraints:
| Reason | Consequence |
|---|---|
| Offline operation | The model and all its features must live on the device; no server round trip |
| Privacy | Raw input never leaves the device; only results or aggregates may be transmitted |
| Latency independence | Network round trip removed entirely; predictable even on poor connectivity |
Edge imposes requirements that must be settled at design time:
- Size and memory budget. Handled with quantization (typically INT8), pruning, and distillation into a smaller student model.
- Runtime. A device-appropriate inference runtime, not the training framework.
- Feature availability. Every feature must be computable on-device; a server-side aggregate is not available offline.
- Update distribution. A mechanism to ship new model versions to a fleet, plus a way to know which version each device runs.
- Monitoring. Devices must report prediction statistics back when connectivity allows, or drift on the fleet is invisible.
Hybrid designs are common and often the best answer: a small on-device model handles the common case offline, and hard cases are escalated to a larger cloud model when connectivity is available.
Exam Traps
- Serving a small model on a GPU because it trained on one.
- Spot capacity for a production endpoint. Reclamation drops requests.
- A large batch window on an interactive latency budget.
- Edge deployment with a server-side feature. It will not exist offline.
- Buying hardware before trying compression or an optimized runtime.
A gradient boosted tree model with 400 trees was trained on a GPU instance and now serves 900 requests per second with a 40 ms p95 budget. An engineer proposes GPU-backed serving replicas. What is the better approach?
Deploy on GPU replicas, since the model was trained on GPU and the frameworks must match.
Load-test CPU replicas first, since tree ensembles infer in well under a millisecond on CPU and are very likely to meet the budget at far lower cost.
Deploy on TPU replicas, which offer better cost per unit of throughput at 900 requests per second.
Deploy on GPU replicas with a 50 ms dynamic batching window to maximize accelerator efficiency.
A team runs a large vision model on GPU-backed serving replicas and, to control cost, configures those replicas to use Spot capacity. What is the consequence?
Cost will fall but model accuracy will degrade because reclaimed replicas serve from stale weights.
Latency will increase during reclamation events but no requests will be lost, because the endpoint queues traffic until capacity returns.
Reclaimed replicas are removed from the serving pool without warning, dropping in-flight requests and reducing capacity; Spot is appropriate for training, not for live serving.
Spot capacity is unavailable for accelerator-backed serving, so the configuration will be rejected at deployment time.
An interactive assistant has a 100 ms p95 latency budget, and model inference alone takes 60 ms on the chosen accelerator. An engineer proposes a 50 ms dynamic batching window to improve throughput. What should be done instead?
Disable batching entirely, since any batching window is incompatible with an interactive budget.
Increase the latency budget to 150 ms, since batching efficiency is the dominant cost lever.
Accept the 50 ms window, since batching runs concurrently with inference and adds no wall-clock latency.
Use a much smaller batching window, around 10 ms, so that queueing plus inference stays inside the 100 ms budget.
A logistics company deploys a package-damage classifier to handheld scanners that operate in warehouses with no reliable connectivity, and images must not leave the device. Which design elements are required?
An on-device model compressed through quantization, pruning, or distillation, using only device-computable features, with a fleet update mechanism and deferred reporting of prediction statistics.
A batch prediction job triggered whenever a scanner docks and regains connectivity, scoring the day's accumulated images.
A Cloud Run service with scale-to-zero, called by the scanners whenever connectivity is briefly available.
A GPU-backed regional endpoint with Private Service Connect, so that image uploads stay on private network paths.
Sections you finish are checked off in the contents.