3.11 Evaluating Compute and Accelerator Options: CPU, GPU and TPU

3.11 Evaluating Compute and Accelerator Options: CPU, GPU and TPU

Blueprint reference: Section 3.3, "Evaluation of compute and accelerator options (e.g., CPU, GPU, and TPU)."

Accelerator questions are usually decidable from two facts in the scenario: the model family and one constraint (framework, custom operations, budget, or latency). This section is about the selection decision; the internal architecture of TPU pods, the parallelism strategies that span multiple accelerators, and the tf.data optimization checklist are covered in Section 3.12, Distributed Training Strategies, Hardware Acceleration and Performance Optimization.

What Each Device Is For

The TPU disqualifier is the point. TPUs run graphs compiled by XLA, which requires the operations in the model to be supported and shapes to be largely static. A model with custom CUDA kernels, unusual control flow, or heavily dynamic shapes will either fail to compile or recompile constantly, and the answer is a GPU. When a scenario mentions a custom operation, a research architecture with dynamic shapes, or a PyTorch codebase built around GPU-specific extensions, GPU is the answer regardless of how large the model is.

Conversely, when a scenario describes training a large transformer on a very large dataset with a standard architecture in TensorFlow or JAX, and mentions cost per unit of throughput at scale, TPU is the intended answer.

Memory Is a Separate Question from Speed

Two independent questions:

1. Does it fit? Parameters, gradients, optimizer state, and activations must fit in device memory. Optimizer state for Adam is roughly twice the parameter count in addition to the parameters and gradients themselves, so a model's memory footprint during training is several times its parameter size.
2. How fast does it run? Throughput, given that it fits.

If the model does not fit on any single device, the answer is not a faster device — it is model parallelism or sharding, covered in the distributed training section. Choosing a device with more memory is the simpler fix when it is available.

GPU Selection Within the GPU Family

Not all GPU questions are GPU-versus-TPU. Within the family:

• Entry-level inference GPUs (for example L4-class) — cost-efficient serving, small-to-medium models, video and image inference.
• Training-class GPUs (A100/H100-class) — large models, high memory bandwidth, fast interconnect between devices in a node.
• Older generation GPUs (T4-class) — cheap inference and light training; noticeably slower on large models.

Match to the workload rather than reaching for the largest: a small model on a top-tier GPU wastes most of the device, and the utilization graph will show it.

Cost Levers

Spot capacity is the exam's favourite because it pairs with checkpointing: the discount is only realizable if reclamation costs minutes, not the entire run. Spot for training with checkpointing is correct; Spot for a latency-sensitive endpoint is not.

The Standing Rule: Feed the Accelerator

No accelerator choice helps a job whose bottleneck is input. The diagnostic signature is unmistakable — accelerator utilization low, host CPU near 100% — and the fixes are data-side: consolidate small files into large sharded records, co-locate data with compute, parallelize and prefetch reads, and move heavy per-example preprocessing offline.

An exam scenario that reports low GPU utilization alongside high CPU utilization is never asking you to buy a bigger GPU.

A Compact Decision Path

Is the model a tree ensemble, linear model, or small classical ML?      → CPU
Does the model use custom ops, dynamic shapes, or GPU-specific kernels? → GPU
Is it a large standard transformer/CNN in TF or JAX, trained at scale,
   where cost per unit throughput dominates?                            → TPU
Does the model not fit in one device's memory?                          → shard / model parallel
Is accelerator utilization low with host CPU saturated?                  → fix the input pipeline first
Is the job fault-tolerant and checkpointed?                              → use Spot capacity

Exam Traps

• TPU for a model with unsupported custom operations. It will not compile.
• A bigger accelerator for an input-bound job. Fix the data pipeline.
• Spot capacity for a production endpoint. Reclamation breaks serving.
• GPU for gradient boosting on modest tabular data. CPU is cheaper and often faster end to end.
• Confusing "does not fit" with "too slow." Memory and throughput are different problems.