3.1 Hardware for Machine Learning Systems

Hardware choices testers have to compare

Machine learning systems run on processors, memory buses, and power budgets, not only on source files. CT-AI v2.0 learning objective AI-1.1.5 (K2) asks you to compare hardware used to implement machine learning systems. That comparison is a testing skill. The chip you train on, the chip you serve on, and the numeric format you ship can each change latency, energy use, and the score you record.

Training hardware and inference hardware are allowed to differ, and in real products they often should. A speech-recognition model may infer on a low-end smartphone while training required the power of cloud computing. The user-facing system under test is the phone path. The training cluster is a different system under test. If you only measure the cluster, you have not measured the product the customer holds.

What ML workloads demand from a chip

Conventional business software spends time on branching, databases, and sequential workflows. Neural networks and many other ML models spend time moving large tensors and multiplying those tensors together. Matrix multiplication is the emblematic parallel job: huge numbers of multiplies and adds can proceed at once if the hardware has enough cores and enough memory bandwidth.

ML therefore benefits from hardware that supports three traits:

• The ability to work with large data structures. Weights, activations, and batches of examples are bulky. A chip that cannot hold or stream them spends its time stalled on memory rather than computing.
• Massively parallel concurrent processing, for example to support matrix multiplication. Throughput comes from thousands of simple arithmetic units working together, not from one core thinking in a clever sequential way.
• Low-precision arithmetic, often called quantization. Many ML tasks do not need 32-bit precision on every operation. Using fewer bits, for example 4-bit instead of 32-bit, typically yields faster processing, lower power consumption, smaller and more cost-effective chips, and reduced bandwidth.

Quantization is a quality trade, not a free upgrade. Testers inherit the trade as a risk: the reduced-precision artifact must still meet the agreed error budget. A 4-bit on-device build that is fast and cheap is still a failing product if it drops below the accepted recall or error threshold.

CPU versus GPU, and why clock speed misleads

A general-purpose CPU supports complex operations at high precision. Those capabilities matter for operating systems and irregular control flow, but they are not typically required for ML arithmetic. CPUs also typically provide only a few cores. The combination of few cores, high precision, and a higher clock is less efficient for training and running ML models than a GPU.

A GPU exposes thousands of cores designed for massively parallel, relatively simple operations that originated in graphics. That pattern maps onto tensor math. GPUs typically outperform CPUs on ML even though CPU cores usually run at higher clock speeds. For small-scale ML work, GPUs generally offer the best option: a workstation GPU or a single cloud GPU is enough to train modest networks and to serve moderate inference loads.

When you read a datasheet, do not rank ML hardware by peak GHz. Parallel width, memory bandwidth, and supported numeric types decide whether a training job finishes overnight and whether an inference call meets a tight latency budget. A tester who compares a 3.5 GHz CPU with a 1.5 GHz GPU as if clock speed were an ML scoreboard is measuring the wrong thing.

ASICs, SoCs, and the edge-versus-cloud split

Some hardware is designed specifically for AI. Purpose-built application-specific integrated circuits (ASICs) and system-on-a-chip (SoC) devices typically feature multiple cores, specialized data management, and the ability to perform in-memory processing. They are best suited to edge computing, while training of the ML model is performed in the cloud using specialized hardware.

The split is deliberate. Cloud training hardware is optimized for throughput on large batches. Edge inference hardware is optimized for power, physical size, and often a batch size of one. A factory camera, a wearable, or an in-vehicle box may need to answer without a round trip to a datacenter. Edge inference can improve privacy because raw sensor data need not leave the device. It also introduces thermal throttling, tiny memory ceilings, and vendor-specific numeric modes that a roomy cloud GPU notebook will never show.

As a tester, you cannot certify an edge ASIC deployment by exercising only a 32-bit GPU workstation. You need tests that run on, or honestly represent, the edge numeric format, memory limits, thermal behavior, and single-example traffic that production will see.

Neuromorphic processors

AI-specific architectures continue to be developed. Neuromorphic processors are the contrast case you must be able to recognize. They do not use the traditional von Neumann architecture, in which a program and data live in memory and a control unit fetches instructions. They use brain-inspired designs that mimic neuronal structures, often with local memory and event-driven computation. You are not expected to design one. You are expected to know that they are not von Neumann machines. A laboratory that only has CPUs and GPUs does not automatically represent neuromorphic timing, energy, or failure modes.

What hardware changes in the test plan

Hardware choice is not a procurement footnote. It changes four testing concerns.

Latency. A cloud GPU, a laptop CPU, a phone, and an ASIC can differ by orders of magnitude on the same model. Voice interfaces, robotics, and assistive products live or die on those milliseconds. Your performance tests must name the hardware class, not only the model file.

Energy. Quantization and edge silicon exist partly because battery life and thermal design are quality characteristics. A model that is accurate in the lab but drains the device has failed a real acceptance criterion.

Numerical non-determinism. Parallel reductions, mixed precision, and different math libraries can change bit-level results. Re-running the same evaluation on a different GPU SKU can move a metric. Record the hardware, driver, and numeric mode with the result. Treat unexplained drift as something to investigate, not as noise to ignore.

Where tests must run. Train-time checks belong on training-class hardware. Inference acceptance belongs on production-class hardware, including the smartphone or SoC you will ship. If operations train in the cloud and ship a quantized SoC build, your regression set must include that SoC path. Otherwise you are testing a cousin of the product.

Use the table as a comparison sheet, not as a shopping list. The K2 skill is to match a hardware class to a job (train versus infer, cloud versus edge) and to predict what that match does to latency, energy, numeric stability, and the place where tests have to run.