14.2 ML System Deployment Testing

Deploying a machine learning system (MLS) is not copy the checkpoint to a bucket. Learning objective AI-7.1.2 (K2) asks you to explain the forms of MLS deployment testing that check whether the AI-based system functions correctly and reliably in its target environment. The syllabus names three environment families you should be able to talk about on the exam: cloud, edge, and mobile. Each test type below addresses a different deployment risk. Learn what each type does, what it does not do, and how it looks in all three environments.

Target environments are part of the story

Cloud deployments usually mean virtual machines or containers with optional GPUs, managed Kubernetes, and a model served behind an HTTP or gRPC API. Failures look like driver mismatches, autoscaling cold starts, and regional image tags that silently differ.

Edge deployments put inference next to a sensor or actuator: a camera on a production line, a gateway in a substation, a robot controller. Failures look like a converted graph that does not fit memory, an accelerator runtime that only exists on one board revision, or an install script that cannot run offline.

Mobile deployments ship a model inside an application on phones and tablets, often after converting from a training format into a mobile runtime. Failures look like a neural processing unit (NPU) delegate that falls back to CPU, a permission prompt that blocks camera input, or an app-store build that still links a debug library.

Every test type below should be planned against the environments you actually ship, not against a single laboratory notebook.

Installability testing

Installability testing verifies that the MLS can be installed, configured, and later uninstalled across supported environments. Include system dependencies such as GPU drivers, compatibility with frameworks, and successful execution of installation scripts.

Cloud installability

A new GPU node image claims CUDA 12 compatibility. Installability testers run the install script, confirm the framework can see the device, configure the serving replica, then uninstall and confirm leftover kernel modules do not break the next image. A green accuracy test on a laptop does not replace this.

Edge installability

A fleet of 200 inspection cameras receives a Debian package. Testers install on a golden board, configure the camera pipeline, reboot, and uninstall without orphaning the previous runtime. Offline plants cannot fetch packages at runtime; scripts must be complete.

Mobile installability

Installing the app is only half of installability. Testers also configure model-download flags, verify the on-device runtime matches the shipped mobile model package, and uninstall to ensure cached models do not remain and later mix with a new app version.

Exam cue: installability is the type in this list whose definition explicitly includes uninstall.

Rollback testing

Rollback testing verifies that the system can revert to a previously stable, operational state after a degraded or failed deployment. The rollback scope may cover the model only or the complete system, including the data pipeline.

Critical exam rule: rollback testing must be performed before production deployment to confirm rollback readiness. You do not wait for a Friday-night incident to discover that roll back is a slide in a runbook with no rehearsed procedure.

Cloud rollback

Deploy model B to staging with the new feature-store mapping, then execute rollback to model A and the previous mapping. If only the weights revert while the pipeline still emits B's feature names, rollback is incomplete.

Edge rollback

A failed quantized model bricks a line-side box. Rollback may be swapping the model file, or reflashing the box so the previous inference engine and preprocessing binary return together.

Mobile rollback

Staged app releases often couple a model file to application code. Rollback testing includes whether store rollback or an in-app model pin actually restores the last good pair.

If you only test happy-path deploy, you have not tested rollback.

Canary testing versus shadow testing — the exam trap

These two types are the pair most commonly swapped on scenario items. Separate them by whether live users receive the new answers.

Canary testing validates a new deployment by releasing an updated model to a small subset of production traffic, for example 5 percent of users. Those users do receive the new model's responses. Testers monitor real-time metrics such as latency, accuracy, and error rates before a full rollout. The business accepts a bounded blast radius in exchange for evidence on real traffic.

Cloud canary. Five percent of recommendation API calls route to the new replica. Watch tail latency, HTTP 5xx rate, and an online proxy for accuracy such as click-through on the canary slice versus control. If the canary is sick, you halt the rollout.

Edge canary. You cannot always split a single robot. You can still canary a subset of devices in the fleet—ten cameras on one line—while the rest stay on the old model.

Mobile canary. A remote-config flag enables the new model for 5 percent of app sessions.

Shadow testing runs the new model in parallel with the current production model, routing the same requests to both, and it does not affect live responses. Users keep getting the current model's answers. Testers compare the two on live data in a controlled, low-risk setup. Shadow testing can reveal performance regressions and data drift before a full deploy—without changing what customers see.

Cloud shadow. A sidecar scores every request; only the old model's output is returned. You measure disagreement rate and latency of the shadow path. Provision extra capacity: shadow load can still hurt the box.

Edge shadow. A gateway duplicates frames to a second interpreter. Actuators still follow the production model.

Mobile shadow. The app may compute a shadow score for telemetry, but the user interface still displays the production model's result.

Trap: canary changes answers for a slice of real users; shadow never changes the user-visible answer. Canary is a partial production release. Shadow is a parallel observer. Do not say shadow slowly ramps traffic to 5 percent. That ramp is canary language.

Model conversion testing

Training formats such as PyTorch checkpoints or SavedModel graphs are often converted into deployment formats such as ONNX Runtime, TensorRT, TensorFlow Lite, or Core ML. Model conversion testing verifies that after conversion the model retains acceptable predictive accuracy, consistent behavior, and operational efficiency such as inference speed and memory usage.

Worked checks. On a frozen evaluation set of 2,000 examples, the training graph scores 0.872 ROC AUC; the INT8 TensorRT engine scores 0.869. That delta may be acceptable under a written threshold. If AUC falls to 0.71, conversion destroyed ranking quality. Testers also compare per-example outputs for consistency (same argmax class, bounded score drift), measure milliseconds per request, and record resident memory on the target hardware—not on the training workstation.

Cloud conversion

Compare a GPU engine against the Python training path on the same evaluation slice, including batch-size effects that only appear under production load.

Edge conversion

A compiler that fuses operators can change numerics; memory must fit the board, including peak allocation during graph initialization, not only steady-state inference.

Mobile conversion

An NPU delegate versus CPU fallback matters. A converted model that silently runs on CPU can pass accuracy and still fail the speed and battery budget.

Conversion testing is not a substitute for canary or shadow. It answers whether the artifact still behaves after the format change. Canary and shadow answer how that artifact behaves on live traffic.

Cross-device testing

Cross-device testing verifies that the MLS performs correctly across its intended range of deployment targets: mobile devices, edge devices, and cloud servers. It is not a single golden laptop. Numeric differences, thread counts, accelerators, and operating-system versions all change behavior.

Plan a matrix: two phone chipsets, two edge board revisions, and the cloud GPU SKU you actually buy. A model that passes conversion testing on one device class can still fail on another—different fused-multiply-add rounding, different thread pools, different camera preprocessing libraries.

Cloud. Confirm the regional GPU SKU matches what staging used. A cheaper inference instance with no Tensor cores can change both latency and, for some kernels, numeric outputs.

Edge. Board revision B may ship a different accelerator firmware than revision A. Cross-device tests catch a model that fits A's memory map and OOMs on B.

Mobile. Test at least one high-end NPU device and one low-end CPU-only device if both are in the support matrix. Permission and thermal throttling are device stories, not notebook stories.

API testing

API testing verifies that the MLS exposes well-defined, standards-compliant interfaces. Validate inputs and outputs, error messages, and integration workflows with data feeds, clients, and the pipeline.

Cloud. Schema checks for tensor shapes, authentication failures, and back-pressure when the feature store times out. A 200 OK with a silently truncated embedding is still an API defect.

Edge. A gRPC camera API must reject an unsupported resolution with a clear error rather than a native crash that takes down the box.

Mobile. Exercise the on-device API between application code and the interpreter, including what happens when the model file is missing, corrupted, or the wrong version for the app binary.

API testing here is a deployment activity: the live contract, not the research notebook. It is also the mitigation named in the previous section when engineers misuse framework APIs, but in this section the object under test is the MLS interface that clients and pipelines call after the model is shipped.

Putting the types together on one release

A realistic cloud release might install a new serving image (installability), convert the checkpoint (conversion), confirm phones and GPU servers still score coherently (cross-device), prove the REST contract (API), run the candidate as a shadow on live traffic (shadow), then expose 5 percent of users (canary), with a rehearsed rollback (rollback) already passing in pre-production. Edge and mobile releases use the same types with different blast-radius mechanics. Name the type by the risk it addresses, not by the vendor's feature name.

Remember the order trap on rollback: prove rollback in a controlled environment before you deploy to production. Remember the pair trap on canary versus shadow: canary changes a slice of live answers; shadow does not.