8.3 Test, Troubleshoot, Progressive Rollout, and Safe Rollback

Test, Troubleshoot, Progressive Rollout, and Safe Rollback

Quick Answer: Prove a new green deployment locally and with zero live traffic, then mirror (shadow) production requests, then shift live traffic (for example 90/10), then 0/100. Rollback is a traffic shift back to blue. Do not delete the old deployment until green is healthy. Debug with local endpoints, the inference HTTP server, container logs (inference server and storage-initializer), Azure Monitor, and Application Insights.

Exam AI-300 Domain 2 clusters test and troubleshoot endpoints with progressive rollout and safe rollback because they are one operate loop. Shipping a new model version is not az ml online-deployment create --all-traffic in production.

Test before any live traffic

Work inward from the cheapest failure:

1. Azure Machine Learning inference HTTP server (azureml-inference-server-http) — run score.py locally without Docker. Syntax errors, missing imports, and init() crashes show up here first, isolated from container configuration.
2. Local endpoint — CLI --local or SDK local=True builds a Docker image that mimics the cloud. You need Docker Engine. Limitations: one deployment per endpoint, no traffic rules, no authentication, no probe settings; models and Conda files must be local (download registered assets first). Studio and Azure Resource Manager templates cannot deploy locally. Visual Studio Code dev containers can attach a debugger inside that Docker image (preview).
3. Cloud deployment at 0 percent traffic — create green without --all-traffic. Invoke it with az ml online-endpoint invoke --deployment-name green or HTTP header azureml-model-deployment: green. Studio's Test tab requires a deployment name; unlike CLI/SDK, studio will not follow the traffic map on the Test tab.

--all-traffic on create is a development convenience that points 100 percent at the new deployment. Do not use it in production rollouts. If you invoke the endpoint without a deployment name, routing follows the live traffic map.

Batch testing is different: invoke with --deployment-name against the non-default deployment. There is no local batch endpoint and no traffic mirror. Confirm blob outputs and job logs, then switch defaults.deployment_name.

Logs, metrics, and Application Insights

You cannot SSH to the managed virtual machine. You can pull two container streams:

• Inference server logs — print and logging from score.py (az ml online-deployment get-logs / SDK get_logs). Use a real logging level such as INFO or the messages never appear.
• Storage-initializer logs — --container storage-initializer (SDK container_type="storage-initializer"). Tells you whether model and code downloaded before the inference server starts.

If init() throws, you often get ResourceNotReady and empty inference logs — check storage-initializer, Conda imports, and probe timeouts. Increase probe failure thresholds or pick a larger SKU if the container is healthy but slow to start. ImageBuildFailure lives in workspace storage build logs (unpinned pip dependencies, Azure Container Registry authorization, or missing image-build compute on a private virtual network). OutOfQuota is CPU, disk, memory, role-assignment limits, endpoint-count, or regional VM capacity — remember the 20 percent upgrade reservation from section 8.1. SecretsInjectionError means the endpoint identity cannot read the referenced workspace connection or key vault, or the secret name is wrong.

Azure Monitor charts availability, latency, and request rate per deployment, which is why blue-green comparison is possible. Enable Application Insights diagnostics on the deployment (studio toggle; still documented as supported, including a legacy path) to inspect user-container events. Bandwidth throttling (default 5 MBps per endpoint) shows up as delay trailers (ms-azureml-bandwidth-request-delay-ms). Mirroring counts against that cap — that is why mirror traffic is capped.

Useful HTTP codes when you score managed online endpoints:

Treat 424 as a problem in score.py, the model, or probes — not as a generic Azure outage. Send x-ms-client-request-id (alphanumeric, hyphen, underscore; truncated at 40 characters) for your traces. x-request-id is reserved and overwritten by the service; attach it to a support ticket.

v2 online endpoints do not natively support Cross-Origin Resource Sharing (CORS). Browser apps need a gateway or Azure Functions in front. That is an operate fact, not a reason to pick batch.

Progressive rollout on online endpoints

Microsoft's documented blue-green path:

1. Endpoint exists; blue has 100 percent live traffic.
2. Scale blue (instance_count or autoscale) if load requires it before you add risk.
3. Create green with 0 percent live traffic.
4. Invoke green by name (CLI, SDK, header, or studio Test).
5. Mirror a percentage of live traffic to green (shadow / dark launch).
6. Shift live traffic, for example blue=90, green=10.
7. Shift green=100, blue=0 when metrics look good.
8. Only then delete blue.

Mirror rules that show up on items:

• Maximum 50 percent mirror
• One shadow deployment per endpoint
• A deployment receives either live or mirrored traffic, not both
• Clients still receive blue's responses; green's output is discarded but metrics and logs are recorded
• Invokes that name a deployment are not mirrored
• Kubernetes online endpoints do not support mirroring
• CLI older than 2.4.0 or SDK older than 1.0.0 can drop the mirror setting on update
• Live percentages must total 0 or 100

CLI sketches:

az ml online-endpoint update --name fraud-online --mirror-traffic "green=10"
az ml online-endpoint update --name fraud-online --mirror-traffic "green=0"
az ml online-endpoint update --name fraud-online --traffic "blue=90 green=10"
az ml online-endpoint update --name fraud-online --traffic "blue=0 green=100"

Rollback is the inverse traffic command, not a delete:

az ml online-endpoint update --name fraud-online --traffic "blue=100 green=0"

The scoring URI and credentials stay the same. Deleting green, deleting blue, or deleting the endpoint is not rollback. Studio refuses to delete a deployment that still has live traffic — set that deployment to 0 percent first. After green has taken 100 percent long enough that you trust it, az ml online-deployment delete --name blue removes unused VMs. Deleting the endpoint deletes every deployment under it.

Batch "rollout" is switching defaults.deployment_name after a named invoke. There is no 90/10 and no mirror. Keep the previous deployment until the new default's jobs look clean.

Putting the operate loop together

A durable MLOps habit for online endpoints:

• Register model version N+1 (Chapter 7) rather than overwriting files in place
• Deploy it as a new named deployment, not by mutating blue in place
• Compare deployments in Azure Monitor (latency, error rate, 424s) before shifting live traffic
• Keep blue running through the canary so rollback is one traffic update
• Only after green is the sole live deployment do you delete blue to stop paying for its VMs

That is why the endpoint/deployment split in section 8.1 exists. If you had only one object, every model bump would change the URL or require a destructive replace.

Exam scenario

blue serves 100 percent of checkout scoring on fraud-xgb:3. Version 4 is registered. Create green at 0 percent, run the inference HTTP server and a local Docker endpoint against score.py, then cloud-invoke with azureml-model-deployment: green. Mirror 10 percent; compare latency and 424 rates in Monitor. Move live traffic to 90/10, then 0/100. If p99 doubles, set traffic back to blue=100 before deleting anything. After a week of green at 100 percent, delete blue. Do not apply this traffic map to a batch endpoint — switch that product's default instead.

Common trap

Deleting blue as soon as green is created, or assuming mirror traffic changes what customers see. Mirror is observation only. Another trap: applying online traffic split to a batch endpoint, or enabling mirror on Kubernetes. A third: treating HTTP 424 as a gateway outage (500) instead of a model/container failure. A fourth: using --all-traffic on the first production create of green, which skips the canary entirely.