8.1 Deploy Models to Managed Online Endpoints

Deploy Models to Managed Online Endpoints

Quick Answer: A managed online endpoint is a durable HTTPS URL plus authentication. A deployment is the model, environment, instance_type, and instance_count that actually score. Clients call the endpoint; Azure Machine Learning routes traffic to one or more deployments. Online scoring is synchronous, low-latency HTTP. Instances cannot scale to zero — you pay while they run. For production on managed endpoints, set auth_mode: aad_token (Microsoft Entra). MLflow models can deploy with no scoring script; custom models need init() and run().

Exam AI-300 Domain 2 asks you to deploy models as real-time (managed online) endpoints. Chapter 7 registered and versioned models. This section is the first production surface those models hit: a live HTTPS service that applications call request by request.

Endpoint versus deployment

Keep these two objects separate. The exam loves mixing them.

An endpoint is the contract your consumers keep:

• A stable scoring URI such as https://<endpoint-name>.<region>.inference.ml.azure.com/score
• An authentication mechanism (key, aml_token, or aad_token)
• Optional traffic and mirror_traffic maps (section 8.3)
• A Swagger document for the scoring contract

A deployment is the implementation behind that contract:

• A registered model (or an inline path used for demos)
• An environment (curated image plus Conda, or bring-your-own container)
• Optional code_configuration.scoring_script
• instance_type (the virtual machine SKU) and instance_count
• Request settings, probes, and the identity used to pull storage and the container registry

One endpoint can host several deployments. Alice can run a CPU TensorFlow model as blue while Bob tests a GPU PyTorch model as green under the same URL. Consumers do not change their scoring URI when you swap implementations. The endpoint also has a routing mechanism that can send a percentage of requests to each deployment.

Endpoint names must be unique in the Azure region, not just in the workspace. Two workspaces in eastus cannot both own fraud-score.

SDK/CLI v2 is the exam surface: az ml online-endpoint / az ml online-deployment, or ManagedOnlineEndpoint and ManagedOnlineDeployment in azure-ai-ml. Do not author Azure Container Instances or Azure Kubernetes Service (AKS) v1 webservices as the current path. Kubernetes online endpoints still exist if you attach your own cluster; they do not get traffic mirroring or Microsoft Entra aad_token. Managed online endpoints provision the VMs, patch the host OS, recover nodes, and expose Azure Monitor metrics and per-deployment cost. You pay for those VMs and for networking; there is no extra Azure Machine Learning surcharge on the SKU.

When online is the right shape

Use a managed online endpoint when:

• Latency must stay in the HTTP request/response (hundreds of milliseconds, not minutes)
• The payload fits in the HTTP body
• You need to scale the number of concurrent requests
• The model answers in a short wall-clock time

Do not use online for overnight scoring of a data lake. That is a batch endpoint (section 8.2). Do not confuse this with standard (serverless) deployments of catalog foundation models. Those consume no VM quota from your subscription, bill per token (plus a small per-minute endpoint fraction), do not host multiple deployments, do not support traffic split or mirroring, and do not let you customize the inference stack. Domain 3 covers Microsoft Foundry serverless APIs and managed-compute foundation models. On AI-300, if the item is a classic registered scikit-learn or MLflow model with a scoring script, the answer is a managed online endpoint, not a Foundry serverless API.

Authentication: key, aml_token, aad_token

Scoring is a data-plane operation. Create/update/delete of the endpoint is control plane and always uses a Microsoft Entra token against https://management.azure.com. Data-plane scoring uses one of three auth_mode values.

Microsoft's current guidance: for production workloads on managed online endpoints, use aad_token. Keys are fine for development. aad_token on a Kubernetes online endpoint is a trap — it is not supported.

Callers send Authorization: Bearer <key-or-token>. az ml online-endpoint get-credentials returns primaryKey / secondaryKey for key mode, or accessToken plus expiry for token modes. Data-plane Entra tokens must be issued for audience https://ml.azure.com, not management.azure.com. Mixing those audiences yields HTTP 401. Built-in AzureML Data Scientist includes both control-plane CRUD and score/action. A caller with only Contributor on a different resource cannot score an aad_token endpoint.

Scoring script, no-code, and bring-your-own container

Custom models need a Python scoring file with:

• init() — runs once when the container starts; load the model from the AZUREML_MODEL_DIR environment variable
• run(raw_data) — runs on every invoke; parse the HTTP body and return JSON-serializable output

No-code deployments skip that file for MLflow and Triton models: Azure Machine Learning autogenerates the inference server. Automated machine learning (AutoML) models also ship a scoring script you can deploy as-is. Low-code means you bring the script plus a curated image. Include the azureml-inference-server-http package in a custom Conda file or the container will not serve. Bring Your Own Container (BYOC) puts the whole stack in a custom image and still rides managed autoscale, logs, and rollout.

If you later delete the registered model or the environment image a deployment still references, reimaging during a host OS patch can fail. Update the deployment before you delete those assets. Microsoft patches base images for known vulnerabilities; you must redeploy to pick up a patched image. If you brought your own image, you own the patch cycle.

Compute, quota, autoscale, and cost

You choose instance_type from the managed online SKU list (CPU or GPU) and instance_count. For high availability Microsoft recommends at least three instances. For many SKUs the service reserves 20 percent extra VM quota for upgrades and recovery: requesting 10 Standard_DS3_v2 (4 cores) needs quota for ceil(1.2 * 10) * 4 = 48 cores. Some SKUs are exempt; check the SKU list. That reserved capacity is not billed unless the upgrade actually runs.

Autoscale is Azure Monitor autoscale on resource use — CPU, memory, request rate — or a schedule such as peak business hours. It is not the batch job-count scaler. Managed online deployments cannot scale to zero. Idle instances still bill. That is the number-one cost surprise versus batch.

A shared quota pool exists only for short testing of selected catalog models (Llama, Phi, Mistral, and similar) from studio, and only on Enterprise Agreement subscriptions. Do not plan production capacity on shared quota.

A workspace managed virtual network can isolate inbound scoring (workspace private endpoint) from outbound pulls (managed VNet private endpoints). Expect extra private-link and fully qualified domain name outbound charges. Customer-managed keys are supported on online and batch, not on standard serverless deployments.

YAML shape (CLI v2)

A minimal endpoint:

$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineEndpoint.schema.json
name: fraud-online
auth_mode: aad_token

A minimal deployment:

$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineDeployment.schema.json
name: blue
endpoint_name: fraud-online
model: azureml:fraud-xgb:3
code_configuration:
  code: onlinescoring
  scoring_script: score.py
environment: azureml:fraud-env:2
instance_type: Standard_DS3_v2
instance_count: 3

Production practice: register the model and environment, then reference azureml:<name>:<version>. Inline path: uploads work for demos and local Docker tests. Local endpoints (--local / local=True) need Docker Engine, support one deployment, and do not implement traffic rules or auth. Studio and Azure Resource Manager templates cannot deploy locally.

Create the endpoint, then the deployment. --all-traffic on az ml online-deployment create is a development convenience that points 100 percent of traffic at the new deployment; production rollouts set traffic explicitly (section 8.3).

Exam scenario

A payments team registered fraud-xgb:3 as an MLflow model. The API gateway needs p99 latency under 200 ms and will send a small JSON body. The security baseline forbids long-lived keys in production. Deploy a managed online endpoint with auth_mode: aad_token, a CPU or GPU SKU that meets latency, instance_count of at least 3, and either no-code MLflow or a reviewed score.py. Do not pick a batch endpoint, and do not pick a Foundry serverless deployment unless the item is explicitly a catalog foundation model.

Common trap

Candidates mix online autoscale with batch scale-to-zero, or they pick key for a production managed endpoint because tokens expire. Tokens are supposed to expire. Another trap: treating standard/serverless foundation-model deployments as the way to host every registered workspace model. Serverless is quota-less and per-token, but only for catalog models that support it — Domain 3, not this Domain 2 bullet. A third trap: putting aad_token on a Kubernetes online endpoint.