8.2 Deploy Models to Batch Endpoints

Deploy Models to Batch Endpoints

Quick Answer: A batch endpoint is a durable HTTPS URL that starts an asynchronous scoring job, not a real-time HTTP response. Inputs are files in storage or data assets; outputs land in blob. Routing is a default deployment switch, not a traffic split. Clusters can scale to zero. Auth is Microsoft Entra. You can deploy a model or a pipeline component. Parallelism is mini-batches of files.

Exam AI-300 Domain 2 splits batch from online because the product is different: jobs, files, and clusters versus always-on HTTP servers. The registered model from Chapter 7 can serve both; the operational contract is not the same.

Endpoint, default deployment, and invoke

The batch endpoint still gives you a stable URL and authentication. The deployment still holds the model (or pipeline), environment, and compute. The routing difference is the exam discriminator:

• Online: traffic percentages that must sum to 100 (or 0 to disable)
• Batch: one default deployment. invoke without a name hits the default. Pass --deployment-name (CLI) or deployment_name (SDK) to test a non-default deployment without flipping production.

Create a second Keras deployment, invoke it by name, confirm outputs, then az ml batch-endpoint update --set defaults.deployment_name=keras-dpl. Consumers keep the same URL. There is no traffic: map and no mirror traffic on batch endpoints.

Azure Machine Learning does not bill you for the endpoint or deployment objects. Organize as many as you need. Endpoints and deployments can share a cluster or use independent clusters. Endpoint names, like online names, must be unique in the Azure region because the name is in the URI.

SDK/CLI v2 types are BatchEndpoint plus ModelBatchDeployment or PipelineComponentBatchDeployment. Studio can create model deployments; Kubernetes batch targets need CLI or SDK.

When batch is the right shape

Use batch when:

• Inference is expensive or long-running
• Data is many files in a storage account or a registered data asset
• You do not need HTTP latency
• You want parallelization across cluster nodes
• You want to operationalize an entire pipeline graph, not only a single model

A nightly fraud-file drop of two million CSV rows is batch. A checkout-page risk call is online. Overcapacity is queued on batch (jobs wait for cluster nodes) versus throttled on online (HTTP 429).

Model deployments versus pipeline component deployments

Model deployment scores a registered model at scale. Required pieces:

• Model
• Compute cluster (azureml:<cluster-name>)
• Scoring script (optional for MLflow)
• Environment (optional for MLflow)

Reuse the same registered model you deployed online; batch runs it over files. If the model needs light pre- or post-processing, put it in the batch scoring script rather than standing up a second service.

Pipeline component deployment hosts a whole directed graph: preprocess, retrieve features, score, post-process. Specify the pipeline component plus cluster configuration. You can also promote an existing pipeline job into a component, but Microsoft calls explicit component authoring the Machine Learning Operations (MLOps) best practice so the graph is versioned independently of one ad-hoc job.

Curated environments are not supported for batch; author a custom environment. Include azureml-core and azureml-dataset-runtime[fuse] plus whatever the model needs. AutoML's generated scoring script is for online endpoints. Reusing it on batch is a documented failure mode — write a batch driver.

Scoring script: files, not HTTP bodies

Online run(raw_data) receives the HTTP payload. Batch run(mini_batch) receives a list of file paths. init() still loads the model once from AZUREML_MODEL_DIR. Return a pandas DataFrame or an array; each row corresponds to a successful file in that mini-batch.

Parallelism is file-level. A folder of 100 files with mini_batch_size: 10 yields 10 mini-batches, regardless of file size. Huge files need splitting; the runtime does not rebalance skewed file sizes. Tune these knobs together:

Override instance count, mini-batch size, retries, timeout, and error threshold per job (--mini-batch-size, --instance-count, or SDK params_override) without editing the deployment. That is how you give a 1-million-file run more nodes than a 10,000-file smoke test.

Outputs must go to a blob datastore. The default is the workspace blob store under a job GUID. You can set --output-path to azureml://datastores/<name>/paths/<folder>/ and a new output_file_name. Duplicate output paths fail the job — pick a unique file name.

Inputs can be Azure Machine Learning datastores, data assets, or storage URIs (uri_folder, uri_file). Model deployments take one data input; pipeline deployments can take a dictionary of named inputs.

Compute, scale-to-zero, autoscale, and cost

Batch runs on Azure Machine Learning compute clusters (AmlCompute) or Kubernetes. Create the cluster with min_instances: 0 and a max_instances cap. Nodes provision when the job starts and deallocate when it finishes. Queued jobs consume no compute. idle_time_before_scale_down still applies if you leave min_instances above zero; production batch clusters almost always start at zero.

Autoscale is based on job count, not CPU. That is the opposite of online. Cost equals nodes actually running during the job, capped by the cluster maximum. Idle min_instances: 0 clusters cost nothing. Creating the endpoint itself does not start VMs.

A typical cluster YAML:

$schema: https://azuremlschemas.azureedge.net/latest/amlCompute.schema.json
name: batch-cluster
type: amlcompute
size: STANDARD_DS3_v2
min_instances: 0
max_instances: 5
idle_time_before_scale_down: 120

Then the deployment points at compute: azureml:batch-cluster and sets resources.instance_count for each job's node request.

Low-priority and Spot compute

Batch (unlike online) can target a cluster with tier: low_priority. Online deployments have no low-priority option. After 31 March 2026 Azure Batch allocates those nodes as Spot virtual machines: surplus capacity, variable Spot price, capacity eviction, no service-level agreement. Skills measured still name low-priority compute; operate it as Spot.

Batch deployments reschedule at mini-batch granularity. Completed mini-batches are kept. There is no checkpoint inside a mini-batch, so eviction redoes the in-flight batch of files.

Constraints you should recite:

• The deployment is bound to the cluster; you cannot pick dedicated versus Spot per job
• If the entire cluster is preempted, or a single-node cluster loses its node, the job is canceled and must be resubmitted
• Frequent eviction plus startup time can cost more than dedicated VMs, especially when the Spot rate is close to pay-as-you-go
• Low-priority cores use a separate quota from dedicated cores
• You are billed for powered-on nodes, including nodes evicted before they finish work

Use Spot for flexible overnight scoring, not for a hard SLA.

Security

Auth is Microsoft Entra — a user principal or a service principal / managed identity. There is no key mode on batch. Jobs run as the invoker, so authorization follows whoever called invoke. Private networking is supported; a workspace managed virtual network needs the extra batch-endpoint configuration Microsoft documents. That is why the comparison table in section 8.1 flags additional configuration for batch on managed networks.

Exam scenario

Risk operations drops 50,000 parquet files into workspaceblobstore/paths/nightly/. Scoring takes about 40 seconds per file and does not need a synchronous reply. Create a compute cluster with min_instances: 0, deploy the registered MLflow model to a batch endpoint (no scoring script), set mini_batch_size and instance_count for file-level parallelism, and write predictions.csv with append_row. Invoke the endpoint; poll the job; read blob output. If a challenger pipeline is ready, add a pipeline component deployment, invoke it by name, then switch defaults.deployment_name.

Common trap

Putting traffic: blue=90, green=10 on a batch endpoint. Batch has no traffic split and no mirror. Another trap: expecting batch run() to parse JSON like online run(raw_data), or assuming a managed online deployment can scale to zero because the cluster min is 0. Online deployments are not clusters. A third trap: using a low-priority cluster for a latency-sensitive online service — online does not support low-priority VMs. A fourth: deploying AutoML's online scoring script as the batch driver.