2.3 Create and Manage Compute Targets

Create and Manage Compute Targets

Quick Answer: Instance = personal workstation (one owner, 120 GB OS disk, idle shutdown; stopped still bills disk/IP/load balancer). Cluster = named autoscaling nodes (min 0). Serverless = omit the compute name and let Azure Machine Learning create capacity. Managed means instance, cluster, and serverless; Kubernetes and remote VMs are unmanaged attaches.

A compute target is the machine, or pool of machines, that runs a training script or hosts a deployment. The point of the abstraction is that you can change where the job runs without rewriting the training code. Attach compute to a workspace; everyone in that workspace can submit jobs to shared targets. A compute instance is the exception to “shared hardware”: it has one owner, even though notebook files on the workspace file share are visible to other instances.

AI-300 Domain 1 asks you to create and manage these targets, not to memorize every retired NV-series SKU. Know the managed versus unmanaged split, when to use instance versus cluster versus serverless, how billing behaves when something looks “off,” and which targets are valid for training versus inference.

Managed versus unmanaged

Managed compute is created and patched by Azure Machine Learning. Only three kinds qualify:

• Compute instance
• Compute cluster (AmlCompute)
• Serverless compute

You create instances and clusters from studio, SDK v2, CLI v2 (az ml compute create), or ARM/Bicep. You do not create serverless compute as a persistent resource; you use it by omitting the compute name on a job (or setting pipeline default_compute to serverless / azureml:serverless).

Unmanaged compute is created outside Azure Machine Learning and then attached: remote virtual machines, Azure HDInsight, Azure Databricks, Azure Data Lake Analytics, and Kubernetes. You own patching, scaling, and often the identity story. Attach Kubernetes when inference or training must land on an existing on-premises, cloud, or edge cluster.

Apache Spark pools can appear as a training target (preview, with limits for AutoML). Do not invent Spark as a replacement for a compute cluster on a standard tabular command job unless the stem is about Spark.

Training targets versus inference targets

Training (SDK/CLI v2) commonly uses:

Inference (production) is a shorter list:

• Managed online endpoints and batch endpoints — Azure Machine Learning hosts the container on managed (serverless) compute. GPU is supported. This is the default production answer for Domain 2.
• Azure Machine Learning Kubernetes — real-time or batch inference on your cluster.

A compute instance can host local test inference for debug. It is not the production online-endpoint answer. When you size inference VMs, Microsoft’s rule of thumb is scale up first (about 150 percent of the RAM the model needs), then scale out for concurrency.

Hub and project workspaces do not support compute clusters. On a hub, training jobs that would have used a cluster should use serverless compute. That limitation is a favorite distractor if the stem already established a Foundry/hub layout in section 2.1.

Compute instance: the personal workstation

A compute instance is a fully managed Ubuntu workstation with Jupyter, JupyterLab, VS Code (including VS Code for the Web), the Azure CLI, Docker, CUDA stacks on GPU SKUs, and the Azure Machine Learning Python SDK v2 environment. Administrators can create an instance on behalf of a user (disable SSO in that flow) and can push a setup script for packages and mounts.

Facts the exam likes:

• One owner. Other people in the workspace do not share the VM as a multi-user login. They can still open notebooks from the workspace file share, because that share is mounted as the default working directory on every instance.
• 120 GB OS disk (P10). You cannot change the OS disk type. Customer-managed key encryption is not currently supported for that OS disk. Temporary data can go to /mnt (size depends on the VM SKU) or /tmp. Do not fill / with training data. If the disk fills, idle shutdown and studio stop/restart can fail; clear at least several GB from the terminal (df -h), then reboot. Do not sudo shutdown as your stop mechanism—that does not deallocate through Azure Machine Learning.
• Idle shutdown and schedules. Enable idle shutdown so an abandoned Jupyter session does not burn GPU hours. Idle means no Jupyter kernel, no Jupyter terminal, no Azure Machine Learning run, no VS Code connection, and no custom app. Stopping the VM stops compute-hour billing. You still pay for the disk, the public IP, and the standard load balancer. Every instance and cluster gets a load balancer, and that balancer exists even while the compute is stopped.
• Quota. Dedicated cores for instances and clusters are a shared regional quota. Stopping an instance does not release quota, so you can restart it. If quota is exhausted, you cannot create another instance in that VM family until you delete something or request more quota (az ml compute list-sizes shows what the region will offer you).
• Hub reuse. On a hub, a compute instance can be reused across the hub’s project workspaces. That is hub-specific sharing of a still-single-owner workstation, not a multi-user cluster.

If you will submit training jobs to the instance (not just run cells), enable root access at create time. You cannot retrofit that later.

Compute cluster: named, autoscaling, reusable

A cluster is the classic “cpu-cluster” or “gpu-cluster” you pass as compute="cpu-cluster" in SDK v2. It can be one node or many. It autoscales when jobs arrive. Set:

• min_instances: 0 so the cluster scales to zero between jobs. A min of 1 is a warm pool you pay for 24/7.
• max_instances to cap spend and to stay inside quota.
• idle_time_before_scale_down (seconds) so a burst of jobs can reuse nodes without waiting for a cold scale-up every time.

CLI v2 YAML sketch:

$schema: https://azuremlschemas.azureedge.net/latest/amlCompute.schema.json
name: cpu-cluster
type: amlcompute
size: STANDARD_DS3_v2
min_instances: 0
max_instances: 4
idle_time_before_scale_down: 120
identity:
  type: system_assigned

az ml compute create --file cpu-cluster.yml

Clusters support CPU and GPU SKUs. GPU jobs need a matching CUDA version in the environment (Hopper H100/H200 families need CUDA 12.0+, Ampere A100 needs 11.0+, and so on). Isolated SKUs such as Standard_M128ms exist for compliance workloads that must be the only tenant on the host.

Identity on a cluster is either one system-assigned identity or one or more user-assigned identities, not both at once. The default identity mounts datastores and pulls ACR images. Code inside the job can use DEFAULT_IDENTITY_CLIENT_ID with ManagedIdentityCredential to reach Azure Storage without an account key.

Serverless compute: omit the name

If you leave compute off a command, sweep, AutoML, or parallel job, the job runs on serverless compute. Azure Machine Learning creates, scales, patches, and deletes the nodes. You can still set resources.instance_type and resources.instance_count, and queue_settings.job_tier of Standard (dedicated) or Spot (low priority). If you specify nothing, the service picks a CPU VM size from quota, cost, performance, and disk size.

Why serverless shows up as the “reduce management overhead” answer:

• No cluster objects to replicate in every workspace for every VM size
• No waiting for a named cluster to scale down before your job can scale up on those nodes
• Quota is still required (workspace and subscription). Usage appears as Serverless in the Azure portal quota blade. Insufficient family quota fails the job with a message to shrink instance_count or request quota.
• Billing matches Azure Machine Learning compute. Identities: user passthrough or workspace user-assigned managed identity. No system-assigned identity on the serverless job.

Pipeline jobs set settings.default_compute: azureml:serverless (CLI) or pipeline_job.settings.default_compute = "serverless" (SDK). Studio job submit and Designer also have a Serverless choice.

You still need Contributor or AzureML Data Scientist on the workspace to submit. Serverless does not bypass RBAC.

Subresources you pay for even when “nothing is running”

Creating instances and clusters also creates subresources: the VMs, a load balancer per instance or cluster, virtual network plumbing, and bandwidth charges for cross-region egress. That is why a “I stopped everything” invoice still shows disk, IP, and load balancer for instances, and why deleting unused compute (not merely stopping it) is a cost-control action.

Retired VM series (NC, NCv2, ND, NV from 2023; Av1 and HB from 2024; NCv3 from 30 September 2025) should be recreated on a supported size. If a GPU SKU is missing from the picker, it is often a quota problem, not an unsupported series.

Exam scenario

A team trains nightly on GPUs, experiments in notebooks during the day, and deploys a real-time classifier. Give each scientist a compute instance with idle shutdown for notebooks. Run the nightly training as a command job with compute omitted (serverless) or on a cluster with min_instances: 0. Deploy to a managed online endpoint, not to the compute instance. If the workspace is a hub project, do not look for a cluster create button; use serverless.

Common trap

Setting cluster min_instances to 1 “so jobs start faster,” then being surprised by 24/7 VM charges. The matching instance trap is enabling idle shutdown and assuming the invoice goes to zero: disk, public IP, and load balancer still bill. A third trap is filling the 120 GB OS disk until the instance cannot shut down automatically.