3.2 Create and Manage Environments

Create and Manage Environments

Quick Answer: An environment is the software stack (container + Python packages) for a job or deployment. Compute is the hardware. Use curated Microsoft images from the azureml registry when they fit; otherwise create a custom environment from a Docker image, a Docker build context, and/or a conda YAML. Jobs pin azureml:<env>:<version>. Builds use the workspace Azure Container Registry (ACR).

Exam AI-300 Domain 1 pairs environments with data assets and components because all three are workspace assets you version, reuse, and later promote. If the data identity is wrong you train on the wrong week of files. If the environment identity is wrong you cannot reproduce the job even when the data is perfect.

Environment versus compute

The job configuration is three pieces: code, environment, and compute target. Mixing the last two is a high-frequency exam distractor.

• Environment — “What is installed?” Base operating system, CUDA/cuDNN if GPU, Python, scikit-learn, mlflow, system libraries. Encapsulated as a Docker image the compute pulls.
• Compute — “Where does it run?” A compute instance, compute cluster, serverless compute, or attached Kubernetes. SKU, node count, and scale-down idle time live here, not in the environment YAML.

The same environment object should travel from a local training script to a scaled cluster job to a managed online endpoint scoring container. That is the reproducibility contract: train and serve in one stack.

Curated versus custom

Azure Machine Learning documents two practical buckets you will configure, plus a third conceptual split for who manages Python.

Conceptually, custom environments are user-managed (you bring a finished image or Docker build context) or system-managed (you give a conda spec; Azure Machine Learning builds a conda env on top of a base image).

Curated names starting with AzureML- or Microsoft are reserved. You cannot submit local edits to a curated environment under those prefixes; change the name and you have created a custom environment, which also means you now own patching.

Three ways to define a custom environment

1. Prebuilt Docker image. Point image at Docker Hub, Microsoft Container Registry, or your ACR (pytorch/pytorch:2.4.0-cuda12.1-cudnn9-runtime, or an mcr.microsoft.com/azureml/openmpi... CPU/GPU base). Azure Machine Learning uses that image as-is.

2. Docker build context. Set build.path to a folder that contains a Dockerfile (max 1 MB) plus files the Dockerfile COPYs (requirements.txt, wheel files). If you omit build.dockerfile_path, the service looks for Dockerfile at the context root. The image build starts when the environment is created. Watch Studio build logs.

The workspace ACR is accessed with a Microsoft Entra token whose lifetime is not configurable and is typically 60–90 minutes. A Dockerfile that compiles CUDA kernels for two hours can fail when that token expires. Split long builds, prebuild in ACR, or shorten the Dockerfile.

3. Base image plus conda YAML. image plus conda_file (path or inline). Azure Machine Learning builds the conda environment on the image when the environment is first used in a job or deployment (you can also trigger a build in Studio). Critical trap: Python packages you pip install in the base image are not visible inside the materialized conda env. The job runs in the conda env. List every pip/conda package the script imports in conda_file, including mlflow, azureml-mlflow, and CUDA-aware wheels if needed.

Microsoft publishes CPU and GPU Ubuntu base images (Miniconda, OpenMPI, CUDA, cuDNN, NCCL on GPU SKUs) in the AzureML-Containers GitHub repository. Prefer those as parents unless you have a compliance-mandated golden image.

$schema: https://azuremlschemas.azureedge.net/latest/environment.schema.json
name: claims-train-sklearn
version: "4"
image: mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04
conda_file: conda-yamls/claims-sklearn.yml
description: Pinned sklearn training stack for claims models

az ml environment create --file env.yml registers it. SDK v2: Environment(image=..., conda_file=..., name=..., version=...) then ml_client.environments.create_or_update.

Versioning, hashing, cache, and pinning

List with az ml environment list; show with az ml environment show --name ... --version .... Only description and tags mutate. Changing the image, Dockerfile, or conda spec requires a new version.

The service hashes base image + docker steps + Python packages (not the display name). Two environments with different names but identical specs share a cached image (azureml/azureml_<hash> in workspace ACR). Reordering conda channels or adding numpy==1.26.4 changes the hash and rebuilds. First use of a new spec can take several minutes; later jobs reuse the cache.

Unpinned specs (numpy without ==) freeze the version that existed at first build. A later environment with the same unpinned list reuses that old wheel. To pick up a new numpy, pin a version so the hash changes. Unpinned base image tags such as :latest can rebuild whenever Microsoft retags the image — good for patches, bad for surprise breakage. Curated images are patched on a roughly two-week cadence; managed online endpoints do not move to a new base until you redeploy.

Jobs reference environment: azureml:claims-train-sklearn:4 or azureml:claims-train-sklearn@latest. Pin in production. @latest is convenient in a sandbox and a classic “it worked Friday” incident on Monday. You may also inline an environment in a job YAML without name/version; that creates an unregistered/anonymous environment that is tracked by hash, not by a friendly asset name — fine for a spike, poor for MLOps catalogs.

Archive (az ml environment archive) hides an environment from default lists; jobs may still reference it. Archiving does not delete the ACR repository. Remove a cached image with az acr repository delete if policy requires it. Restore with az ml environment restore.

Network isolation can block the default serverless image-build compute; restricted workspaces often need a dedicated build compute. That detail returns in Chapter 4; know that environment build is a control-plane operation against ACR, not something the training cluster does first.

Exam scenario

A fraud model trained on curated sklearn-1.5 in January. The job YAML used .../labels/latest. In March a new curated version shipped a patched sklearn. A “quick retrain” pulled the new image, numeric predictions shifted, and the online endpoint — still on the January image — disagreed with batch scores. The MLOps engineer clones the working spec into custom fraud-sklearn:1, pins environment: azureml:fraud-sklearn:1 on both the training component and the online deployment, and treats any conda or apt change as version 2.

Common trap

Do not answer “put more RAM on the cluster” when the failure is ModuleNotFoundError. That is an environment miss, not a compute miss. The sibling trap is assuming a Dockerfile pip install survives a conda_file layer — it does not. A third trap is editing a curated AzureML- name in place; the platform will reject or fork you into a custom image you now must patch yourself.