5.3 Notebooks, Command Jobs, and Training Scripts

Notebooks, Command Jobs, and Training Scripts

Quick Answer: Notebooks on a compute instance are for exploration. Production training is a command job (type: command) whose YAML sets code, command, environment, optional compute, and inputs. Submit with az ml job create -f job.yml. Azure Machine Learning snapshots the code folder at submit time. Do not train production models only in untitled notebooks.

Domain 2 clusters two official bullets: use notebooks for experimentation and exploration and run model training scripts. AI-300 expects you to know when each is legitimate — and why MLOps treats them as different surfaces.

Notebooks are the lab, not the factory

A compute instance is a managed development VM. You open Jupyter or VS Code against it, query data assets, plot distributions, test a 50-line scikit-learn fit, and call mlflow.start_run() as in section 5.1. That loop is how you discover whether a feature even belongs in the model. It is the wrong loop for a model you will register, deploy, and defend in an audit.

Problems with “we trained it in a notebook”:

• The kernel state is not a code snapshot. A reviewer cannot re-run exactly those cells.
• Untitled files have no Git history. Chapter 4 already required source control for Machine Learning projects.
• Package versions drift with whatever the instance image happened to contain that morning unless you pin an environment.
• Compute instances are one box. They do not autoscale like a cluster, and they keep billing while you leave them running.
• Promotion scripts expect a job name whose outputs contain an MLflow model folder, not a casual model.pkl on the instance disk.

The healthy pattern is: explore in a notebook, then copy the working training path into src/main.py, parameterize it, and submit a command job. Keep the notebook as a report or as the place you debug a failing job by pulling the same inputs.

Anatomy of a command job

SDK/CLI v2 training is a job with type: command. A typical job.yml:

$schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json
code: src
command: >-
  python main.py
  --train-data ${{inputs.train_data}}
  --learning-rate ${{inputs.learning_rate}}
environment: azureml:AzureML-sklearn-1.0-ubuntu20.04-py38-cpu@latest
compute: azureml:cpu-cluster
inputs:
  train_data:
    type: uri_folder
    path: azureml:claims-features:2026-08-15
  learning_rate: 0.05
experiment_name: claims-lgbm
display_name: claims-lgbm-baseline

Submit:

az ml job create -f job.yml

SDK v2 is the same contract in Python: command(code="./src", command="python main.py --train-data ${{inputs.train_data}}", environment=..., inputs=..., compute="cpu-cluster") then ml_client.jobs.create_or_update(command_job). To try serverless compute, omit compute (delete the YAML key or the compute= argument). Do not invent a compute named serverless.

Expressions such as ${{inputs.train_data}} are SDK/CLI v2 substitutions resolved when the job starts. Your Python argparse then receives a local path (mount or download) or a URI, depending on the input mode you learned with data assets.

Snapshots, environments, and what the job actually runs

When you submit, Azure Machine Learning uploads the code directory and records that snapshot. The running job uses that copy. If you edit main.py on disk two minutes later, the in-flight job does not pick up the edit. That is a feature: reproducibility. It is also why “I fixed the bug but the job still failed” is usually a second submit, not a save in the editor.

The environment is the container plus Python packages — not “whatever was pip-installed on my compute instance last week.” Curated environments (AzureML-lightgbm-*, AzureML-sklearn-*, and their successors) are the fast path. Custom environments are a base image plus a conda YAML, versioned as a workspace (or registry) asset. Jobs that point at an Azure Container Registry with a customized domain name label can fail on image pull; use the default *.azurecr.io login server.

Status walks Starting → Preparing → Running → Completed (or Failed/Canceled). Preparing includes image build and node allocation. Monitor with az ml job stream -n <name>, az ml job show --web, or ml_client.jobs.stream. Register an MLflow model from the job with a path such as azureml://jobs/{job_name}/outputs/artifacts/paths/model/ or az ml model create ... -p runs:/$run_id/model --type mlflow_model.

REST exists underneath both SDK and CLI (jobType: Command). AI-300 will not ask you to memorize Resource Manager JSON; it will ask you to know that CLI/SDK jobs are the supported authoring path and that a service principal token is required for unattended REST.

From notebook prototype to a training script

A production main.py should:

1. Parse hyperparameters and data paths from arguments (so a sweep in 5.4 can inject them).
2. Call mlflow.autolog() without start_run (the job already started a run).
3. Read inputs from the paths Azure Machine Learning mapped, not from a laptop ~/Downloads.
4. Log the primary metric under a stable name.
5. Write the MLflow model to a known artifact path.

Keep data preparation that is expensive out of every training restart when you can: a pipeline step (next chapter) or a registered data asset that already holds the features. Every command job rebuilds its environment and reloads data unless you designed otherwise.

GitHub Actions (Chapter 4) should call az ml job create -f job.yml on a branch or on main, not open a notebook. The YAML and src/ live in Git. The job name in studio is the audit trail.

Exam scenario

A data scientist has a working fraud classifier in Untitled12.ipynb on a compute instance, with pip install cells and a personal SAS token. The MLOps engineer copies the training cells into src/train.py, replaces the SAS path with azureml:claims-features:2026-08-15, pins environment: azureml:<curated>@latest, writes job.yml with type implied by the command-job schema, and runs az ml job create -f job.yml against cpu-cluster (or omits compute for serverless). The notebook remains for exploratory plots. The registered model’s lineage points at the job, not at the notebook kernel.

Common trap

Do not train the production candidate only in an untitled notebook. Do not put compute: azureml:serverless — serverless is the absence of a compute property. Do not assume a job sees unsaved editor buffers; it sees the snapshot. Do not reuse the compute instance’s interactive conda as if it were the job environment. Do not hard-code /mnt/batch/tasks/... paths you copied from one log; use ${{inputs.*}} so the next cluster still works.