6.2 Implement Training Pipelines

Implement Training Pipelines

Quick Answer: An Azure Machine Learning pipeline is a directed acyclic graph (DAG) of components (usually type: command steps, optionally sweeps). The job YAML itself is type: pipeline. Steps pass typed outputs (uri_folder, mltable, mlflow_model) through ${{parent.jobs.<step>.outputs.<name>}}. Unchanged deterministic steps reuse by default; settings.force_rerun: true reruns everything. Schedules are recurrence or cron, not event triggers. Prefer YAML / SDK v2 over Designer-only designs on this exam.

Exam AI-300 Domain 2 asks you to implement training pipelines. A single command job is the right unit for one script. A pipeline is the right unit when preprocess, train, evaluate, and (later) register or deploy must run as one workflow with lineage, reuse, and schedules.

Why a pipeline instead of one command job

Microsoft's pipeline concept page gives two operational reasons: standardize Machine Learning Operations (MLOps) so data engineers, scientists, and ML engineers own different steps, and cut cost by reusing outputs and by placing each step on the cheapest adequate compute.

Do not confuse Azure's three "pipeline" products:

• Azure Machine Learning pipelines — data to model. Caching, distribution, component reuse. This exam bullet.
• Azure Data Factory pipelines — data to data. Strongly typed movement.
• Azure Pipelines (Azure DevOps) — code and model to app. CI/CD, approvals, GitHub Actions' cousin.

A GitHub Actions workflow that calls az ml job create --file pipeline.yml is DevOps around an Azure Machine Learning pipeline, not a substitute for the DAG.

YAML anatomy (type: pipeline)

Schema: https://azuremlschemas.azureedge.net/latest/pipelineJob.schema.json. Required keys are type: pipeline and jobs.

$schema: https://azuremlschemas.azureedge.net/latest/pipelineJob.schema.json
type: pipeline
display_name: claims-train-eval
settings:
  default_compute: azureml:cpu-cluster
  force_rerun: false
  continue_on_step_failure: false
inputs:
  raw_images:
    type: uri_folder
    path: azureml:claims-images:2026-08-15
outputs:
  trained_model:
    type: mlflow_model
jobs:
  prep:
    type: command
    component: azureml:claims_prep@latest
    inputs:
      raw: ${{parent.inputs.raw_images}}
    outputs:
      cleaned:
        type: uri_folder
  train:
    type: command
    component: ./train.yml
    compute: azureml:gpu-cluster
    inputs:
      data: ${{parent.jobs.prep.outputs.cleaned}}
    outputs:
      model: ${{parent.outputs.trained_model}}

Submit with az ml job create --file pipeline.yml. SDK v2 uses an @pipeline decorated function whose body calls component callables and returns named outputs; set pipeline_job.settings.default_compute = "cpu-cluster" or "serverless". For serverless in YAML, default_compute: azureml:serverless.

Current child job types inside jobs are command and sweep. A distributed trainer is still a command step that happens to include a distribution block from section 6.1. The pipeline does not become type: pytorch.

Settings you will be tested on:

• default_compute — every step inherits it unless the step sets its own compute. That is how prep stays on CPU and train moves to GPU.
• default_datastore — where pipeline outputs land if you do not pin a path; otherwise the workspace blob datastore.
• continue_on_step_failure — schema default is true (later independent steps still run). Production retrains often set false so a failed prep cancels the graph.
• force_rerun — default false. Covered below.

Studio shows a graph of child jobs under the parent pipeline job. Each child is a real job with its own logs, metrics, and outputs.

Components, typed data, and the DAG

A component is a versioned step interface: name, inputs, outputs, environment, command, code. You studied creating components in Chapter 3. Pipelines consume them.

• Literal inputs (string, number, integer, boolean) are hyperparameters (max_epochs, learning_rate). They do not draw graph edges.
• Object inputs/outputs (uri_file, uri_folder, mltable, mlflow_model) do draw edges. The prep step's uri_folder cleaned output becomes the train step's input via ${{parent.jobs.prep.outputs.cleaned}}. Pipeline-level inputs use ${{parent.inputs.raw_images}}. Bind a child output up to the pipeline with ${{parent.outputs.trained_model}} on that child's outputs map, as in the CLI v2 examples.

Modes match other v2 jobs: inputs ro_mount / download / direct; outputs rw_mount / upload. Passing mltable is the right type when the step needs a table blueprint rather than a raw folder. Passing mlflow_model is how evaluate consumes what train just wrote without a side-channel registry write.

is_deterministic on a component defaults to true (reuse by default). Set it false when the step is inherently non-repeatable: pulling from a live URL, drawing fresh random synthetic data, or hitting an external API whose bytes change under the same URI.

Register components with az ml component create --file train.yml and reference component: azureml:my_train@latest (or a pinned version) so the DAG does not embed a local folder path. Local component: ./train.yml is fine while iterating.

Force rerun versus reuse

Reuse is how pipelines save money. Criteria, all required:

1. Component is_deterministic is true.
2. Pipeline settings.force_rerun is false.
3. Component code snapshot, environment definition, inputs and parameters, output settings, and run settings are the same as a previous successful run.

When reuse hits, the service copies status, logs, metrics, and outputs from the original child; Studio draws a recycle icon. It does not re-read a lake folder that you overwrote in place under the same URI. If the asset identity did not change, reuse will happily hand you last week's bytes. That is the same immutability lesson as data assets: pin a new version or a new path when the data actually changed.

force_rerun: true reruns every child. Microsoft also notes that children of a force-rerun pipeline cannot be reused by later jobs. Do not leave force rerun on in production schedules or you pay to recompute prep forever and you poison the reuse cache.

Debug unexpected reruns in this order: force_rerun on either job, is_deterministic, ContentSnapshotId (code changed), environment definition, then Studio graph compare for inputs, parameters, and run settings.

Schedules for retraining

When the DAG is good enough for production, attach a schedule. v2 schedules work on v1 or v2 pipeline jobs and do not require publishing a classic pipeline endpoint.

• Recurrence: trigger.type: recurrence with frequency (minute / hour / day / week / month) and interval. Optional schedule.hours, minutes, weekdays, start_time, end_time, time_zone (default UTC).
• Cron: trigger.type: cron with a crontab expression (MINUTES HOURS DAYS MONTHS DAYS-OF-WEEK). In Azure Machine Learning, DAYS and MONTHS values are ignored and treated as *. 15 16 * * 1 is 16:15 UTC every Monday.

Hard limitation: Azure Machine Learning v2 schedules do not support event-based triggers (no "new blob arrived"). If you need storage events, use an external orchestrator and often a batch endpoint wrapping the pipeline.

create_job can point at a YAML file or azureml:<job-name>. You may override settings, inputs, outputs, tags, and experiment name so the scheduled run uses production data while your debug run used a sample. Macro expressions: ${{name}} in output paths; ${{creation_context.trigger_time}} in string inputs.

Manage with az ml schedule create|list|show|update|enable|disable|delete. Disable before delete; deletion is permanent. Even after you assign a managed identity to the schedule, the author must keep job-run permission or the schedule stops. Recurrence on day 31 skips short months. Triggered jobs are named <schedule_name>-YYYYMMDDThhmmssZ.

Designer versus code

You can author the same component DAG in Azure Machine Learning studio Designer (drag and drop), in Python SDK v2, or in CLI v2 YAML. Designer is useful for demos and for people who will not touch YAML. This exam's tooling profile is SDK/CLI v2. When a stem asks how to implement a reusable, scheduled training pipeline in source control, the answer is a type: pipeline YAML (or SDK equivalent) in Git, submitted by az ml job create or GitHub Actions — not "only Designer."

Exam scenario

Claims images land weekly in Azure Data Lake Storage. Prep (CPU, Spark-ish Python) writes a uri_folder of tiles; train (GPU, PyTorch DDP from 6.1) writes an mlflow_model; eval writes a metrics JSON. Scientists kept pasting three command jobs by hand and rerunning prep for every learning-rate change. The MLOps engineer registers claims_prep, claims_train, and claims_eval components, authors pipeline.yml with default_compute: azureml:cpu-cluster, overrides compute on the train step to the GPU cluster, binds outputs with ${{parent.jobs.prep.outputs.cleaned}}, and creates a cron schedule 0 4 * * 1 that overrides the pipeline input to azureml:claims-images@latest. Force rerun stays false so unchanged prep is reused. They do not wait for a blob-created event — v2 schedules cannot fire on that.

Common trap

Calling Azure Pipelines or Data Factory the Azure Machine Learning training DAG. Sibling traps: leaving force_rerun: true on a weekly schedule; expecting reuse after you overwrite files at the same URI; using Designer as the only source of truth with no YAML in Git; assuming cron DAYS/MONTHS fields work; treating a pipeline as type: pytorch instead of putting distribution on the train child command.