7.1 YAML Pipeline Anatomy: Stages, Jobs, Steps & Workspaces

7.1 YAML Pipeline Anatomy: Stages, Jobs, Steps & Workspaces

Azure Pipelines uses YAML (YAML Ain't Markup Language) to define CI/CD pipelines as version-controlled code residing directly within your source code repository. Modern enterprise delivery architectures demand pipelines that can cleanly separate concerns: compiling code, executing static analysis, deploying infrastructure, and running integration tests across diverse environments with granular gates.

To pass the AZ-400 exam and architect resilient enterprise delivery systems, candidates must master the core hierarchical anatomy of Azure Pipelines YAML, understand agent and workspace execution boundaries, orchestrate parallel and sequential execution using Directed Acyclic Graphs (DAGs), and configure advanced execution condition expressions.

---

1. Core Hierarchical Anatomy: Pipeline, Stages, Jobs, Steps

The Azure Pipelines YAML document follows an explicit four-tier hierarchy. Every element in the pipeline belongs to a parent container that dictates scheduling, agent assignment, and execution isolation.

Pipeline (Root Configuration, Triggers, Global Variables, Resources)
  └── Stages (Major Milestone Boundaries: Build, QA, Staging, Production)
        └── Jobs (Scheduling Units Assigned to Individual Agents)
              └── Steps (Sequential Execution Units: Tasks, Scripts, Checkouts)

The Root Pipeline

The root level of the YAML document defines global attributes governing the entire pipeline lifecycle:

• trigger: Dictates continuous integration (CI) triggers based on branch commits, tags, or path filters.
• pr: Controls pull request validation triggers.
• schedules: Configures cron-based recurring execution schedules.
• resources: Declares dependent external artifacts, repositories, container registries, packages, or build pipelines.
• variables: Sets pipeline-wide variables accessible across all stages.
• pool: Defines the default agent pool if not overridden at the stage or job level.
• stages: The top-level collection containing one or more deployment or build stages.

Stages (stage:) — Architectural Isolation Boundaries

A Stage represents the highest-level operational milestone within a pipeline (e.g., Build, SecurityScan, DeployDev, DeployProd). Stages provide critical isolation boundaries:

• Security & Governance: Approvals, checks, and release gates can be enforced at stage boundaries.
• Agent Pool Selection: Stage A can run on a Linux Microsoft-hosted pool (ubuntu-latest), while Stage B executes on a private, self-hosted Windows agent pool with on-premises network line-of-sight.
• Independent Topologies: Stages can run sequentially or in parallel based on explicit dependsOn declarations.
• If a pipeline defines only jobs without an explicit stages: block, Azure Pipelines automatically wraps all jobs in a single synthetic, invisible default stage.

Jobs (job: and deployment:) — Agent Scheduling Units

A Job is the fundamental scheduling and execution unit dispatched to an individual pipeline agent. All steps within a single job run on the exact same agent virtual machine or container. Azure Pipelines supports two primary job types:

1. Standard Jobs (job: <name>):

   • Used for compilation, automated testing, container image packaging, and script execution.
   • Executes within a fresh workspace on the assigned agent.
   • Supports execution strategies: matrix (running the same job across multiple OS/runtime configurations) and parallel (slicing workloads across N agents).

2. Deployment Jobs (deployment: <name>):

   • Specifically designed for delivering software to an Environment (e.g., Azure Kubernetes Service, App Service, Virtual Machine pools).
   • Automatically records deployment history, work items, and commits against the target environment in the Azure DevOps portal.
   • Replaces the generic steps: block with structured deployment lifecycle hooks:
     • preDeploy: Steps executed before deployment begins (e.g., draining connections, provisioning prerequisites).
     • deploy: Steps that perform the actual deployment action.
     • routeTraffic: Steps that shift traffic to the updated version (e.g., swapping App Service slots, updating ingress rules).
     • postRouteTraffic: Steps executed after traffic routing to verify health (e.g., synthetic monitoring).
     • on: failure / on: success: Automated rollback or cleanup steps.
   • Supports deployment strategies: runOnce, rolling, and canary.

# Example Deployment Job Anatomy
- deployment: DeployProductionWeb
  displayName: 'Deploy to Production AKS'
  pool:
    vmImage: 'ubuntu-latest'
  environment: 'production.microservices'
  strategy:
    runOnce:
      deploy:
        steps:
          - checkout: self
          - task: KubernetesManifest@1
            displayName: 'Deploy Kubernetes Manifests'
            inputs:
              action: 'deploy'
              manifests: '$(Pipeline.Workspace)/manifests/*.yaml'

Steps (steps:) — Sequential Execution Units

A Step is the lowest-level linear action executed within a job. Steps execute sequentially on the agent. If any step fails, subsequent steps are skipped by default unless an explicit condition override is configured.

Common step primitives include:

• task:: Executes a pre-packaged task from the Azure DevOps Marketplace (e.g., task: DotNetCoreCLI@2, task: AzureCLI@2, task: Docker@2).
• script:: Runs a cross-platform shell script (Bash on Linux/macOS, Command Prompt on Windows).
• bash:: Explicitly invokes a Bash shell environment (/bin/bash).
• powershell:: Invokes Windows PowerShell (powershell.exe).
• pwsh:: Invokes cross-platform PowerShell Core (pwsh).
• checkout:: Configures repository source code retrieval (checkout: self, checkout: none, or referencing external repository resources).
• template:: References a reusable step template file.

---

2. Execution Boundaries and Workspace Lifecycles

A critical concept tested on the AZ-400 exam is the boundary of state and persistence across stages and jobs.

Workspace Isolation Mechanics

When a job begins execution on an agent:

1. The agent allocates a dedicated working directory: $(Agent.BuildDirectory) (also aliased as $(Pipeline.Workspace)).
2. The repository is cloned into $(Pipeline.Workspace)/s (unless checkout: none is specified).
3. Binaries and test outputs written to disk during Job 1 do not exist on the agent assigned to Job 2, even if both jobs belong to the same stage. To share files between jobs or stages, you must publish them as pipeline artifacts using PublishPipelineArtifact@1 (or publish: shorthand) and retrieve them using DownloadPipelineArtifact@2 (or download: shorthand).

# Job 1: Publishing Build Artifacts
- job: BuildJob
  steps:
    - script: npm run build
    - publish: $(System.DefaultWorkingDirectory)/dist
      artifact: WebAppDrop

# Job 2: Downloading Artifacts
- job: TestJob
  dependsOn: BuildJob
  steps:
    - download: current
      artifact: WebAppDrop
    - script: ls -la $(Pipeline.Workspace)/WebAppDrop