12.2 Pipeline Optimization: Caching, Slicing & Concurrency
Key Takeaways
- Shallow fetch and disabling submodule or LFS checkout remove clone time from every run of a large repository.
- The Cache@2 task restores a keyed archive; the key normally hashes a lock file so a dependency change invalidates the cache automatically.
- Cache entries are evicted by least-recently-used after seven days without access, and each entry is capped at 10 GB.
- Test slicing with a parallel strategy distributes a suite across agents, cutting wall-clock time in proportion to the parallel jobs purchased.
- Matrix strategies multiply a job across configurations and consume one parallel job per leg, so concurrency cost scales with matrix size.
12.2 Pipeline Optimization: Caching, Slicing & Concurrency
As enterprise software portfolios grow, CI/CD pipelines naturally accumulate build tasks, security scans, packaging steps, and automated tests. Without deliberate performance engineering, pipeline runtimes degrade from minutes to hours. Long pipeline durations stifle developer experimentation, delay hotfixes, and escalate cloud infrastructure costs.
Concurrently, organizations generate petabytes of build artifacts, symbols, container layers, and test logs. DevOps architects must balance the need for fast feedback and rapid execution with strict regulatory governance, auditing retention mandates, and storage cost controls. On the AZ-400 exam, candidates must design high-throughput pipelines using the Cache@2 task, parallel test slicing, agent autoscaling, and comprehensive build retention leases.
1. Optimizing Pipeline Duration & Fast Feedback Velocity
A central tenet of DevOps is the Fast Feedback Loop. When a developer commits code, automated feedback should arrive within minutes, while the mental context of the change is still fresh.
┌─────────────────────────────────────────────────────────────────────────────────┐
│ PIPELINE BOTTLENECK ANALYSIS MATRIX │
├──────────────────────────┬───────────────────────┬──────────────────────────────┤
│ Pipeline Stage │ Typical Bottleneck │ Architectural Solution │
├──────────────────────────┼───────────────────────┼──────────────────────────────┤
│ 1. Source Checkout │ Multi-GB git history, │ • Shallow clones │
│ │ submodules, tags │ (fetchDepth: 1) │
│ │ │ • Exclude tags (fetchTags: false)│
├──────────────────────────┼───────────────────────┼──────────────────────────────┤
│ 2. Dependency Resolution │ Downloading hundreds │ • Pipeline Caching (Cache@2) │
│ │ of npm/NuGet packages │ • Local Artifact Feeds │
│ │ over public internet │ • Upstream Feed Caching │
├──────────────────────────┼───────────────────────┼──────────────────────────────┤
│ 3. Code Compilation │ Rebuilding unchanged │ • Incremental builds │
│ │ code modules from zero│ • Compiler caches (ccache) │
│ │ │ • Pre-baked golden VM images │
├──────────────────────────┼───────────────────────┼──────────────────────────────┤
│ 4. Automated Testing │ Sequential execution │ • Parallel Test Slicing │
│ │ of thousands of tests │ • Test Impact Analysis (TIA) │
│ │ │ • Distributed test runners │
├──────────────────────────┼───────────────────────┼──────────────────────────────┤
│ 5. Artifact Publishing │ Uploading uncompressed│ • Targeted path filters │
│ │ binaries and symbols │ • Pipeline Artifacts (v2) │
│ │ │ • Compress before upload │
└──────────────────────────┴───────────────────────┴──────────────────────────────┘
Source Checkout Optimization
By default, Azure Pipelines performs a deep git fetch. For repositories with long commit histories, cloning the entire history consumes significant time and disk bandwidth:
steps:
- checkout: self
fetchDepth: 1 # Shallow clone: fetches only the tip commit
fetchTags: false # Skips downloading git tags
submodules: false # Disables recursive submodule fetching unless required
2. Pipeline Caching with the Cache@2 Task
Ephemeral build agents (such as Microsoft-hosted agents or auto-reimaged VMSS agents) start with an empty local filesystem for every job. Without caching, package managers like npm, NuGet, Maven, pip, or Gradle must download every dependency from remote registries over the internet on every single run.
Pipeline Caching stores build outputs, package dependencies, and compiler caches in Azure DevOps cloud storage, making them available across pipeline runs.
[Pipeline Run Starts]
│
▼
[Cache@2 Task: Evaluates Key]
'npm | "$(Agent.OS)" | package-lock.json'
│
┌─────────────────┴─────────────────┐
▼ ▼
[CACHE HIT] [CACHE MISS]
│ │
▼ ▼
Downloads tarball from cloud Evaluates restoreKeys fallback
Extracts to ~/.npm in seconds (e.g., 'npm | "$(Agent.OS)"')
│ │
▼ ▼
Sets CACHE_RESTORED = 'true' Runs full npm ci from registry
Skips redundant download tasks At end of job: Uploads cache tarball
Cache Keys and Restore Keys Syntax
The Cache@2 task relies on a primary key and optional fallback restore keys:
- Primary Key (
key): A structured string comprising literal strings, dynamic pipeline variables, and file hashes (using thehashFiles()utility). When any dependency changes inpackage-lock.json, the hash changes, generating a new primary key. - Restore Keys (
restoreKeys): Fallback keys searched if the primary key misses. Azure DevOps searches restore keys using prefix matching, restoring the most recently saved cache matching the prefix. This enables incremental updates. - Cache Hit Variable (
cacheHitVar): A variable populated with'true'when the primary key matches exactly. Subsequent steps can evaluate this variable in theircondition:to skip package installation altogether.
Complete Cache@2 YAML Example (Node.js / npm)
variables:
npm_config_cache: $(Pipeline.Workspace)/.npm
steps:
# 1. Evaluate and restore npm cache
- task: Cache@2
displayName: 'Cache npm packages'
inputs:
key: 'npm | "$(Agent.OS)" | package-lock.json'
restoreKeys: |
npm | "$(Agent.OS)"
npm
path: $(npm_config_cache)
cacheHitVar: CACHE_RESTORED
# 2. Run npm ci only if cache missed or needs reconciliation
- script: npm ci --cache $(npm_config_cache) --prefer-offline
displayName: 'Install npm dependencies'
condition: ne(variables.CACHE_RESTORED, 'true')
# 3. Compile application
- script: npm run build
displayName: 'Compile Angular/React Client'
Caching Across Common Package Ecosystems
┌─────────────────────────────────────────────────────────────────────────────────┐
│ CACHE@2 STRATEGIES BY ECOSYSTEM │
├──────────────┬────────────────────────────────┬─────────────────────────────────┤
│ Ecosystem │ Path to Cache │ Primary Key Pattern │
├──────────────┼────────────────────────────────┼─────────────────────────────────┤
│ npm / Yarn │ $(Pipeline.Workspace)/.npm │ 'npm | "$(Agent.OS)" | │
│ │ │ package-lock.json' │
├──────────────┼────────────────────────────────┼─────────────────────────────────┤
│ NuGet (.NET) │ $(UserProfile)/.nuget/packages │ 'nuget | "$(Agent.OS)" | │
│ │ │ **/packages.lock.json' │
├──────────────┼────────────────────────────────┼─────────────────────────────────┤
│ Python (pip) │ ~/.cache/pip │ 'pip | "$(Agent.OS)" | │
│ │ │ requirements.txt' │
├──────────────┼────────────────────────────────┼─────────────────────────────────┤
│ Java (Maven) │ ~/.m2/repository │ 'maven | "$(Agent.OS)" | │
│ │ │ **/pom.xml' │
├──────────────┼────────────────────────────────┼─────────────────────────────────┤
│ C/C++ │ $(Pipeline.Workspace)/.ccache │ 'ccache | "$(Agent.OS)" | │
│ (ccache) │ │ src/**/CMakeLists.txt' │
└──────────────┴────────────────────────────────┴─────────────────────────────────┘
Cache Constraints and Invalidation Rules
- Size Limits: An individual cache entry can be up to 10 GB. Organizations have total project cache limits based on subscription tier.
- Eviction Policy: Azure Pipelines automatically evicts cache entries that have not been accessed within 7 days using a Least Recently Used (LRU) algorithm.
- Post-Job Execution: The
Cache@2task restores dependencies during the step execution, but uploads the cache during a hidden post-job step. If any preceding step in the job fails, the cache upload is skipped by default to avoid persisting corrupted or partial dependencies.
3. Parallel Execution Optimization: Slicing, Matrix & Autoscaling
When optimizing build and test pipelines, parallelization delivers dramatic runtime reductions.
Parallel Test Slicing (strategy: parallel)
Large enterprise test suites (e.g., 5,000 automated functional tests) can take hours on a single agent. Test slicing splits test execution across multiple parallel agents:
jobs:
- job: SlicedIntegrationTests
displayName: 'Execute Sliced Tests across Agents'
strategy:
parallel: 4 # Spawns 4 concurrent agent jobs
pool:
vmImage: 'ubuntu-latest'
steps:
- checkout: self
- task: DotNetCoreCLI@2
displayName: 'Run Sliced Unit Tests'
inputs:
command: 'test'
projects: '**/*Tests.csproj'
arguments: '--configuration Release'
When combined with VSTest@2 test slicing, Azure Pipelines automatically monitors test runtimes and divides the test assemblies across the 4 agents such that each agent completes approximately at the same time.
Matrix Strategies (strategy: matrix)
A matrix strategy runs the same job across permutations of operating systems, runtime targets, or database versions:
jobs:
- job: CrossPlatformValidation
strategy:
matrix:
Linux_Node18:
osImage: 'ubuntu-latest'
nodeVersion: '18.x'
Linux_Node20:
osImage: 'ubuntu-latest'
nodeVersion: '20.x'
Windows_Node20:
osImage: 'windows-latest'
nodeVersion: '20.x'
maxParallel: 2 # Throttles concurrency to protect agent pool capacity
pool:
vmImage: $(osImage)
steps:
- task: NodeTool@0
inputs:
versionSpec: $(nodeVersion)
- script: npm test
A web development team uses npm to install dependencies in an Azure Pipelines CI build running on Microsoft-hosted Ubuntu agents. Downloading and installing node modules takes 6 minutes on every single run because hosted agents are ephemeral and start with an empty cache. The team implements the Cache@2 task. To maximize cache hit rates, which cache key strategy should the team implement so that the cache is restored whenever package-lock.json is unchanged, but falls back to the most recent OS-level cache if dependencies were updated?
An enterprise application test suite contains 4,000 integration tests that take 75 minutes to execute sequentially on a single self-hosted build agent. The DevOps team needs to reduce the total test execution time to under 15 minutes without rewriting or deleting any tests. How should the team configure the pipeline job in YAML?