6.4 VMSS Elastic Pools, Container Jobs & Agent Governance

6.4 VMSS Elastic Pools, Container Jobs & Agent Governance

Virtual Machine Scale Set agent pools close the gap between hosted and self-hosted agents: private network reach with hosted-style cleanliness. Container jobs add per-job toolchain isolation on top, and agent pool permissions decide which pipelines may consume either.

1. Azure Virtual Machine Scale Set (VMSS) Agent Pools

Azure Virtual Machine Scale Set (VMSS) agents deliver the ideal synthesis of hosted and self-hosted models: the network security and custom sizing of self-hosted machines combined with the elastic scalability and ephemeral clean-state security of hosted agents.

                                [Azure DevOps Organization]
                                             │
                               1. Queue Depth Surge Detected
                                             │
                                             ▼
                              [Azure Resource Manager (ARM)]
                                             │
                                2. Scales Out VM Instances
                                             │
                                             ▼
                       [Azure Virtual Machine Scale Set (VMSS)]
           ┌─────────────────────────────────┴─────────────────────────────────┐
           ▼                                                                   ▼
┌─────────────────────┐                                             ┌─────────────────────┐
│ VMSS Instance 001   │                                             │ VMSS Instance 002   │
│ • Runs in Corp VNet │                                             │ • Runs in Corp VNet │
│ • Executes Job 1    │                                             │ • Executes Job 2    │
│ • AUTO-REIMAGED ────┼──► [Clean OS Disk Applied via Golden Image] │ • AUTO-REIMAGED     │
└─────────────────────┘                                             └─────────────────────┘

Architectural Mechanics

1. Integration: An Azure DevOps administrator provisions an Azure VMSS in an Azure subscription. In Azure DevOps Project Settings → Agent Pools, they create a new pool of type Azure Virtual Machine Scale Set and point it to the VMSS resource.
2. Elastic Autoscaling Governed by Azure DevOps: Azure DevOps acts as the autoscaling engine—not Azure Monitor autoscale rules. Azure DevOps monitors pipeline queue depth. When jobs queue up, it commands ARM to scale out VM instances up to maxCapacity. When jobs finish, instances idle for a configured grace period and are then deprovisioned down to minCapacity (which can be set to 0 to eliminate idle compute costs).
3. Automatic OS Disk Re-imaging (Ephemeral Clean State):
   • By default, Azure DevOps re-images the VM instance after every single job execution.
   • Azure DevOps applies a clean OS disk from the original image (or tears down the VM instance and replaces it), ensuring that subsequent builds receive a completely pristine environment with zero residual state.
4. Custom Golden Images: Organizations can use Azure Image Builder or HashiCorp Packer to pre-bake enterprise SDKs, static code analysis binaries, and internal root certificates into a custom Azure Compute Gallery image. This eliminates lengthy setup tasks during pipeline execution while maintaining compliance.

---

2. Container-Based Job Execution

Both Azure Pipelines and GitHub Actions support running jobs inside custom Docker containers directly on Linux agents/runners. This is configured using the container: directive in YAML.

# Azure Pipelines Container Job
pool:
  name: 'Default' # Self-hosted Linux agent pool

container:
  image: mcr.microsoft.com/dotnet/sdk:8.0-alpine
  options: --privileged # Optional container options

steps:
  - script: |
      dotnet --version
      dotnet build src/Contoso.sln
    displayName: 'Compile inside Alpine Linux Container'

How Container Jobs Work Under the Hood

1. The host agent must have the Docker engine (or Moby) installed and running.
2. When the job starts, the agent pulls the specified container image from Docker Hub, Azure Container Registry (ACR), or GitHub Packages.
3. The agent creates an isolated container and bind-mounts the working directory (_work) into the container (typically mapped to /__w/1/s).
4. All pipeline steps (Bash scripts, CLI tasks) are executed inside the container using docker exec commands rather than on the host operating system.
5. When the job finishes, the container is stopped and removed.

Service Containers (Sidecars) for Integration Testing

Pipelines frequently require supporting services (such as databases or cache servers) during automated integration testing. The services: block spins up sidecar containers networked to the job container:

services:
  redis:
    image: redis:7-alpine
    ports:
      - 6379:6379
  postgres:
    image: postgres:15
    env:
      POSTGRES_PASSWORD: secretpassword
    ports:
      - 5432:5432

---

3. Agent Security, Governance & Permissions

Securing agent infrastructure is vital to prevent unauthorized pipeline execution and lateral network movement.

Registration Authentication: Personal Access Tokens (PATs)

• When registering a self-hosted agent via config.cmd or config.sh, the installer prompts for authentication.
• Least Privilege Principle: Generate a PAT restricted exclusively to the Agent Pools (Read & Manage) scope. Never use an organization-wide administrative PAT. For automated VM bootstrap scripts (e.g., cloud-init or PowerShell DSC), generate short-lived PATs that expire within hours.

Host Service Account Security

• Never run the agent service as root (Linux) or LocalSystem / Domain Admin (Windows).
• Create a dedicated, unprivileged local service account (e.g., azagent).
• Grant this service account write access strictly to the agent working directory (_work). If malicious code is executed via an unreviewed pull request script, the attacker's blast radius is contained to the unprivileged service account.

Agent Pool Access Control in Azure DevOps

• Pool Isolation: Create separate agent pools for production deployments versus pull request builds.
• Pipeline Permissions: By default, new pipelines must be explicitly authorized by an administrator to use a restricted agent pool.
• Environment Approvals: Pair restricted self-hosted pools with Azure DevOps Environments that enforce manual approvals or business hour checks before deploying sensitive artifacts.

---

4. Comprehensive Decision Matrix: Hosted vs. Self-Hosted vs. VMSS

---

5. Custom Agent Images and VM Templates

A VMSS agent pool is only as reproducible as the image behind it, so "design and implement complex pipeline scenarios, including VM templates" resolves to a golden-image pipeline:

1. Author the image definition as code. A Packer template (or an Azure Image Builder imageTemplate ARM/Bicep resource) installs the agent prerequisites, SDKs, licensed compiler toolchains and security agents onto a base marketplace image.
2. Publish to an Azure Compute Gallery (formerly Shared Image Gallery). The gallery gives you image versions, replication to every region that hosts agents, and zone-redundant storage, which is what makes a rollback to last week's image a one-line change.
3. Point the scale set at the gallery image version and let Azure DevOps manage instance count. Because Azure DevOps re-images each VMSS instance after its job, every build starts from the exact image version you published.

resource agentImage 'Microsoft.VirtualMachineImages/imageTemplates@2024-02-01' = {
  name: 'build-agent-ubuntu-2204'
  location: location
  identity: { type: 'UserAssigned', userAssignedIdentities: { '${builderIdentityId}': {} } }
  properties: {
    source: { type: 'PlatformImage', publisher: 'canonical', offer: 'ubuntu-24_04-lts', sku: 'server', version: 'latest' }
    customize: [ { type: 'Shell', name: 'installToolchain', scriptUri: toolchainScriptUri } ]
    distribute: [ { type: 'SharedImage', galleryImageId: galleryImageId, runOutputName: 'agentImage', replicationRegions: [ 'eastus', 'westeurope' ] } ]
  }
}

The same pattern covers hybrid pipelines: a single YAML definition can target a Microsoft-hosted pool for cross-platform unit tests and a self-hosted or VMSS pool for the stages that need private connectivity, by setting pool: per job rather than once at the pipeline root.

---

6. Realistic Exam Scenario & Common Traps

Scenario: Healthcare Provider Hybrid CI/CD Architecture

Organization: Contoso Health systems operates an electronic medical records (EMR) application. Builds require running automated integration tests against an on-premises Oracle database containing synthetic test records. Corporate compliance mandates that:

1. The test runner must reside within the private healthcare network (no public IP).
2. Build environments must be completely pristine for every execution to comply with HIPAA data integrity rules.
3. The organization must avoid paying for idle compute during weekends, while supporting 40 concurrent builds during peak weekday mornings.

DevOps Solution:

• Deploy an Azure Virtual Machine Scale Set (VMSS) connected to an Azure VNet that peers with the on-premises datacenter via ExpressRoute.
• Register the VMSS as an agent pool in Azure DevOps.
• Set minCapacity: 0 and maxCapacity: 40.
• Enable automatic OS disk re-imaging after each job.
• Configure a custom Azure Compute Gallery image pre-installed with Oracle client libraries to ensure rapid job startup.

Common Exam Traps to Avoid

• Trap: Opening Inbound Firewall Port 443. Self-hosted agents poll outbound over HTTPS 443. Any exam answer that suggests configuring inbound port forwarding or opening inbound firewall rules to the agent machine is incorrect.
• Trap: Configuring Azure Monitor Autoscale Rules on a VMSS Agent Pool. Azure DevOps directly manages the scaling of VMSS agent instances based on queue depth. Enabling native Azure Autoscale rules on the scale set causes race conditions, premature VM termination, and corrupt builds.
• Trap: Running Self-Hosted Agents with Administrative Privileges. Running agents as root or LocalSystem introduces severe privilege escalation vulnerabilities. The correct approach is a dedicated, unprivileged service account.