9.2 Minimizing Downtime: Deployment Slots & Rolling Updates

9.2 Minimizing Downtime: Deployment Slots & Rolling Updates

Eliminating deployment downtime requires orchestration mechanisms at the infrastructure and application hosting layers that smoothly transition active workloads. In Microsoft Azure, two primary technologies govern zero-downtime execution across PaaS, IaaS, and CaaS workloads: Azure App Service Deployment Slots and Rolling Updates in Virtual Machine Scale Sets (VMSS) and Azure Kubernetes Service (AKS).

For the AZ-400 certification exam, candidates must understand the internal routing mechanics of deployment slots, configure application warmup to prevent cold-start penalties, differentiate sticky configuration settings from swappable parameters, execute advanced two-phase preview swaps, and configure Kubernetes rolling update parameters.

---

1. Azure App Service Deployment Slots Architecture

Deployment slots are live apps with their own hostnames, running within the same Azure App Service Plan as the primary production web application. They are available in the Standard, Premium, and Isolated App Service tiers.

                                  [Inbound User Traffic]
                                             │
                                             ▼
                              [Azure Front-End Routing Fabric]
                                             │
                                    (Routing Rules & VIP)
                         ┌───────────────────┴───────────────────┐
                         ▼                                       ▼
        [Production Slot: contoso.azurewebsites.net]   [Staging Slot: contoso-staging.azurewebsites.net]
        • 100% Production Traffic                      • 0% External Traffic (Isolated)
        • Live Code Version 1.0                        • Newly Deployed Code Version 2.0
        • Production DB Connection (Sticky)            • Staging DB Connection (Sticky)
                         │                                       │
                         └───────────────────┬───────────────────┘
                                             ▼
                              [Shared App Service Plan Compute]
                              (Shared CPU, RAM & Worker Cores)

Core Benefits of Deployment Slots

• Isolated Validation: Developers can deploy new container images or code packages to a non-production slot (e.g., staging) and execute comprehensive smoke tests, API integration tests, and security scans against real Azure infrastructure rather than a simulated local machine.
• Zero-Downtime Swapping: When promoting code from staging to production, Azure does not copy files or restart the production web worker processes. Instead, it reconfigures the virtual IP routing rules on the underlying Azure front-end load balancers, switching traffic instantaneously.
• Instant Fallback: If an issue is identified immediately after swapping, the team can swap back to return the previous version to production within seconds.
• Compute Resource Sharing Warning: Because deployment slots reside within the same App Service Plan, the staging slot consumes CPU, memory, and disk I/O from the exact same underlying VM instances. Running heavy performance or load tests against a staging slot can saturate system resources and cause degradation or outages in the production slot.

---

2. Anatomy of a Slot Swap Operation & Warmup Mechanics

Understanding the exact sequence of events during an App Service slot swap is essential for the AZ-400 exam.

[Step 1: Apply Target Settings] ──► Target slot (Staging) receives Production configuration
               │
               ▼
[Step 2: Warmup Pings]          ──► Azure sends HTTP requests to warmup paths on all instances
               │
               ▼
[Step 3: Health Probe Check]    ──► Wait for HTTP 200 responses; verify app is fully initialized
               │
               ▼
[Step 4: Front-End VIP Swap]    ──► Front-end routing redirects live traffic to warmed instances
               │
               ▼
[Step 5: Previous Prod Staged]  ──► Old production code is now running in Staging slot

What Happens During a Swap

1. Configuration Application: Azure applies the destination slot's settings (e.g., production settings) to the source slot (e.g., staging). This causes the staging worker process to restart and reload configuration.
2. Application Warmup: Azure issues HTTP requests to the root path (/) or to custom warmup paths defined in the application configuration on every scaled-out VM instance in the staging slot.
3. Readiness Confirmation: Azure waits for the worker instances to return successful HTTP status codes (200–299). If any instance fails the warmup probe, the swap operation is aborted, and production remains completely untouched.
4. Routing Redirection: Once all staging instances are fully warmed and initialized, the Azure front-end load balancer switches routing tables. Inbound requests to contoso.azurewebsites.net are now routed to the newly warmed worker instances.
5. Completion: The previous production version now runs in the staging slot (contoso-staging.azurewebsites.net), serving as an instant fallback.

Mitigating Cold Starts with Application Initialization

In JIT-compiled runtimes (such as ASP.NET Core or Java Spring Boot), the very first HTTP request to a newly launched process incurs significant latency as assemblies are compiled, dependency injection containers are wired, and database connection pools are established. If a swap occurs before initialization completes, initial users experience slow page loads or gateway timeouts.

To mitigate cold starts, developers configure the Application Initialization module in web.config:

<system.webServer>
  <applicationInitialization doAppInitAfterRestart="true">
    <add initializationPage="/health/warmup" hostName="contoso.azurewebsites.net" />
    <add initializationPage="/catalog/prime-cache" />
  </applicationInitialization>
</system.webServer>

Additionally, developers can configure specific Azure App Service application settings to govern warmup probes:

• WEBSITE_SWAP_WARMUP_PING_PATH: Specifies the exact relative URL path Azure pings to warm up the slot (e.g., /health/ready).
• WEBSITE_SWAP_WARMUP_PING_STATUSES: Specifies the comma-separated acceptable HTTP response codes (e.g., 200,202).

---

3. Slot Settings: Sticky vs. Swappable (Unpinned)

One of the most heavily tested topics on the AZ-400 exam is the distinction between Slot Settings (Sticky) and Standard Settings (Swappable).

[!IMPORTANT] By default, application settings and connection strings are unpinned (swappable)—meaning their values swap across environments during a swap operation. To anchor a setting to a specific slot, you must explicitly flag it as a Deployment Slot Setting (Sticky).

Categorization of Settings Behavior

The "Catastrophic Unpinned Database" Disaster

Consider an application where the database connection string DefaultConnection in the staging slot points to sql-test.database.windows.net, and in the production slot points to sql-prod.database.windows.net.

• If DefaultConnection is NOT marked as a slot setting, during the swap, the staging connection string is applied to production!
• As a result, the production site begins writing live customer orders into the test database, while the staging site connects to production data. Marking connection strings as Sticky (Slot Setting) prevents this disaster completely.

---

4. Advanced Slot Swap Patterns

Auto-Swap

Auto-swap streamlines continuous deployment by automatically triggering a slot swap as soon as new code or container images are pushed to a designated slot.

• Configuration: Set on the staging slot via the WEBSITE_DEFAULT_AUTO_SWAP_SLOT_NAME = production setting or through the Azure Portal.
• Workflow: A CI/CD pipeline pushes code to staging. Azure detects the completion of the deployment, automatically warms up the staging worker instances, and immediately swaps them into production.
• Constraint: Auto-swap is not supported if the application uses sticky configuration settings that require custom testing, or if interactive manual verification is required before cutover.

Swap with Preview (Two-Phase Swap)

When deploying major architecture refactors or framework upgrades, DevOps engineers need to test the new code running with production configuration settings before committing to live traffic cutover. Swap with Preview provides this capability through a two-phase process:

[Phase 1: Swap with Preview Triggered]
  • Azure applies Production configuration to Staging slot
  • Staging worker process restarts under Production settings
  • Live production traffic STILL points to original Production app
  • Engineers browse Staging slot to test against Production DB & Key Vault
               │
       ┌───────┴───────┐
       ▼               ▼
[Validation Succeeded]  [Validation Failed]
       │                       │
       ▼                       ▼
[Phase 2: Complete Swap] [Phase 2: Reset / Cancel Swap]
  • Instant VIP traffic cutover    • Revert Staging configuration
  • Zero user impact               • Production untouched

1. Phase 1 (Preview): Initiated via az webapp deployment slot swap --action preview. Azure applies production configuration settings (connection strings, Key Vault references) to the staging slot and warms it up. Production traffic continues flowing to the existing production app without interruption. Engineers and automated integration suites test contoso-staging.azurewebsites.net directly against production resources.
2. Phase 2 (Resolution):
   • Complete: Run az webapp deployment slot swap --action complete. Azure performs the traffic cutover.
   • Reset: If tests reveal defects, run az webapp deployment slot swap --action reset. Azure restores staging to its original configuration without production users ever knowing a swap was attempted.

Testing in Production (Traffic Routing Rules)

Azure App Service supports routing a configurable percentage of production traffic to a deployment slot:

# Route 15% of live traffic to the staging slot
az webapp traffic-routing set \
  --resource-group rg-contoso-prod \
  --name app-contoso \
  --distribution staging=15

• Session Stickiness: Azure manages user consistency by issuing a routing cookie named x-ms-routing-name. As long as the user retains this cookie, all subsequent HTTP requests route to the same slot, ensuring users do not flicker between versions during a single browsing session.

---

5. Rolling Deployments in VMSS and Azure Kubernetes Service (AKS)

In containerized and IaaS environments, zero downtime is achieved through Rolling Deployments, where instances or pods are replaced incrementally in batches.

Rolling Upgrades in Azure Virtual Machine Scale Sets (VMSS)

When updating the OS image or custom extension on a VMSS, the Rolling Upgrade Policy guarantees operational availability across fault domains:

• batchPercent: The maximum percentage of total VM instances updated in a single batch (e.g., 20%).
• maxUnhealthyPercent: The maximum percentage of instances that can be unhealthy simultaneously.
• maxUnhealthyUpgradedPercent: Maximum unhealthy instances allowed among upgraded instances before the upgrade halts.
• pauseTimeBetweenBatches: A soak time (e.g., PT5M for 5 minutes) allowing new VMs to pass load balancer health probes before the next batch begins.

Rolling Updates in Azure Kubernetes Service (AKS)

In Kubernetes, deployments manage rolling updates via the spec.strategy.rollingUpdate manifest block using two critical parameters:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
spec:
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1 # Maximum pods unavailable during update
      maxSurge: 2       # Maximum pods created above replica count
  template:
    metadata:
      labels:
        app: payment
    spec:
      containers:
        - name: payment
          image: contosoacr.azurecr.io/payment:v2.1
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5

• maxUnavailable: Defines the maximum number (or percentage) of pods that can be taken down during the update process. Setting maxUnavailable: 1 on 10 replicas ensures at least 9 pods are always running and serving traffic.
• maxSurge: Defines the maximum number (or percentage) of pods that can be created above the desired replica count. Setting maxSurge: 2 allows Kubernetes to spin up 2 new pods (total 12) while waiting for readiness probes to pass before terminating old pods.
• readinessProbe: Crucial exam rule! If a container lacks a readiness probe, Kubernetes assumes the pod is healthy the instant its process starts, routing traffic prematurely and causing request drops during rolling updates.

---

6. Azure CLI & YAML Pipeline Implementation

# 1. Create a staging deployment slot
az webapp deployment slot create \
  --name app-contoso-prod \
  --resource-group rg-contoso \
  --slot staging

# 2. Configure a database connection string as a STICKY slot setting
az webapp config connection-string set \
  --name app-contoso-prod \
  --resource-group rg-contoso \
  --slot staging \
  --connection-string-type SQLAzure \
  --settings DefaultConnection="Server=tcp:sql-staging.database.windows.net..." \
  --slot-settings DefaultConnection

# 3. Perform a Swap with Preview
az webapp deployment slot swap \
  --name app-contoso-prod \
  --resource-group rg-contoso \
  --slot staging \
  --action preview

# 4. Complete the swap following successful validation
az webapp deployment slot swap \
  --name app-contoso-prod \
  --resource-group rg-contoso \
  --slot staging \
  --action complete

# Azure Pipelines YAML Slot Swap Task
- task: AzureAppServiceManage@0
  displayName: 'Swap Staging Slot to Production'
  inputs:
    azureSubscription: 'ContosoServiceConnection'
    Action: 'Swap Slots'
    WebAppName: 'app-contoso-prod'
    ResourceGroupName: 'rg-contoso'
    SourceSlot: 'staging'
    SwapWithProduction: true

---

7. Realistic Exam Scenario & Common Traps

Scenario: Healthcare EHR Portal Deployment

Organization: A regional healthcare network hosts an Electronic Health Records (EHR) patient portal on an Azure App Service P3v3 plan. Deployments occur bi-weekly. During the last release, patient appointments could not be loaded for 4 minutes immediately following the deployment, and telemetry reported hundreds of HTTP 504 Gateway Timeouts.

DevOps Root Cause & Remediation:

• Root Cause: The deployment pushed code directly to production, forcing worker process recycling. Because the Java application required 90 seconds to initialize Hibernate entities and build cache schemas, incoming user requests hit uninitialized workers, causing 504 timeouts.
• Remediation:
  1. Deploy an Azure App Service deployment slot named staging.
  2. Configure WEBSITE_SWAP_WARMUP_PING_PATH pointing to /health/warmup.
  3. Configure database connection strings as Sticky Deployment Slot Settings.
  4. Update the Azure Pipelines release pipeline to deploy artifacts to staging, await warmup completion, and execute a slot swap. Zero user downtime is achieved.

Common Exam Traps to Avoid

• Trap: Assuming slot swaps copy files between slots. Slot swaps do not copy or sync code; they update load balancer routing rules. Files remain physically on their original worker instances.
• Trap: Forgetting to mark database connection strings as slot settings. If an exam scenario describes test database data appearing in production post-swap, the root cause is always failure to designate the setting as a sticky deployment slot setting.
• Trap: Using auto-swap when manual sign-off or two-phase preview testing is required. Auto-swap triggers immediately upon code upload. If testing against production data before cutover is mandated, choose Swap with Preview.