9.5 Ordered Dependency Deployments, Resiliency & Rollback

9.5 Ordered Dependency Deployments, Resiliency & Rollback

A hotfix fixes one component. A release usually touches several, and the pipeline itself has to encode the ordering contract between them - then survive the case where one of those stages fails in production.

1. Dependency-Ordered Pipeline Deployments

In complex microservices and distributed cloud architectures, applications cannot be deployed simultaneously in random order. Deploying a front-end API gateway before the back-end services are available causes immediate client 502 Bad Gateway errors; deploying a microservice before its database schema exists causes startup crashes.

DevOps architects model these relationships using Topological Dependency Ordering in Azure Pipelines multi-stage YAML definitions via the dependsOn keyword.

                        [Stage 1: Foundation Infrastructure]
                        (VNet, Key Vault, Service Bus Topics)
                                          │
                                          ▼
                        [Stage 2: Database Schema Migration]
                        (Backward-Compatible Expand Script)
                                          │
                     ┌────────────────────┴────────────────────┐
                     ▼                                         ▼
       [Stage 3A: Billing Microservice]          [Stage 3B: Order Microservice]
                     │                                         │
                     └────────────────────┬────────────────────┘
                                          ▼
                        [Stage 4: API Management Gateway]
                        (Update Routing Rules to New Service Endpoints)

Stage Dependency Rules in Azure Pipelines

• Sequential Execution: Declaring dependsOn: StageA ensures StageB will not start until StageA completes successfully.
• Parallel Execution (Fan-Out): If multiple stages declare dependsOn: Stage2 (like BillingService and OrderService above), Azure Pipelines runs them concurrently across available agents, reducing overall pipeline execution time.
• Synchronization (Fan-In): Stage 4 declares dependsOn: [BillingService, OrderService]. It waits until both parallel microservice deployments finish before updating the API Gateway.
• Conditions: By default, dependent stages only run if previous stages succeed (condition: succeeded()). If the database migration fails, all subsequent application stages are aborted automatically.

---

2. Resilient Pipelines, Health Gates & Automated Rollbacks

Automated rollbacks are essential for maintaining high availability SLAs. Relying on an on-call engineer to notice a failure, log into the Azure portal, and manually execute a rollback during an off-hours incident increases MTTR from seconds to tens of minutes.

Health Check Probes in Pipeline Environments

Azure DevOps YAML multi-stage pipelines integrate with Environments that enforce automated Checks & Approvals:

1. Azure Monitor Alerts Gate: The pipeline queries active Azure Monitor alerts. If any Sev1 or Sev2 alerts (e.g., HTTP 500 error spike or database DTU saturation) fire during or immediately after deployment, the environment gate fails.
2. REST API Health Check Gate: The pipeline issues recurring HTTP GET requests to the application's /healthz or /ready endpoint, confirming deep dependency reachability (SQL connection pool, Redis cache ping, message queue access).

Automated Rollback Stage Architecture

When an error occurs during a multi-stage deployment, Azure Pipelines supports automated recovery stages governed by the condition: failed() expression:

stages:
  - stage: DeployToStaging
    displayName: 'Deploy to Staging Slot'
    jobs:
      - deployment: DeployWeb
        environment: 'Production-Staging'
        strategy:
          runOnce:
            deploy:
              steps:
                - task: AzureWebAppDeployment@1
                  inputs:
                    appName: 'app-contoso-core'
                    deployToSlotOrASE: true
                    resourceGroupName: 'rg-contoso'
                    slotName: 'staging'
                    package: '$(Pipeline.Workspace)/drop/*.zip'

  - stage: SwapAndValidate
    displayName: 'Swap Slots and Evaluate Health'
    dependsOn: DeployToStaging
    jobs:
      - job: ExecuteSwap
        steps:
          - task: AzureAppServiceManage@0
            displayName: 'Swap Staging to Production'
            inputs:
              Action: 'Swap Slots'
              WebAppName: 'app-contoso-core'
              ResourceGroupName: 'rg-contoso'
              SourceSlot: 'staging'
              SwapWithProduction: true

  - stage: AutomatedRollback
    displayName: 'Automate Rollback on Deployment Failure'
    dependsOn: SwapAndValidate
    condition: failed() # Only executes if SwapAndValidate fails!
    jobs:
      - job: RevertSwap
        steps:
          - task: AzureAppServiceManage@0
            displayName: 'Revert Production Slot Swap'
            inputs:
              Action: 'Swap Slots'
              WebAppName: 'app-contoso-core'
              ResourceGroupName: 'rg-contoso'
              SourceSlot: 'staging'
              SwapWithProduction: true
          - script: |
              echo "ALERT: Production deployment failed health validation. Automated slot swap reversal executed."

If the validation steps in SwapAndValidate fail or report an unhealthy state, Azure Pipelines immediately activates AutomatedRollback, executing a reverse slot swap that restores the previous working version within seconds.

---

3. Circuit Breakers, Bulkheads & Graceful Degradation

Modern cloud-native resilience extends beyond CI/CD pipelines into application architecture. During rolling deployments or partial infrastructure failovers, distributed systems face transient faults.

[Client Request] ──► [API Gateway]
                           │
                           ▼
        [Circuit Breaker State Machine (Polly)]
        ┌────────────────────────────────────────────────────────┐
        │ • CLOSED: Normal operation; requests forwarded         │
        │ • OPEN: Failure threshold exceeded; fast-fail fallback  │
        │ • HALF-OPEN: Trial requests test if downstream healed  │
        └────────────────────────────────────────────────────────┘
                           │
                           ▼
        [Downstream Microservice (Undergoing Deployment)]

The Circuit Breaker Pattern

Implemented via libraries such as Polly (.NET) or Resilience4j (Java), the circuit breaker prevents an application from repeatedly executing operations that are likely to fail:

• Closed: Requests pass through normally. The breaker tracks recent failures.
• Open: If the failure rate breaches a threshold (e.g., 50% failures over 10 seconds), the circuit "trips" open. Subsequent requests fail immediately without hitting the struggling downstream service. The application returns a cached response, degraded data, or a fallback message. This prevents thread starvation and database connection pool exhaustion.
• Half-Open: After a configured sleep duration, the breaker allows a limited number of trial requests through. If they succeed, the circuit resets to Closed; if they fail, it trips back to Open.

---

4. Realistic Exam Scenario & Common Traps

Scenario: Global Airline Booking Engine Failure

Organization: An international airline deploys an emergency hotfix to resolve a seat reservation glitch in their production booking service (v4.1.0).

• Incident Sequence:
  1. The on-call engineer fixes the code on a branch created from main, rather than the production release branch.
  2. In the rush, the engineer disables database migration dependency ordering, triggering the API gateway update and the SQL schema script simultaneously.
  3. When deployed, the API Gateway immediately routes live traffic to the new container pods before the SQL script finishes, causing 10,000 reservation requests to fail with database constraint errors.
  4. Three weeks later, after manually fixing production, the team deploys scheduled release v4.2.0 from main. Because the original hotfix was never merged back into main properly, the original seat reservation glitch re-appears, stranding thousands of passengers.

DevOps Solution Required:

1. Branching: Branch emergency fixes exclusively from the production release tag (release/v4.1.0).
2. Topological Ordering: Structure the multi-stage pipeline so that microservices declare dependsOn: DatabaseMigration, guaranteeing that schema updates finish before web containers start.
3. Automated Rollback: Add a rollback stage with condition: failed() to swap deployment slots back if post-swap health probes fail.
4. Mandatory Merge-Back: Mandate a pull request that cherry-picks the hotfix commit into main before closing the incident ticket.

Common Exam Traps to Avoid

• Trap: Branching emergency hotfixes from the main branch. main contains unreleased sprint code. Always branch hotfixes from the production release branch or tag.
• Trap: Forgetting to merge hotfixes back to main. Deploying a hotfix to a release branch without cherry-picking or merging it back to main guarantees a regression in the next general release.
• Trap: Conflating manual approvals with automated rollback gates. Manual approvals pause pipelines before deployment. Automated rollbacks evaluate system telemetry after deployment and trigger recovery scripts automatically when alerts breach.