3.4 Blue/Green & Canary Deployments for ECS, EKS & Lambda

Progressive Delivery Overview

Progressive delivery builds upon continuous delivery by gradually rolling out new software versions while measuring real-time telemetry and error budgets. In microservices and serverless architectures, deploying updates all at once exposes the entire customer base to latent regressions. For the DOP-C02 exam, you must master the mechanics of progressive delivery across three compute platforms:

1. Amazon Elastic Container Service (ECS): Managed blue/green deployments using AWS CodeDeploy and Application Load Balancer dual-listener routing.
2. AWS Lambda: Alias traffic shifting using AWS Serverless Application Model (SAM) and CodeDeploy lifecycle hooks.
3. Amazon Elastic Kubernetes Service (EKS): Native rolling updates with readiness probes and progressive delivery operators (Argo Rollouts / Flagger).

---

Amazon ECS Blue/Green Deployments with AWS CodeDeploy

To perform automated blue/green deployments on Amazon ECS, the ECS service must be configured with the CODE_DEPLOY deployment controller rather than the default ECS rolling controller.

Application Load Balancer Architecture Requirements

An ECS blue/green deployment orchestrated by CodeDeploy requires a specific Application Load Balancer (ALB) network topology:

                                +---------------------------------+
                                |    Application Load Balancer    |
                                +----------------+----------------+
                                                 | 
                         +-----------------------+-----------------------+
                         |                                               |
                         v                                               v
               +--------------------+                          +-------------------+
               | Production Listener|                          |   Test Listener   |
               |    (Port 443)      |                          |    (Port 8443)    |
               +---------+----------+                          +---------+---------+
                         |                                               |
       [Initial State: 100% Traffic]                           [Test Traffic: Synthetic Validation]
                         |                                               |
                         v                                               v
               +--------------------+                          +-------------------+
               |   Target Group 1   |                          |   Target Group 2  |
               |    (Blue / Old)    |                          |   (Green / New)   |
               +---------+----------+                          +---------+---------+
                         |                                               |
                         v                                               v
               +--------------------+                          +-------------------+
               | Original Task Set  |                          |Replacement TaskSet|
               |  (ECS Revision 1)  |                          | (ECS Revision 2)  |
               +--------------------+                          +-------------------+

1. Two Target Groups: TargetGroup1 (Blue) and TargetGroup2 (Green). At any given time, one receives production traffic while the other stands by or receives test traffic.
2. Production Listener (e.g., Port 443): Directs live customer traffic to the current active production target group.
3. Test Listener (e.g., Port 8443): Directs internal verification traffic exclusively to the replacement target group before production traffic is shifted.

ECS Deployment Workflow & Lifecycle Hooks

When a new ECS Task Definition revision is registered, the CodeDeploy workflow executes as follows:

1. Create Replacement Task Set: CodeDeploy launches the desired count of replacement (green) container tasks.
2. Route Test Traffic (AfterAllowTestTraffic): The test listener on port 8443 is pointed to the green target group. CodeDeploy triggers an optional Lambda function hook (AfterAllowTestTraffic) to run automated synthetic integration tests against port 8443.
3. Shift Production Traffic: If tests succeed, CodeDeploy initiates production traffic shifting on the main listener (port 443) according to the configured traffic routing strategy.
4. Post-Traffic Validation (AfterAllowTraffic): After 100% of production traffic is routed to the green target group, a second Lambda hook runs final validation.
5. Blue Task Set Termination: The original blue task set is kept alive for a configurable termination wait time (e.g., 60 minutes) to allow instant, zero-downtime rollback before tasks are permanently drained and terminated.

Traffic Shifting Configurations for ECS

• CodeDeployDefault.ECSAllAtOnce: 100% of traffic shifts immediately from Blue to Green.
• CodeDeployDefault.ECSLinear10PercentEvery1Minute: Traffic shifts 10% each minute over 10 equal increments.
• CodeDeployDefault.ECSCanary10Percent5Minutes: Shifts 10% of traffic initially, waits 5 minutes to monitor CloudWatch alarms, then shifts the remaining 90% in a single batch.

ECS appspec.yaml Definition

version: 0.0
Resources:
  - TargetService:
      Type: AWS::ECS::Service
      Properties:
        TaskDefinition: "arn:aws:ecs:us-east-1:123456789012:task-definition/payment-api:14"
        LoadBalancerInfo:
          ContainerName: "payment-container"
          ContainerPort: 8080
        CapacityProviderStrategy:
          - CapacityProvider: "FARGATE"
            Weight: 1
Hooks:
  - AfterAllowTestTraffic: "arn:aws:lambda:us-east-1:123456789012:function:ValidatePaymentGreenTask"
  - AfterAllowTraffic: "arn:aws:lambda:us-east-1:123456789012:function:PostDeploymentSanityCheck"

---

Serverless Progressive Deployments with AWS Lambda & AWS SAM

When deploying serverless microservices, deploying new AWS Lambda code directly to an unqualified function ARN modifies execution instantly. AWS Lambda Version Aliases enable progressive delivery by shifting traffic weights between two immutable published versions (e.g., 90% to Version 1, 10% to Version 2).

AWS SAM DeploymentPreference Syntax

Using the AWS Serverless Application Model (SAM), alias traffic shifting is configured declaratively under DeploymentPreference:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Resources:
  OrderProcessingFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: index.handler
      Runtime: nodejs20.x
      CodeUri: ./src
      AutoPublishAlias: live
      DeploymentPreference:
        Type: Canary10Percent10Minutes
        Alarms:
          - !Ref OrderFunction5xxErrorsAlarm
          - !Ref OrderFunctionLatencyAlarm
        Hooks:
          PreTraffic: !Ref ValidateOrderProcessingHook
          PostTraffic: !Ref PostDeploymentValidationHook

Lifecycle Validation Hooks Contract (putLifecycleEventHookExecutionStatus)

The PreTraffic and PostTraffic hooks reference independent AWS Lambda functions that perform health validation:

• PreTraffic Hook: Executes before any traffic shifts to the new version. Tests database migrations, cache warming, or schema compatibility.
• PostTraffic Hook: Executes after 100% of traffic has shifted to the new version.

Exam Trap: A hook Lambda function MUST explicitly call the AWS CodeDeploy API PutLifecycleEventHookExecutionStatus returning either Succeeded or Failed. If this API call is omitted, CodeDeploy blocks waiting for a status until it reaches the default 1-hour timeout and fails the deployment!

const { CodeDeployClient, PutLifecycleEventHookExecutionStatusCommand } = require("@aws-sdk/client-codedeploy");
const codedeploy = new CodeDeployClient();

exports.handler = async (event) => {
  const deploymentId = event.DeploymentId;
  const lifecycleEventHookExecutionId = event.LifecycleEventHookExecutionId;
  
  let status = 'Succeeded';
  try {
    // Execute synthetic integration test against newly published Lambda version
    await runSmokeTests();
  } catch (err) {
    console.error("Smoke tests failed!", err);
    status = 'Failed';
  }
  
  const params = {
    deploymentId: deploymentId,
    lifecycleEventHookExecutionId: lifecycleEventHookExecutionId,
    status: status
  };
  
  await codedeploy.send(new PutLifecycleEventHookExecutionStatusCommand(params));
};

---

Kubernetes / Amazon EKS Progressive Delivery Patterns

In Amazon EKS, deployments can be orchestrated using either native Kubernetes rolling update semantics or advanced progressive delivery operators.

Native Kubernetes Rolling Updates

Kubernetes Deployment resources manage pod replacements using the RollingUpdate strategy controlled by two parameters:

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0

• maxSurge: The maximum number of pods that can be created above the desired replica count during an update (e.g., 25% or absolute count 2).
• maxUnavailable: The maximum number of pods that can be unavailable during the process. Setting maxUnavailable: 0 ensures that 100% of desired application capacity is maintained at all times.

Health Probes

Rolling updates rely on container Readiness Probes (readinessProbe). Until a newly launched pod passes its readiness probe, the Kubernetes endpoint controller refrains from adding the pod IP to the Service endpoints, ensuring live traffic is never routed to uninitialized containers.

Advanced Progressive Delivery: Argo Rollouts & Flagger

For enterprise progressive delivery in EKS, native rolling updates lack automated traffic shifting and automated metric analysis. Modern EKS platforms adopt Argo Rollouts or Flagger:

• Argo Rollouts: Replaces the Kubernetes Deployment object with a custom Rollout CRD. Integrates with the AWS Load Balancer Controller (TargetGroupBinding) or ingress controllers to execute canary shifting (e.g., 5% -> 20% -> 50% -> 100%).
• Automated Metric Analysis: Queries Prometheus or Amazon CloudWatch during each canary step. If HTTP 5xx error rates exceed thresholds or latency spikes, the controller automatically halts the rollout and rolls back traffic instantly without human intervention.

---

Progressive Delivery Decision Matrix