Deployment Strategies and Auto-Remediation

Improve the deployment process, then automate the healing

Task 3.1 knowledge includes CI/CD pipelines and deployment strategies (for example, blue/green, all-at-once, rolling) and alerting and automatic remediation. The matching skills are evaluating current deployment processes, prioritizing automation, and engineering failure scenario activities so recovery is practiced, not theoretical. Disaster recovery patterns (backup versus multi-site) live primarily in the resilience chapter; here you focus on how you ship and how the stack reacts when a ship goes bad.

AWS CodeDeploy is the service whose documented vocabulary matches the exam examples. Elastic Beanstalk, Amazon ECS rolling updates, Kubernetes rolling updates, and Lambda weighted aliases implement the same ideas with different knobs. Pick the strategy from blast radius and rollback speed, not from which logo is trendy.

All-at-once, rolling, canary, and blue/green

All-at-once updates every eligible target as fast as the platform allows. On the EC2/on-premises compute platform, CodeDeployDefault.AllAtOnce tries to deploy to as many instances as possible in parallel. AWS documents a sharp trap: if the revision reaches one or more instances, the overall deployment is successful; it fails only if no instance succeeds. Nine instances with eight failures and one success still “succeed.” For blue/green on that platform, traffic is rerouted to all replacement instances at once, and success similarly requires at least one successful reroute. Use all-at-once only for development, immutable cattle you can replace instantly, or Lambda where the previous version remains and an alarm can flip an alias back.

Rolling (in-place) updates a fraction of capacity at a time while the rest serves traffic. CodeDeployDefault.HalfAtATime deploys to at most half the instances (rounded down) and requires at least half to succeed. CodeDeployDefault.OneAtATime deploys to one instance at a time; for multi-instance groups the deployment can still succeed if only the last instance fails, because CodeDeploy allows a single host to be offline. If you do not specify a configuration on EC2/on-premises, CodeDeploy uses OneAtATime. Amazon ECS rolling updates use minimum and maximum healthy percent on the service (replace tasks in the same service) rather than a CodeDeploy traffic-shift config.

Canary exposes a small percentage of traffic to the new revision, waits, then shifts the rest if alarms stay OK. It is not the same as blue/green: canary is a partial production exposure; blue/green is a parallel environment with a switch (sometimes all at once, sometimes weighted). Predefined examples:

Blue/green deploys a replacement environment (new Auto Scaling group, new ECS task set, new Beanstalk environment) and shifts traffic with Elastic Load Balancing, CodeDeploy, or weighted Amazon Route 53. Rollback means shifting back to the still-running blue side. Bake time (ECS) keeps both revisions serving until you are confident. Lambda weighted aliases implement canary by splitting between two versions of the same function; AWS documents that this is canary, not a full blue/green fleet swap.

Always attach CloudWatch alarms to CodeDeploy deployment groups (or CodePipeline actions) so a rising 5xx rate or failed Synthetics canary stops the deployment and rolls back. A failed canary that you “finish” with AllAtOnce is how you take 100 percent of customers down after 10 percent already proved the revision is bad.

EventBridge, Lambda, SSM, and Config as triggers

Event-driven auto-remediation

Amazon EventBridge delivers a near-real-time stream of events from AWS services, partner sources, and custom buses. Rules filter those events and invoke targets: AWS Lambda, Systems Manager Automation, Step Functions, SNS, SQS, and API destinations. The AWS Well-Architected Framework reliability guidance describes using EventBridge (and CloudWatch alarms) to run custom remediation, and using EC2 Auto Scaling health checks to replace instances that are not running or have impaired system status. For large-scale replacements (an entire Availability Zone), static stability (pre-provisioned capacity) is preferred over scrambling to launch everything during the event.

Typical SAP-C02 auto-remediation loops:

1. CloudWatch alarm → SNS → Lambda that drains connections and calls StartAutomationExecution.
2. CloudWatch alarm → SSM OpsItem → operator or automatic runbook.
3. EventBridge rule on ECS Task State Change or GuardDuty findings → Lambda to isolate a task or snapshot a volume.
4. Auto Scaling replaces failed instances while the alarm still pages on SLO burn.

AWS Config evaluates resource configuration against rules. Automatic remediation runs a Systems Manager Automation document against noncompliant resources, optionally passing the resource ID as a parameter, with retries (AWS notes costs if a script retries many times). You manage remediations for non-service-linked rules. This chapter treats Config as a trigger into the same Automation plane as EventBridge. Rule catalogs, conformance packs, and data-retention/sensitivity design are in the later existing-security-performance chapter (config-patch-backup-compliance). Do not duplicate that depth here.

Failed canary scenario

A team ships an ECS service with CodeDeployDefault.ECSCanary10Percent5Minutes. After the first increment, Synthetics success percent drops and ALB 5xx rises on the green task set. Correct actions: stop the deployment, keep traffic on the original task set, roll back the revision, and inspect X-Ray plus Logs Insights using the canary’s trace IDs. Incorrect actions: switching to ECSAllAtOnce “to get it over with,” deleting the alarm that fired, or terminating the blue task set so rollback is impossible. After the fix, add the Synthetics metric to the deployment’s alarm set so CodeDeploy auto-rollback is guaranteed next time.

Game days and failure-scenario engineering

Task 3.1 skill: engineering failure scenario activities to support and exercise an understanding of recovery actions. A game day is a scheduled exercise that injects a realistic failure—Availability Zone impairment, exhausted RDS connections, a bad canary, a patch window that overruns cutoff, EventBridge rule disabled—and scores whether alarms, runbooks, and people meet the recovery objective. Use AWS Fault Injection Service (FIS) where it fits, or controlled CodeDeploy/Route 53 failovers. Write down expected composite-alarm behavior, Session Manager access if SSH is gone, and which Automation runbook is allowed in production versus in a clone account.

Game days are not chaotic production sabotage and not a substitute for never testing because “we have a wiki.” They are how operational excellence improves an existing solution: you discover that AllAtOnce is still on the production deployment group, that the canary has no alarm, or that the patch window’s error rate is so high it never pauses. Capture actions as Automation and EventBridge rules so the next incident does not require the same heroics.

When you evaluate a legacy pipeline on the exam, ask: Can we move from AllAtOnce to canary or rolling? Are alarms actually bound to the deploy? Does remediation loop through EventBridge/Lambda/SSM instead of email-only? Is there a practiced failure scenario? Those four questions are Domain 3 operational excellence in deployment clothing.