High Availability, Failover, Chaos, and Self-Healing

Fail small, fail tested, fail without a conference call

Quick Answer: Keep individual EC2 instances recoverable with simplified automatic recovery or CloudWatch action-based recovery. Keep applications recoverable with Multi-AZ, decoupled APIs (Amazon Simple Queue Service (Amazon SQS), Amazon Simple Notification Service (Amazon SNS), AWS Step Functions), and traffic shift (load balancers, zonal shift). Prove it with disaster recovery (DR) drills and AWS Fault Injection Service (AWS FIS). Automate what is safe; keep a runbook for what is not.

High availability (HA) is “survive the failure of a component without a Regional declaration of disaster.” Disaster recovery is “the Region, the account, or the data is gone.” SAP-C02 fails candidates who answer every stem with Aurora Global Database when the stem is a system status check on one instance, and who answer every stem with auto recovery when the stem is a poison message taking down a tightly coupled payments API.

Auto recovery is not architecture

Amazon EC2 system status checks watch the underlying host (network, power, hypervisor software). Instance status checks watch the guest (kernel hangs, exhausted network in the OS). Automatic instance recovery runs only when a system check fails, and only if you configured a mechanism before the failure.

Recovery looks like an unplanned reboot on a new host. If recovery fails, AWS may leave the instance on the bad host; the documented manual step is stop and start (new public IPv4 unless you use an Elastic IP). Metal sizes can use CloudWatch recovery where simplified recovery is excluded. Neither mechanism moves the instance to another AZ or Region. AWS’s own guidance after describing auto recovery is to put instances behind Elastic Load Balancing and Amazon EC2 Auto Scaling so traffic fails over to healthy instances instead of waiting on a single guest.

For existing fleets (Task 3.4), turning on CloudWatch recovery is a cheap win for dedicated instances that cannot yet be stateless. For new solutions (Task 2.4), do not design a singleton EC2 with recovery as the HA story.

Loosely coupled failovers

A tightly coupled payments API that HTTP POSTs to a ledger in the same AZ will time out when that AZ is impaired. The Well-Architected reliability skill is to fail over to healthy resources without making the caller wait on the impaired path.

• Amazon SQS buffers work. Standard queues maximize throughput and retries; FIFO queues preserve order and exactly-once processing within the window SQS documents. A dead-letter queue (DLQ) captures messages that exceed maxReceiveCount so poison payloads do not infinite-loop. The API’s RTO becomes “accept the message,” not “commit the ledger in-line.”
• Amazon SNS fans out the same event to SQS queues, Lambda, HTTPS, and more. That is how you notify fraud, analytics, and customer-email without chaining those teams on the authorization hot path. Amazon EventBridge is the richer event bus when you need archive/replay and schema discovery; SNS remains the simple pub/sub primitive the exam still uses heavily.
• AWS Step Functions orchestrates retries, backoff, catchers, and compensating transactions. A ledger write that fails after SQS delivery becomes a state-machine retry with a human approval state if the compensating path needs it. Step Functions is how you encode the runbook that used to live in a wiki.

New solutions should not introduce a synchronous chain of three Regional APIs. Existing solutions can insert a queue in front of the most failure-prone hop without rewriting the ledger. That is Task 3.4: improve reliability of what already runs.

Auto Scaling, SQS visibility timeouts, and Step Functions retries are self-healing. They still fail when:

• The payload is wrong (retrying a double-charge).
• Both Regions apply the same bad schema (replication of corruption).
• Quotas block scale-out.
• Split-brain writers appear after a dual-Region failover without fencing.

Those cases need runbooks: who calls switchover-global-cluster versus unplanned failover, who updates ARC routing controls, who halts writers, who restores from a vault-locked backup instead of promoting a replica that contains ransomware.

DR testing and failure-scenario exercises

Untested RTO is fiction. AWS’s DR whitepaper and DRS product pages both assume non-disruptive drills. Organizational Task 1.3 work is a game-day calendar, blast-radius rules, and a rule that production experiments need stop conditions.

Exercise catalog that maps to this chapter:

1. AZ impairment — shift traffic (zonal shift) or inject faults (FIS). Remaining AZs must already have capacity.
2. Regional evacuation — routing control plus database promotion. Measure actual RPO (replica lag) and RTO (DNS TTL + boot + queue drain).
3. Data corruption / ransomware — restore from a point-in-time backup in a locked or air-gapped vault, not from the infected replica.
4. Quota exhaustion — attempt scale-out in the DR Region on a normal Tuesday.
5. Control-plane dependence — try to deploy CloudFormation during a simulated console outage; discover you needed pre-provisioned capacity.

AWS Fault Injection Service (current)

AWS FIS (AWS Fault Injection Service) runs real actions on real resources. AWS tells you to plan in pre-production before production. You author an experiment template: actions, targets, and stop conditions (CloudWatch alarms that abort the experiment). Targets can be selected by resource ID, tags, or filters (including AZ). Selection modes include ALL, COUNT(n), and PERCENT(n). Multi-account experiments can target resources in other accounts. Logging can go to CloudWatch Logs or S3. You are billed per action-minute and by number of target accounts (see current FIS pricing; do not memorize a dollar figure AWS can change).

The Scenario Library (console) includes AZ Availability: Power Interruption, which AWS documents as interrupting compute, networking, storage, and managed-service behavior in a chosen AZ—EC2/Auto Scaling capacity errors, subnet disruption, EBS, RDS clusters (writer AZ), ElastiCache power interruption, and related actions. Shared parameters such as affectedAz keep the AZ consistent across actions. Use stop conditions so a runaway experiment does not become an unplanned incident.

FIS is appropriate on SAP-C02 when the stem wants evidence that Multi-AZ works. It is the wrong answer when the stem wants point-in-time restore (that is Backup) or block replication (that is DRS).

Self-healing versus runbook

Payments game day

Existing authorization API still calls the ledger with synchronous HTTPS. Multi-AZ RDS is on. Nobody has failed an AZ on purpose. Task 3.4 plan: (1) put authorize-requests on SQS with a DLQ and idempotent consumers; (2) run FIS AZ Availability: Power Interruption in the staging account with stop conditions; (3) enable zonal shift on the ALB after prescaling; (4) table-top the Regional runbook (ARC routing control + Aurora failover) without executing it in production that day; (5) write the runbook step that says “do not promote a replica if integrity alarms fired—restore from the locked vault.” Self-healing covers the AZ. The runbook covers the day replication lies to you.