7.1 Designing Resilient Multi-AZ & Multi-Region Architectures

High Availability Foundations & Availability Mathematics

Designing resilient cloud infrastructure for the AWS Certified DevOps Engineer - Professional (DOP-C02) exam requires a rigorous, mathematical understanding of system reliability, disaster recovery objectives, and component dependency modeling.

Core Reliability Metrics

• RTO (Recovery Time Objective): The targeted duration of time between system disruption and service restoration. RTO defines the maximum acceptable operational downtime.
• RPO (Recovery Point Objective): The maximum acceptable age of data that can be lost when an unplanned outage occurs, determined by data replication intervals and transaction log flushing.
• MTBF (Mean Time Between Failures): The average operational time between inherent system failures during normal operation. MTBF measures component and architectural reliability.
• MTTR (Mean Time to Repair): The average time required to troubleshoot, repair, and recover a failed system or component back to healthy operation.

Availability = MTBF / (MTBF + MTTR)

Composite SLA Mathematics

Cloud applications are composed of interconnected services (e.g., CloudFront, Application Load Balancer, EC2 Auto Scaling, RDS Multi-AZ). Each AWS service publishes an official Service Level Agreement (SLA). When calculating the composite SLA of a distributed application, DevOps engineers must differentiate between serial dependencies and parallel (redundant) components.

Serial System Dependencies

If a transaction requires N components in sequence to succeed, failure of any single component breaks the transaction. The composite availability is the product of each individual component's availability:

Availability_serial = A_1 * A_2 * A_3 * ... * A_N

Consider an application consisting of Amazon CloudFront (99.9%), an Application Load Balancer (99.99%), Amazon EC2 compute instances (99.99%), and an Amazon RDS Multi-AZ Database (99.95%):

Availability_serial = 0.999 * 0.9999 * 0.9999 * 0.9995 = 0.998301 (99.83%)

Even though individual components provide four nines (99.99%), the composite serial architecture drops below three nines, allowing approximately 14.88 hours of downtime per year.

Parallel Redundant Architectures

When independent, redundant paths or regions run in parallel (active-active or hot-standby), the entire system fails only if all parallel paths fail simultaneously. The composite availability is calculated by multiplying the unavailabilities:

Availability_parallel = 1 - [(1 - A_1) * (1 - A_2) * ... * (1 - A_M)]

Deploying an application across two independent AWS Regions, each with an isolated regional composite SLA of 99.9% (A_1 = A_2 = 0.999):

Availability_parallel = 1 - (1 - 0.999) * (1 - 0.999) = 1 - (0.001 * 0.001) = 1 - 0.000001 = 0.999999 (99.9999%)

Parallel regional redundancy elevates a 99.9% workload to six nines of theoretical availability, reducing annualized maximum unrecoverable downtime to just 31.5 seconds.

---

Multi-AZ Compute Capacity Planning & Auto Scaling

Designing a Multi-AZ compute tier on Amazon EC2 requires sizing Auto Scaling groups (ASGs) to absorb the total loss of a single Availability Zone without performance degradation or capacity starvation.

The N+1 Multi-AZ Sizing Formula

To ensure an application requiring a minimum of C healthy instances can withstand the failure of an entire Availability Zone without performance degradation, the required capacity per Availability Zone is calculated as:

Capacity per AZ = ceil( C / (N - 1) )

where N is the number of active Availability Zones configured in the VPC subnets.

[!IMPORTANT] DOP-C02 Exam Trap: Deploying across 3 Availability Zones is the architectural standard for cost-effective enterprise resilience. Sizing across 3 AZs requires running at only 150% baseline capacity to achieve complete single-AZ failure tolerance, whereas a 2-AZ topology requires running at 200% capacity (doubling baseline infrastructure costs).

Auto Scaling AZRebalance Mechanics and Suspension

Amazon EC2 Auto Scaling groups maintain balance across configured Availability Zones by default via the AZRebalance scaling process. When an AZ becomes unhealthy or when instances are terminated unevenly, AZRebalance acts:

1. Launch Before Terminate: AZRebalance launches a new EC2 instance in the underrepresented AZ first.
2. Balance Evaluation: Once the new instance is healthy and registered with target groups, AZRebalance terminates an instance in the overrepresented AZ.

However, during active deployments, large-scale batch processing, or stateful session draining, AZRebalance can disrupt workloads by terminating running instances unexpectedly. DevOps engineers can suspend the process via the AWS CLI:

aws autoscaling suspend-processes \
    --auto-scaling-group-name prod-fleet-asg \
    --scaling-processes AZRebalance

Suspending AZRebalance leaves Launch, Terminate, HealthCheck, and AlarmNotification active, allowing the ASG to scale out or replace unhealthy instances without shifting healthy nodes between zones.

---

Multi-AZ State Management: RDS Multi-AZ vs. Read Replicas

Stateful database tiers require distinct architectural strategies depending on whether the objective is high availability (HA) or read scalability.

[ Primary Database Instance (AZ-A) ]
       |
       +----- Synchronous EBS Physical Replication -----> [ Passive Standby (AZ-B) ]
       |      (Zero Data Loss: RPO = 0)                     (Automatic CNAME Failover: 60-120s)
       |
       +----- Asynchronous Engine Replication (WAL/Binlog) -> [ Read Replica (AZ-C) ]
              (Replication Lag > 0: RPO > 0)                  (Active Read-Only Traffic)

---

Elastic Load Balancing (ELB) Resiliency Dynamics

Elastic Load Balancing distributes incoming application traffic across multiple targets. Choosing between Application Load Balancers (ALB) and Network Load Balancers (NLB) depends directly on network layer protocols, health checking granularity, and cross-zone load balancing characteristics.

ALB vs. NLB Resiliency Comparison

• Application Load Balancer (Layer 7): Evaluates HTTP/HTTPS request headers, paths, cookies, and methods. Cross-zone load balancing is enabled by default at the load balancer level and cannot be turned off there; it can, however, be turned off for an individual target group by setting the target group attribute load_balancing.cross_zone.enabled to false. ALB scales elastically using dynamic IP addresses per AZ.
• Network Load Balancer (Layer 4): Operates at the transport layer (TCP, UDP, TLS), routing millions of requests per second at ultra-low sub-millisecond latencies. NLB provides a static IP address per enabled AZ and can use one Elastic IP per AZ for an internet-facing load balancer. Anycast addressing is provided by AWS Global Accelerator, not by NLB itself. Cross-zone load balancing is disabled by default.

Cross-Zone Load Balancing Behavior & Cost Implications

When cross-zone load balancing is disabled, each load balancer node distributes traffic only to registered targets within its own local Availability Zone. If client traffic or DNS resolution distributes requests evenly (50/50) between two AZs, but AZ-A contains 8 instances while AZ-B contains 2 instances, instances in AZ-B experience four times the load of instances in AZ-A.

[ Client Traffic (100 Requests) ]
         |
         +------ 50 Requests (50%) ------> [ NLB Node AZ-A ] --> 10 Instances (5 reqs/instance)
         |
         +------ 50 Requests (50%) ------> [ NLB Node AZ-B ] --> 2 Instances  (25 reqs/instance) <-- SKEWED!

When Cross-Zone Load Balancing is ENABLED on NLB:
[ NLB Node AZ-A ] ----- Cross-AZ Routing ---> Balanced evenly across all 12 instances (~8.3 reqs/instance)

[!TIP] Enabling cross-zone load balancing on a Network Load Balancer distributes connections evenly across all healthy targets across all enabled AZs, preventing localized target saturation. However, cross-AZ traffic routed across NLB nodes incurs standard inter-AZ regional data transfer charges ($0.01/GB). On ALBs, cross-zone data transfer is always included at no extra charge.

Health Check Algorithms & Target Deregistration Delay

Target groups continuously assess target health using active polling:

• Healthy Threshold: Consecutive successful health checks required to transition a target from unhealthy to healthy (default: 5 for ALB, 3 for NLB).
• Unhealthy Threshold: Consecutive failed checks required to mark a target unhealthy (default: 2 for ALB, 3 for NLB).
• Deregistration Delay (Connection Draining): The duration (0–3600 seconds; 300 seconds default) ELB keeps existing in-flight connections open while stopping the transmission of new requests to a deregistered or terminating target. In CI/CD pipelines deploying ephemeral containers, reducing deregistration delay to 15–30 seconds prevents deployment stalls.

---

AWS Global Accelerator vs. CloudFront for Multi-Region Resiliency

While Amazon Route 53 handles multi-region routing via DNS, DNS-based failover is fundamentally throttled by DNS TTL caching on client devices, recursive resolvers, and corporate proxies. AWS Global Accelerator provides static global entry addresses and health-check-driven endpoint routing without changing DNS records.

Architectural Mechanics of Global Accelerator

1. Static Anycast IPv4 Addresses: Global Accelerator provides two static Anycast IP addresses that serve as single points of entry for global client traffic.
2. BGP Anycast Routing: Anycast routes client connections to the geographically closest AWS Edge Location (Point of Presence).
3. AWS Global Backbone Transit: From the edge location, traffic enters the AWS global network rather than remaining on the public internet for the full path.
4. Health-Check-Driven Failover: For endpoints that Global Accelerator checks directly, configure a 10- or 30-second health-check interval and an appropriate threshold. After an endpoint becomes unhealthy, new connections are directed to healthy endpoints without waiting for a DNS record change. Detection time depends on the interval, threshold, endpoint type, and propagation; AWS does not state a hard sub-30-second end-to-end SLA.
5. Client IP Preservation: Preserves the client's original IP address across NLB and ALB endpoints without requiring X-Forwarded-For header parsing.