7.2 Route 53 Traffic Flow, Health Checks & DNS Failover

Amazon Route 53 Routing Policies Overview

Amazon Route 53 is a highly available and scalable Domain Name System (DNS) web service. Beyond basic domain registration and zone management, Route 53 acts as an intelligent traffic routing engine for distributed cloud architectures. Selecting the appropriate routing policy is critical for meeting performance, compliance, and disaster recovery objectives.

Comprehensive Routing Policy Comparison

Geolocation vs. Geoproximity with Traffic Flow Bias

A critical distinction tested on the DOP-C02 exam is the difference between Geolocation and Geoproximity routing:

• Geolocation Routing: Routes traffic based strictly on static geographic boundaries (e.g., all queries originating from Europe route to eu-central-1; queries from California route to us-west-1). If an organization needs to ensure German user data stays in Frankfurt for legal compliance, Geolocation routing is mandatory.
• Geoproximity Routing: Calculates the physical distance between the client's IP location and the AWS region or on-premises coordinates. Geoproximity introduces a configurable parameter called bias (ranging from -99 to +99):
  • A positive bias expands the geographic region served by a resource, drawing traffic away from neighboring regions.
  • A negative bias shrinks the geographic region served by a resource, shedding traffic to adjacent regions.
  • Geoproximity routing requires using Route 53 Traffic Flow visual rules.

[ US-West Region: Bias = -25 (Shrunk) ] <===== Boundary Shift =====> [ US-East Region: Bias = +25 (Expanded) ]
                     |                                                                   |
                     +------------- Shifts Eastern/Midwest Traffic Eastward -------------+

---

Route 53 Health Checking Mechanics

Route 53 health checks continuously monitor the availability and performance of application endpoints. Approximately 15 global health checker nodes probe endpoints from various geographic vantage points across the world.

Endpoint Health Checks

Endpoint health checks monitor an IP address or fully qualified domain name (FQDN) using three protocols:

• HTTP / HTTPS: Route 53 must establish a TCP connection within 4 seconds, and the endpoint must return an HTTP 2xx or 3xx status code within 2 seconds of connecting.
• TCP: Completes a standard TCP three-way handshake within 10 seconds.

Key configuration parameters include:

• Request Interval: Standard (30 seconds) or Fast (10 seconds). Fast health checks incur additional cost but detect outages three times faster.
• Failure Threshold: The number of consecutive health check failures (1 to 10; default: 3) required to mark an endpoint unhealthy. At standard intervals (30s) with a threshold of 3, Route 53 takes between 60 and 90 seconds to transition a record to unhealthy.
• String Matching: Inspects the first 5,120 bytes of the HTTP/HTTPS response body for a specific expected string. If the application server encounters an internal database deadlock and outputs an error page with an HTTP 200 OK header, a string matching check searching for "SYSTEM_HEALTH_OK" will correctly identify the failure and mark the endpoint unhealthy.
• Inverted Health Checks: Inverts the pass/fail logic. If an endpoint is reachable, the health check reports unhealthy. This is frequently used for dead-man switches or maintenance banner routing.

Calculated Health Checks

A Calculated Health Check does not monitor an endpoint directly; instead, it aggregates the statuses of up to 256 individual health checks using boolean logic:

1. AND Logic: All child health checks must be healthy for the calculated check to be healthy.
2. OR Logic: At least one child health check must be healthy.
3. Threshold Logic (M of N): A configurable number of child health checks (e.g., at least 3 out of 5 API microservice health checks) must be healthy.

[ Child Check 1: Web ALB Endpoint ] ----+
[ Child Check 2: Database Latency ] ----+--> [ Calculated Health Check ] --> Route 53 Failover Record
[ Child Check 3: Redis Cache ]      ----+    (Condition: At least 2 of 3 healthy)

CloudWatch Alarm-Based Health Checks

Route 53 can monitor the state of an Amazon CloudWatch Alarm. If the CloudWatch alarm transitions to the ALARM state, the Route 53 health check is marked unhealthy. This allows DNS routing decisions to be driven by internal metrics that external probes cannot see, including:

• EC2 hypervisor CPU utilization or memory exhaustion.
• Custom CloudWatch metrics emitted by application code (e.g., payment gateway error rate > 5%).
• RDS Database connection pool exhaustion (DatabaseConnections metric).

---

DNS Failover Patterns & The DNS TTL Dilemma

Route 53 supports two primary failover topologies:

1. Active-Passive Failover: Directs all traffic to a Primary resource. If the primary health check fails, Route 53 routes queries to the Secondary resource (e.g., a static S3 website holding a maintenance page or a warm standby region).
2. Active-Active Failover: Directs traffic across multiple resources simultaneously using Latency, Weighted, or Geolocation routing. If any individual resource becomes unhealthy, Route 53 automatically removes that record from DNS query responses, shifting traffic dynamically to the surviving active endpoints.

Alias Records & Evaluate Target Health (ETH)

When routing traffic to AWS services (ALBs, NLBs, CloudFront distributions, or S3 website buckets), DevOps engineers should use Route 53 Alias Records rather than standard CNAME records:

• Alias records are resolved internally by Route 53 at the DNS zone apex (example.com), which standard DNS RFC specifications prohibit for CNAME records.
• When Evaluate Target Health (ETH) is enabled on an Alias record pointing to an Application Load Balancer, Route 53 evaluates the health of the backend targets registered to the ALB target groups. If all EC2 instances behind the ALB become unhealthy, the ALB itself is evaluated as unhealthy by Route 53 without needing a separate paid Route 53 health check probe.
• Fail-Open Behavior: If all endpoints in an Alias failover set are evaluated as unhealthy, Route 53 considers all of them healthy and returns all records. This prevents a complete blackout if a shared dependency causes false-positive health check failures across the entire fleet.

The Inherent DNS Caching Problem

DNS failover relies on recursive DNS resolvers and client operating systems respecting the Time to Live (TTL) value configured on the DNS record. In practice:

• Many Internet Service Providers (ISPs), corporate DNS forwarders, and client applications cache DNS records past their configured TTL.
• A record configured with a 60-second TTL may remain cached in client resolvers for 15 to 30 minutes during an actual regional disaster.
• For mission-critical workloads requiring sub-minute Recovery Time Objectives, standard DNS failover alone is insufficient.

---

Route 53 Application Recovery Controller (ARC)

To solve the DNS caching dilemma and automate cross-region disaster recovery safely, AWS introduced Route 53 Application Recovery Controller (ARC). ARC provides highly reliable, redundant control plane capabilities for managing traffic shifts across AWS Regions and Availability Zones.

[ Route 53 ARC Control Plane Cluster (5 Redundant Regional Endpoints) ]
  |
  +-- Routing Control: Primary Region (us-east-1)  [ Status: ON ]
  +-- Routing Control: Secondary Region (us-west-2) [ Status: OFF ]
  |
  +-- Safety Rule: Assertion Rule
  |     └── "Minimum Healthy Regions: At least 1 routing control must be ON"
  |
  +-- Readiness Checks
        +-- Auto Scaling Group Capacity Parity in us-west-2: READY
        +-- Aurora Replica Lag & Quota Headroom: READY
        +-- IAM Role Parity: READY

Core Components of Route 53 ARC

1. Routing Control: A simple on/off switch (boolean flag) that governs whether a Route 53 health check evaluates as healthy or unhealthy. When a routing control is toggled to OFF, the associated Route 53 health check reflects the state through the ARC data plane; Route 53 then evaluates routing and resolvers honor their cached DNS TTLs, so the design should not promise an instantaneous client traffic shift.
2. Extremely Resilient 5-Region Cluster: Routing controls are hosted on a dedicated cluster running across five AWS Regions. Even if three AWS regions suffer complete outages simultaneously, the ARC control plane maintains quorum and continues processing traffic shift commands.
3. Safety Rules (Guardrails Against Cascading Outages):
   • Assertion Rules: Enforce operational constraints on routing controls. For example, an assertion rule can declare: "At least one regional routing control must be set to ON at all times." If an automated script or fatigued engineer accidentally attempts to turn off both us-east-1 and us-west-2 routing controls, ARC blocks the API call, preventing a self-inflicted total outage.
   • Gating Rules: Establish hierarchical control dependencies where a group of target routing controls can only be altered if a designated "gate" routing control is open.
4. Readiness Checks: Continuously audit secondary standby infrastructure against primary infrastructure to detect configuration drift and capacity deficits before failover occurs. Readiness checks verify:
   • EC2 Auto Scaling group maximum and desired instance quotas.
   • Database read replica sizing and replication lag.
   • Service quotas (e.g., Elastic IP limits, VPC endpoint limits).
   • Resource tag and security group parity.