11.1 Application & Infrastructure Health Checks with Route 53 & ALB

Application Load Balancer Target Group Health Checks

In modern cloud architectures, the Application Load Balancer (ALB) acts as the primary ingress controller, distributing HTTP, HTTPS, and gRPC traffic across dynamic target groups composed of Amazon EC2 instances, ECS container tasks, Lambda functions, or private IP addresses. To ensure high availability and prevent routing traffic to degraded compute targets, the ALB continuously executes automated health checks against registered targets.

Core Health Check Parameters

ALB target group health checks are configured at the target group level and apply uniformly to all registered targets within that group. DevOps engineers must master the behavior, constraints, and operational defaults of each parameter:

[!IMPORTANT] Configuration Constraint Trap: In CloudFormation and the AWS CLI, attempting to set HealthCheckTimeoutSeconds equal to or greater than HealthCheckIntervalSeconds triggers an immediate InvalidConfigurationRequest error. The timeout must always be strictly smaller than the interval (e.g., Interval: 10s, Timeout: 5s).

Health State Transitions and Timing Formulas

Targets registered to an ALB transition through five distinct lifecycle states:

1. initial: Target is undergoing registration and the initial batch of health checks is in progress.
2. healthy: Target has met HealthyThresholdCount consecutive successful checks.
3. unhealthy: Target failed UnhealthyThresholdCount consecutive checks or timed out.
4. draining: Target has been deregistered or marked unhealthy, and in-flight requests are finishing.
5. unused: Target is not registered to an active target group or the target group is not bound to a listener rule.

The time required for an ALB to detect an outage and stop routing requests is determined by the formula:

$\text{Detection Latency} = \text{HealthCheckIntervalSeconds} \times \text{UnhealthyThresholdCount}$

With default values ($30\text{s} \times 2$), an application outage takes up to 60 seconds to be detected per target, during which users receive HTTP 502/504 errors. In mission-critical environments, DevOps engineers optimize this by tuning the interval to 5 seconds and the unhealthy threshold to 2, reducing failure detection latency to 10 seconds.

Graceful Draining and Deregistration Delay

When a target is deregistered—whether through an automated blue/green deployment, an Auto Scaling scale-in event, or manual intervention—the ALB does not sever active TCP connections abruptly. Instead, it places the target into the draining state for the duration of the Deregistration Delay (also known as connection draining):

• Default Value: 300 seconds (5 minutes). Configurable from 0 to 3600 seconds (1 hour).
• Routing Behavior: The ALB immediately ceases sending new HTTP requests or TCP connections to the draining target. However, existing in-flight HTTP requests and active TCP sessions are permitted to continue until they complete cleanly.
• Premature Completion: If all active in-flight requests finish before the deregistration delay expires, the ALB immediately closes remaining idle keep-alive connections and transitions the target to unused, without waiting for the full timer.
• Tuning for CI/CD Pipelines: In containerized microservices (Amazon ECS / EKS) where API transactions execute in under 50 milliseconds, leaving deregistration delay at 300 seconds unnecessarily stalls rolling deployments for 5 minutes per batch. Lowering deregistration delay to 15 - 30 seconds dramatically accelerates deployment velocity while preventing dropped connections.

---

Shallow vs. Deep Health Checks: Preventing Cascading Failures

A critical design decision on the DOP-C02 exam is selecting the appropriate depth of the health check probe.

The Cascading Failure Anti-Pattern

Consider an architecture where 20 EC2 instances serve an e-commerce catalog behind an ALB. The target group is configured with a deep health check (/health/full) that executes a query against an Amazon Aurora MySQL database on every check. The health check interval is 10 seconds, and there are 3 ALB nodes in 3 Availability Zones:

ALB Node 1 (AZ-a) ──┐
ALB Node 2 (AZ-b) ──┼──> Probes 20 Instances every 10s = 60 DB queries / 10s = 6 QPS baseline
ALB Node 3 (AZ-c) ──┘

If the Aurora database experiences a sudden query lock or CPU spike, query latency jumps from 5 ms to 6,000 ms:

1. The health check queries time out (exceeding the 5-second HealthCheckTimeoutSeconds).
2. The ALB marks 5 instances as unhealthy.
3. The remaining 15 healthy instances must now absorb 100% of incoming user traffic.
4. Increased application load on the 15 instances drives even more connection pool contention and load to Aurora.
5. Within 30 seconds, the remaining 15 instances fail their deep health checks.
6. The ALB marks all 20 targets unhealthy and returns 503 Service Unavailable to 100% of end users.

[!CAUTION] Architectural Standard: Never configure an ALB target group health check to query downstream transactional databases or external synchronous dependencies directly. Implement a shallow liveness check for the ALB. To monitor downstream dependency health, use an asynchronous background thread that executes health pings periodically and sets a local circuit-breaker flag with exponential backoff and cached status.

---

Route 53 Health Checks & Multi-Region DNS Observability

Amazon Route 53 provides globally distributed DNS health checking to drive automated failover across multi-region and hybrid architectures. Route 53 operates outside the VPC from over 15 globally dispersed health checker locations.

Three Core Types of Route 53 Health Checks

1. Endpoint Health Checks: Probes an IP address or fully qualified domain name (FQDN) via HTTP, HTTPS, or TCP on a specified port (default 80 or 443).
   • Request Interval: Standard (30 seconds, default) or Fast (10 seconds, incurs higher cost).
   • Failure Threshold: Number of consecutive failed checks (range 1 - 10, default 3) required before Route 53 marks the endpoint unhealthy.
   • String Matching: Inspects the first 5,120 bytes of the HTTP response body for a specific text string (e.g., "system_status": "OPERATIONAL").
2. CloudWatch Metric Health Checks: Monitors the status of an existing Amazon CloudWatch alarm. When the alarm enters ALARM state, the Route 53 health check transitions to unhealthy. Essential for alarming on application metrics (e.g., ALB 5xx error rate, SQS queue depth, or custom business KPIs) that cannot be probed via simple HTTP requests.
3. Calculated Health Checks: Aggregates up to 256 individual health checks into a single composite health decision using boolean logic (AND, OR, NOT) or threshold rules (e.g., health check passes if at least 3 of 5 regional endpoints are healthy). Useful for complex microservice dependency webs.

Route 53 Alias Records with EvaluateTargetHealth=true

When routing traffic to AWS resources—such as an Application Load Balancer, Network Load Balancer, or CloudFront distribution—using Route 53 Alias records, you should configure EvaluateTargetHealth=true:

DNS Query: api.example.com
      │
      ▼
Route 53 Alias Record (EvaluateTargetHealth = true)
      │
      ├─> Reads ALB Target Group Health directly from CloudWatch/ELB internal plane
      │   (No external probe charges, zero latency overhead)
      ▼
ALB Target Group Health Status
      ├── All Targets Unhealthy?  ──> Route 53 fails over to Secondary Region / S3 Error Page
      └── At least 1 Healthy?     ──> Route 53 returns ALB DNS name / IP addresses

• Zero Cost: Unlike standard Route 53 endpoint health checks, using EvaluateTargetHealth=true on Alias records pointing to an ALB incurs no additional health check fee.
• No Security Group Ingress Needed: Standard endpoint health checks require opening your security group to hundreds of public Route 53 health checker IP addresses. In contrast, EvaluateTargetHealth queries the internal AWS control plane metrics of the load balancer directly, allowing your ALB security group to restrict ingress strictly to authorized client CIDRs or CloudFront managed prefix lists.
• Target Group Sensitivity: If an ALB has multiple target groups bound to different listener rules, Route 53 evaluates the overall health: if any target group with registered targets has at least one healthy target, the ALB is considered healthy. If all targets across all target groups are unhealthy, Route 53 triggers DNS failover.

---

EC2 Status Checks & Automated Recovery

Amazon EC2 continuously monitors instances through two independent status check dimensions. Understanding the precise boundary between these checks is a frequent DOP-C02 question:

EC2 Auto Recovery Architecture

When a CloudWatch alarm triggers on StatusCheckFailed_System:

• The alarm executes the recovery action (ec2:recover).
• AWS migrates the instance to a new, healthy physical server host within the same Availability Zone.
• Preserved Attributes: The instance retains its original Instance ID, Private IPv4 address, Elastic IP address (public IPv4), IPv6 addresses, and all attached EBS volumes and metadata.
• Data In-Flight: Data in instance memory (RAM) is lost, exactly like a power cycle. Instance store (ephemeral) volume contents are lost. The instance must use an EBS-backed root volume to support Auto Recovery.

---

Auto Scaling Health Checks & Grace Periods

By default, an Amazon EC2 Auto Scaling group (ASG) evaluates only EC2 status checks. If an EC2 instance experiences an operating system freeze or hardware degradation, the ASG marks the instance unhealthy and initiates replacement.

The ELB Health Check Type Requirement

Default ASG Behavior (HealthCheckType = EC2):
EC2 Hypervisor OK + Guest OS Network OK ──> ASG considers instance HEALTHY!
(Even if Web Server is crashed, returning 500 Internal Server Error, or ALB marks target UNHEALTHY)

Production ASG Behavior (HealthCheckType = ELB):
EC2 Status Checks OK  AND  ALB Target Group reports HEALTHY ──> ASG considers instance HEALTHY
If ALB marks target UNHEALTHY ──> ASG terminates and replaces instance automatically

[!IMPORTANT] DOP-C02 Core Mandate: To enable Auto Scaling to automatically replace instances that fail application-layer health checks (e.g., returning 5xx status codes to the ALB), you must explicitly configure HealthCheckType: "ELB" on the Auto Scaling group. If set to EC2, the ASG will keep instances running indefinitely even when the ALB has removed them from service due to failing HTTP health probes.

Health Check Grace Period (HealthCheckGracePeriod)

When Auto Scaling launches a new EC2 instance, the application runtime (e.g., JVM startup, container image download, database connection pool establishment, local cache pre-warming) requires time before it can successfully respond to HTTP health probes with 200 OK.

• The Health Check Grace Period defines the duration (default 300 seconds, range 0 - 7200 seconds) that the ASG waits after an instance enters the InService state before checking its ELB health status.
• The Premature Termination Death Loop: If the grace period is set too low (e.g., 30 seconds) for an enterprise Java application that takes 90 seconds to boot, the ALB probe will fail during startup. The ASG immediately flags the instance as unhealthy, terminates it, launches a replacement, and repeats the cycle perpetually, resulting in zero available capacity.
• During the grace period, if the instance fails EC2 System Status Checks, the ASG replaces it immediately; the grace period applies strictly to application health checks (ELB).

---

DOP-C02 Exam Watchouts & Troubleshooting