5.1 Elastic Load Balancing (ALB/NLB) Configuration & Troubleshooting

5.1 Elastic Load Balancing (ALB/NLB) Configuration & Troubleshooting

CloudOps Blueprint Focus: AWS Certified CloudOps Engineer – Associate (SOA-C03) candidates must configure, manage, and troubleshoot Elastic Load Balancing (ELB) architectures. This requires deep proficiency in differentiating Application Load Balancer (ALB) and Network Load Balancer (NLB) capabilities, tuning target group health checks, managing connection draining lifecycles, understanding cross-zone load balancing cost dynamics, and rapidly resolving HTTP 502, 503, and 504 operational failures.

Load Balancer Architectural Comparison: ALB vs. NLB

Elastic Load Balancing automatically distributes incoming application traffic across multiple targets, such as Amazon EC2 instances, containers (Amazon ECS/EKS), IP addresses, and AWS Lambda functions. Modern AWS infrastructure relies primarily on two specialized load balancers:

1. Application Load Balancer (ALB): Functions at the application layer (OSI Layer 7). ALB inspects application-layer payloads and HTTP/HTTPS headers, enabling advanced content-based request routing. It natively supports modern web protocols including HTTP/2, gRPC, and WebSockets. ALB integrates directly with AWS WAF for web application firewall inspection, authenticates users via Amazon Cognito or OpenID Connect (OIDC) identity providers, and terminates TLS certificates managed by AWS Certificate Manager (ACM) with Server Name Indication (SNI) for multi-domain hosting.
2. Network Load Balancer (NLB): Functions at the transport layer (OSI Layer 4). NLB is engineered for ultra-high performance, capable of processing millions of requests per second with sustained single-digit sub-millisecond latencies. Operating at Layer 4, NLB routes raw TCP, UDP, and TLS connections without inspecting application-layer HTTP payloads. A defining operational capability of NLB is its provision of a single, static IP address per enabled Availability Zone - either AWS-assigned or a customer-allocated Elastic IP (EIP). These are zonal addresses, not anycast addresses; anycast entry points are an AWS Global Accelerator capability. This makes NLB the mandatory choice when clients require deterministic firewall IP whitelisting or when exposing services privately via AWS PrivateLink (VPC Endpoint Services).

Target Types & Cross-Resource Routing

Target groups define where traffic is directed. Both ALB and NLB support:

• instance: Routes traffic to EC2 instances specified by their Instance IDs. The instance primary network interface (eth0) receives traffic.
• ip: Routes traffic to static private IPv4 or IPv6 addresses. This enables routing to ECS tasks in awsvpc networking mode, on-premises physical servers connected via AWS Direct Connect or Site-to-Site VPN, and secondary ENIs.
• alb: Allows an NLB to register an Application Load Balancer as a target. This powerful architectural pattern combines NLB static IP addressing and PrivateLink connectivity with ALB Layer 7 routing and AWS WAF security.
• lambda: Supported exclusively by ALB, transforming HTTP requests directly into JSON payloads that invoke serverless AWS Lambda functions.

---

Target Group Health Checks & Configuration Parameters

ELB periodically sends health check pings to registered targets to determine their availability. Only targets in an healthy state receive client traffic. Health check parameters must be tuned carefully to avoid premature target failover while quickly isolating degraded backends.

Essential Health Check Settings

• HealthCheckProtocol & HealthCheckPort: The network protocol (HTTP, HTTPS, TCP) and destination port. By default, ELB uses traffic-port (the port where the target receives traffic), but a dedicated management port can be specified.
• HealthCheckPath: (ALB and HTTP/HTTPS NLB) The specific URI path requested by the load balancer. Best practice requires a lightweight, dedicated health endpoint (e.g., /healthz or /status) that checks core operational dependencies (such as local process execution) without running expensive, blocking database queries that could exhaust backend connection pools.
• HealthCheckIntervalSeconds: The elapsed time between consecutive health check probes (range: 5–300 seconds; default: 30 seconds for ALB, 10 or 30 seconds for NLB).
• HealthCheckTimeoutSeconds: The maximum duration the load balancer waits for a target response before marking that probe a failure (range: 2–120 seconds; default: 5 seconds). The timeout value must always be strictly less than the interval.
• HealthyThresholdCount: The number of consecutive successful health checks required before transitioning an unhealthy target back to healthy (range: 2–10; default: 5 for ALB/NLB).
• UnhealthyThresholdCount: The number of consecutive failed health checks required to transition a healthy target to unhealthy (range: 2–10; default: 2).
• Matcher (HTTP Codes): Configures expected HTTP status codes. The default is 200. CloudOps engineers can configure ranges (e.g., 200-299) or comma-delimited lists (e.g., 200,301,302).

[!WARNING] The HTTP 301/302 Redirect Health Check Trap: A frequent operational incident occurs when an application team enforces an HTTP-to-HTTPS redirect rule on backend web servers. If the ALB sends health check pings over HTTP on /health and expects a 200 response, the server returns 301 Moved Permanently or 302 Found. Because 301/302 does not match 200, the ALB marks every backend instance unhealthy, resulting in an immediate HTTP 503 outage. To resolve this, configure the matcher to 200,301,302 or perform health checks directly over HTTPS.

# Inspect target health and diagnostic error reasons
aws elbv2 describe-target-health \
    --target-group-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/web-tg/a1b2c3d4e5f6g7h8

---

Connection Lifecycle: Deregistration Delay & Sticky Sessions

Deregistration Delay (Connection Draining)

When a target is deregistered—either manually during maintenance or automatically by an Auto Scaling group during a scale-in event—the load balancer enters the deregistration delay phase (formerly connection draining):

Target Deregistered  --->  New Requests Stopped  --->  In-Flight Requests Complete  --->  Target Deregistered / Terminated
[ Event Initiated ]        [ Immediate Cutoff ]        [ Draining Window: 0 - 3600s ]     [ Instance Safe to Stop ]

1. The load balancer immediately ceases forwarding new connections to the deregistering target.
2. The load balancer keeps existing, in-flight TCP connections open, allowing backend processes to complete outstanding transactions.
3. Once all in-flight requests finish or the configured deregistration delay timeout expires (default: 300 seconds / 5 minutes; range: 0–3,600 seconds), the target transitions from draining to unused, and Auto Scaling safely terminates the instance.

Operational Tuning: For containerized microservices running on Amazon ECS or EKS where REST requests execute in under 500 milliseconds, a 300-second draining delay causes deployment rollouts to stall for 15–20 minutes. CloudOps engineers tune the delay down to 15–30 seconds, drastically accelerating continuous deployments without dropping client requests. Conversely, for workloads with large batch file uploads or long-lived streaming connections, increasing the delay up to 1,800 seconds prevents abrupt client disconnections.

Sticky Sessions (Session Affinity)

Sticky sessions bind a client session to a specific backend target instance:

• ALB Duration-Based Cookies: The ALB generates an encrypted cookie named AWSALB with a configurable duration (1 second to 7 days). Subsequent requests presenting this cookie route to the original target.
• ALB Application-Based Cookies: The application generates its own session cookie. The ALB issues a companion cookie named AWSALBAPP that mirrors the application cookie's lifespan.
• NLB Target Group Stickiness: NLB enforces stickiness at Layer 4 by hashing the client's source IP address and destination port, directing repeat connections from that IP to the same target.

Operational Risks of Session Stickiness: Stickiness introduces traffic hot spots. If a large corporate customer accesses the system through a unified forward proxy or NAT gateway, all corporate users share the same IP/cookie, concentrating massive traffic onto a single backend instance. This overwhelms the target while sibling instances remain idle, and prevents Auto Scaling from evenly balancing the cluster.

---

Cross-Zone Load Balancing Mechanics & Cost Optimization

Cross-zone load balancing determines how load balancer nodes distribute client traffic across target instances located in multiple Availability Zones within an AWS Region:

             [ Internet Clients ]
                      |
         +------------+------------+
         |                         |
    [ ALB Node AZ-1 ]         [ ALB Node AZ-2 ]
         |                         |
   (Cross-Zone: Always ON)   (Cross-Zone: Always ON)
         |                         |
   +-----+-----+             +-----+-----+
   |     |     |             |     |     |
[Inst] [Inst] [Inst]       [Inst] [Inst] [Inst]
 (AZ-1) (AZ-1) (AZ-1)       (AZ-2) (AZ-2) (AZ-2)

• Application Load Balancer: Cross-zone load balancing is always enabled at the load balancer level and cannot be disabled. Requests received by an ALB node in AZ-1 are evenly distributed across all registered, healthy targets in AZ-1 and AZ-2. AWS charges no inter-AZ data transfer fees for cross-zone load balancing on ALBs.
• Network Load Balancer: Cross-zone load balancing is disabled by default at the target group level. When disabled, an NLB node in AZ-1 forwards traffic strictly to targets located in AZ-1. If AZ-1 contains 2 instances and AZ-2 contains 8 instances, each instance in AZ-1 receives 25% of overall regional traffic, while each instance in AZ-2 receives only 6.25%, leading to severe CPU exhaustion in AZ-1. Enabling cross-zone balancing on NLB balances traffic evenly (10% per instance across all 10 instances). However, standard inter-AZ data transfer fees ($0.01/GB in each direction) apply when NLB nodes route packets across AZ boundaries.

# Enable cross-zone load balancing on an NLB target group
aws elbv2 modify-target-group-attributes \
    --target-group-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/nlb-tg/b2c3d4e5f6g7h8i9 \
    --attributes Key=load_balancing.cross_zone.enabled,Value=true

---

Diagnostic Troubleshooting of ELB Error Codes

When applications fail, the load balancer emits status codes that isolate whether the fault resides in the load balancer infrastructure or the upstream backend targets.

The HTTP 502 Keep-Alive Race Condition

A classic SysOps/CloudOps problem occurs when an ALB maintains persistent TCP connections to backend web servers (e.g., NGINX, Apache, Node.js). ALB has a default idle timeout of 60 seconds. If the backend web server has an idle keep-alive timeout set to 60 seconds or lower (e.g., 15 seconds in Node.js or 60 seconds in Apache), the backend target may initiate a TCP FIN/RST packet at the exact millisecond the ALB dispatches an incoming client request on that reused socket. The ALB detects the closed socket and immediately returns HTTP 502 Bad Gateway to the client. Resolution: Always configure the backend server's keep-alive timeout to be higher than the ALB idle timeout (e.g., 65–75 seconds on the target when ALB is set to 60 seconds).