13.4 Root Cause Analysis for Auto Scaling & Container Failures

Root Cause Analysis for EC2 Auto Scaling Launch Failures

When an Amazon EC2 Auto Scaling group (ASG) fails to launch replacement instances or scale out during high traffic, application availability quickly degrades. Rather than guessing, DevOps engineers must query the scaling activities via the AWS CLI or console:

aws autoscaling describe-scaling-activities \
    --auto-scaling-group-name production-asg \
    --max-items 5

Diagnostic Matrix for Auto Scaling Launch Failures

---

Amazon ECS Task & Service Failure Modes

Amazon Elastic Container Service (ECS) manages container lifecycle on EC2 container instances and AWS Fargate. When an ECS task stops unexpectedly, ECS records a Stopped Reason and Container Exit Code.

Decoding Container Exit Codes

When a container marked essential: true exits, ECS immediately stops all other containers in the task:

• Exit Code 137 (SIGKILL - Out-Of-Memory Killer):
  • Root Cause: The container's memory consumption exceeded its allocated limit (memory parameter in task definition), or the sum of container memory exceeded the task-level memory limit. The Linux kernel OOM killer sent a SIGKILL (128 + 9 = 137) to terminate the container.
  • Resolution: Increase the container memory limit or task-level memory allocation in the task definition. Alternatively, tune application memory management (e.g., JVM heap limits -Xmx).
• Exit Code 1 (Application Runtime Error):
  • Root Cause: The application process exited with a general error (e.g., unhandled exception, syntax error, missing mandatory environment variable, database connection failure on startup).
  • Resolution: Query Amazon CloudWatch Logs for the container log stream to inspect stdout/stderr stack traces.
• Exit Code 143 (SIGTERM):
  • Root Cause: Graceful termination (128 + 15 = 143). ECS sent a SIGTERM signal during a service deployment, Auto Scaling scale-in event, or task retirement. If the container does not exit within the stopTimeout (default 30 seconds), ECS sends SIGKILL (Exit Code 137).
• Exit Code 125: Docker daemon error (e.g., invalid docker run command parameters).
• Exit Code 126: The specified container entrypoint or command cannot be invoked (e.g., file permissions error: permission denied).
• Exit Code 127: The specified command was not found in the container's PATH.

CannotPullContainerError

If the task stops before the application starts with CannotPullContainerError:

1. ECR Authentication: Verify the ECS Task Execution Role has permissions to call ecr:GetAuthorizationToken, ecr:BatchCheckLayerAvailability, and ecr:BatchGetImage.
2. Network Routing: For tasks running on Fargate or EC2 in private subnets, outbound traffic to ECR must be routed through a NAT Gateway or interface VPC Endpoints (com.amazonaws.region.ecr.api, com.amazonaws.region.ecr.dkr, and an S3 Gateway Endpoint com.amazonaws.region.s3 to download image layers stored in S3).

Architectural Distinction: Task Role vs. Task Execution Role

---

Amazon EKS Workload & Networking Failures

Amazon Elastic Kubernetes Service (EKS) introduces Kubernetes-native failure abstractions that DevOps engineers must diagnose using kubectl and AWS telemetry.

1. Pod Failure States & Root Causes

• CrashLoopBackOff: The application container starts, fails, crashes, and is restarted repeatedly by the kubelet with exponential backoff. Diagnosis: Run kubectl logs <pod-name> --previous to inspect logs from the crashed container before the restart.
• OOMKilled: The container exceeded its Kubernetes resource limit (resources.limits.memory). Diagnosis: Run kubectl describe pod <pod-name>, which reveals Last State: Terminated, Reason: OOMKilled, Exit Code: 137.
• ImagePullBackOff / ErrImagePull: The kubelet cannot pull the specified container image due to an invalid image tag, missing image pull secret, or unauthorized private registry.
• Pending: The pod cannot be scheduled onto any worker node. Diagnosis: Check kubectl describe pod under Events: FailedScheduling. Causes include insufficient CPU/memory requests (Insufficient cpu, Insufficient memory), node taints without matching tolerations, or conflicting nodeSelector / nodeAffinity rules.

2. AWS VPC CNI Secondary IP Exhaustion

The AWS VPC Container Network Interface (CNI) plugin assigns native VPC private IP addresses directly to pods. Each EC2 worker node can only attach a fixed number of ENIs, and each ENI supports a limited number of secondary private IPv4 addresses based on the EC2 instance type.

• The Failure: Pods remain in Pending or fail with FailedCreatePodSandBox: failed to assign an IP address to container.
• The Resolution: Prefix Delegation: Rather than allocating individual secondary IP addresses per ENI slot, enable Prefix Delegation on the AWS VPC CNI:

kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true

Prefix delegation assigns a /28 IPv4 subnet prefix (16 IP addresses) to each ENI slot instead of a single IP, dramatically increasing pod density per worker node without exhausting ENI limits.

---

Investigating Infrastructure Root Cause with AWS Health & CloudTrail

When multiple unrelated applications simultaneously experience intermittent connectivity, high latency, or sudden instance termination, the root cause often resides in the underlying AWS infrastructure rather than application code.

Correlating Telemetry

1. AWS Health API: Query DescribeEvents and DescribeAffectedEntities to verify if AWS has flagged degraded underlying hardware or scheduled retirement:

aws health describe-affected-entities \
    --filter "eventArns=['arn:aws:health:us-east-1::event/EC2/AWS_EC2_PERSISTENT_INSTANCE_RETIREMENT_SCHEDULED/...']"

1. CloudTrail Lake Forensics: When suspicious configuration drift or accidental resource termination occurs, use CloudTrail Lake to query multi-account, multi-region API logs with SQL:

SELECT
    eventTime,
    eventName,
    userIdentity.arn AS actor,
    requestParameters
FROM
    eds-secops-trail
WHERE
    eventName IN ('TerminateInstances', 'DeleteSecurityGroup', 'UpdateSecurityGroupRuleDescriptionsIngress')
    AND eventTime > '2026-09-11 14:00:00'
ORDER BY
    eventTime DESC;