8.2 Container & Serverless Scalability: ECS Capacity Providers, EKS & API Gateway

Amazon ECS Auto Scaling & Capacity Providers

Scalability in Amazon Elastic Container Service (ECS) operates on two distinct planes: the application service plane (scaling the number of running container tasks) and the infrastructure compute plane (scaling the underlying EC2 instances or utilizing AWS Fargate).

ECS Service Auto Scaling

ECS Service Auto Scaling leverages AWS Application Auto Scaling to adjust task desired counts automatically based on target tracking or step scaling policies:

• ECSServiceAverageCPUUtilization: Average percentage CPU utilized across service tasks.
• ECSServiceAverageMemoryUtilization: Average percentage memory consumed across service tasks.
• ALBRequestCountPerTarget: Request throughput directed to target group containers.

ECS Capacity Providers & Managed Scaling

When hosting containers on EC2 clusters, task scaling is constrained by available cluster memory, CPU, and port allocations. If an ECS service attempts to scale from 10 to 30 tasks, but the underlying EC2 instances only have capacity for 12, tasks remain stuck in PROVISIONING or fail with placement errors unless the underlying EC2 Auto Scaling Group (ASG) expands synchronously.

ECS Capacity Providers bridge this gap through Managed Scaling. The Capacity Provider continuously calculates the CloudWatch metric CapacityProviderReservation ($M$):

$M = \left( \frac{\text{Number of Instances Needed for All Tasks}}{\text{Current Active EC2 Instances}} \right) \times 100$

CapacityProviderReservation Target Value = 100%
- If M > 100: Tasks need more instances than exist ──> ASG Scales OUT
- If M = 100: All instances fully packed with tasks ──> Equilibrium
- If M < 100: Spare instance capacity available   ──> ASG Scales IN

Setting the target capacity to 100 ensures maximum instance packing and zero wasted compute. Setting target capacity to 80 maintains an intentional 20% compute headroom buffer so newly scaled tasks place instantly without waiting for EC2 instances to boot.

Managed Termination Protection

[!IMPORTANT] DOP-C02 Exam Trap: Standard EC2 Auto Scaling group scale-in algorithms select instances for termination based on AZ balance and oldest launch template, completely unaware of whether an instance is hosting critical ECS tasks. To prevent Auto Scaling from killing instances running active containers, you must enable Managed Termination Protection (managedTerminationProtection: ENABLED) on the Capacity Provider and enable instance protection on the ASG (NewInstancesProtectedFromScaleIn: true). ECS then dynamically manages instance protection locks, releasing them only when an instance is cleanly drained of all tasks.

Amazon EKS Autoscaling: HPA, VPA & Karpenter

Amazon Elastic Kubernetes Service (EKS) requires coordinating pod-level scaling with worker node infrastructure provisioning.

Horizontal Pod Autoscaler (HPA) vs. Vertical Pod Autoscaler (VPA)

• Horizontal Pod Autoscaler (HPA): Scales the number of pod replicas up or down by querying the Kubernetes Metrics Server (or custom metrics via Prometheus / KEDA). The formula evaluated every 15 seconds is:

$\text{desiredReplicas} = \left\lceil \text{currentReplicas} \times \left( \frac{\text{currentMetricValue}}{\text{desiredMetricValue}} \right) \right\rceil$

• Vertical Pod Autoscaler (VPA): Dynamically sizes CPU and memory requests/limits for existing pods. Modes include Off (recommendation only), Initial (assigns requests on pod creation), and Recreate (evicts running pods and restarts them with updated resource boundaries).

[!CAUTION] Critical Scheduling Conflict: Never configure HPA and VPA to manage the same resource metric (such as CPU utilization) on the same workload! When CPU utilization spikes, HPA attempts to add more pod replicas while VPA attempts to evict existing pods to increase their memory/CPU requests. This creates an infinite, flapping disruption loop. Use HPA for scaling replicas based on CPU/traffic, and use VPA in Off or recommendation mode, or scale HPA on custom business metrics.

Karpenter vs. Kubernetes Cluster Autoscaler

Karpenter NodePool and Consolidation Mechanics

Karpenter watches the Kubernetes API for pods marked as Pending with Unschedulable status due to resource deficits. Instead of expanding a static ASG, Karpenter evaluates the aggregate requirements of all pending pods (vCPU, memory, GPU, architecture like ARM64 vs. x86_64, topology spread constraints, and availability zone affinities). It then invokes ec2:CreateFleet to launch the most cost-effective instance type that satisfies all constraints.

Karpenter features Node Consolidation. In Karpenter v1.0 and later the NodePool field is disruption.consolidationPolicy: WhenEmptyOrUnderutilized (renamed from WhenUnderutilized in pre-v1 releases); the alternative value is WhenEmpty:

• When pods terminate and cluster load drops, Karpenter calculates if surviving pods across multiple nodes can fit onto fewer nodes or smaller, cheaper instance types.
• It drains the excess nodes cleanly, cordons them, evicts the pods, launches cheaper instances if necessary, and terminates the abandoned nodes, saving up to 40% on infrastructure costs.

---

Amazon API Gateway Scalability, Caching & Throttling

Amazon API Gateway acts as a fully managed, scalable front door for microservices and serverless architectures, handling millions of concurrent requests while protecting backend systems from degradation.

Token Bucket Throttling Algorithm

API Gateway enforces request limits using the standard Token Bucket algorithm:

• Steady-State Rate: The number of tokens added to the bucket per second (the sustained request rate, e.g., 10,000 requests per second).
• Burst Capacity: The maximum volume of tokens the bucket can hold at any single instant (the maximum concurrency surge, e.g., 5,000 tokens).

When a request arrives, API Gateway checks for an available token in the bucket:

• If a token is present, it is consumed, and the request passes to the backend integration.
• If the bucket is empty, API Gateway drops the request immediately and returns an HTTP 429 Too Many Requests error, shielding downstream Lambda functions, ECS tasks, and databases from collapse.

Incoming Request Burst ──> [ Token Bucket (Burst: 5000) ]
                                    │
              ┌─────────────────────┴─────────────────────┐
              ▼ (Token Available)                         ▼ (Bucket Empty)
      Pass to Integration                         Return HTTP 429
  (Lambda / ECS / DynamoDB)                   (Client Exponential Backoff)

Usage Plans & API Keys

For multi-tenant SaaS environments, API Gateway provides Usage Plans bound to API Keys:

• Client-Level Throttling: Configure distinct rate and burst limits for specific customer tiers (e.g., Free Tier: 100 RPS / 200 Burst; Enterprise Tier: 5,000 RPS / 10,000 Burst).
• Quotas: Define aggregate request volume caps over extended intervals (e.g., 1,000,000 requests per month).

API Gateway Caching & Invalidation

API Gateway supports dedicated response caching per stage or method, provisioned with cache memory ranging from 0.5 GB to 237 GB:

• TTL (Time to Live): Default is 300 seconds (configurable from 0 to 3600 seconds).
• Cache Keys: Cached responses are keyed on incoming parameters, including URL path segments, query strings, and request headers.
• Cache Invalidation: Clients can invalidate cached entries by passing the header Cache-Control: max-age=0. To prevent denial-of-service cache wiping, API Gateway allows you to restrict invalidation to authorized IAM callers, returning an HTTP 403 Forbidden if an unauthorized client attempts cache bypass.