8.1 EC2 Auto Scaling Policies, Lifecycle Hooks & Warm Pools

EC2 Auto Scaling Policy Types & Mechanics

Amazon EC2 Auto Scaling automatically adjusts compute capacity to maintain steady, predictable performance at the lowest possible cost. Selecting the correct scaling policy is critical for AWS Certified DevOps Engineer - Professional (DOP-C02) architectures, as mismatched policies cause either severe application latency or excessive cloud expenditure.

Comparison of Scaling Policy Types

Target Tracking Scaling Policies

Target Tracking operates like a home thermostat. You specify a target metric value, and Auto Scaling automatically adjusts group capacity up or down to keep the metric at or near that target. Target tracking provides predefined metrics:

• ASGAverageCPUUtilization: Average CPU utilization across the group.
• ASGAverageNetworkIn / ASGAverageNetworkOut: Average bytes received/transmitted per instance.
• ALBRequestCountPerTarget: Number of requests completed per target instance in an Application Load Balancer target group.

{
  "TargetValue": 65.0,
  "PredefinedMetricSpecification": {
    "PredefinedMetricType": "ASGAverageCPUUtilization"
  },
  "ScaleInCooldown": 300,
  "ScaleOutCooldown": 60,
  "DisableScaleIn": false
}

[!IMPORTANT] DOP-C02 Exam Trap: Target Tracking creates and manages the underlying CloudWatch alarms automatically. If your application handles spiky traffic, scale-in can prematurely terminate instances between bursts. Setting DisableScaleIn: true enables you to use Target Tracking exclusively for rapid scale-out while managing scale-in via a separate, conservative Step Scaling policy.

Step Scaling vs. Simple Scaling & Cooldowns

Simple Scaling relies on a single metric threshold and enforces a global cooldown period (DefaultCooldown, typically 300 seconds). During this cooldown, all scaling activities initiated by simple scaling policies are suspended. If a traffic spike requires 20 additional instances, Simple Scaling might add 2, wait 300 seconds, add 2 more, and take 30 minutes to absorb the load, leading to degraded performance.

Step Scaling eliminates the cooldown bottleneck by introducing metric intervals and adjustments:

CloudWatch Alarm Breach: CPU Utilization > 60%
  ├── Step 1: 60% to 70%  ──> Add 10% capacity
  ├── Step 2: 70% to 85%  ──> Add 30% capacity
  └── Step 3: > 85%       ──> Add 60% capacity

Instead of a group-wide cooldown, Step Scaling utilizes EstimatedInstanceWarmup. Newly launched instances do not contribute to CloudWatch group metrics until their warmup period expires, preventing metric distortion while still allowing subsequent alarm breaches to trigger additional scaling actions immediately.

Predictive Scaling

Predictive Scaling uses machine learning to analyze up to 14 days of historical Amazon CloudWatch metric data (a minimum of 24 hours is required before any forecast is produced) and generates an hourly forecast for the next 48 hours, refreshed every 6 hours. It operates in two modes:

1. ForecastOnly: Generates capacity forecasts without adjusting capacity, allowing validation of the ML model against actual usage.
2. ForecastAndScale: Scales the group out at the start of each forecast hour. Use the SchedulingBufferTime property (the Pre-launch instances console setting) to launch capacity earlier, so slow-booting instances are in service before forecast demand arrives. Predictive scaling only scales out; scale-in still requires a dynamic scaling policy.

---

Custom Metric Scaling: SQS Backlog Per Instance

A classic architectural anti-pattern on the DOP-C02 exam is scaling an Auto Scaling worker tier directly on the Amazon SQS ApproximateNumberOfMessagesVisible metric. An absolute queue depth of 50,000 messages indicates severe congestion if you have 2 worker instances, but represents a negligible backlog if you have 1,000 worker instances.

The BacklogPerInstance Formula

To scale proportionally, you must calculate the backlog per healthy running instance:

$\text{BacklogPerInstance} = \frac{\text{ApproximateNumberOfMessagesVisible}}{\text{GroupInServiceInstances}}$

Next, calculate the target value based on your acceptable message latency:

$\text{Target Backlog} = \frac{\text{Acceptable Processing Latency (seconds)}}{\text{Average Processing Time per Message (seconds)}}$

Example: If business requirements dictate that messages must be processed within 60 seconds of arrival, and each worker instance processes 1 message every 3 seconds, each instance can maintain a backlog of $60 / 3 = 20$ messages. The Target Tracking policy target is set to 20.

Implementing Metric Math in Target Tracking

You can feed a CloudWatch Metric Math expression directly into an Auto Scaling Target Tracking policy using a CustomizedMetricSpecification:

{
  "TargetValue": 20.0,
  "CustomizedMetricSpecification": {
    "Metrics": [
      {
        "Id": "m1",
        "MetricStat": {
          "Metric": {
            "Namespace": "AWS/SQS",
            "MetricName": "ApproximateNumberOfMessagesVisible",
            "Dimensions": [{"Name": "QueueName", "Value": "OrderProcessingQueue"}]
          },
          "Stat": "Sum"
        },
        "ReturnData": false
      },
      {
        "Id": "m2",
        "MetricStat": {
          "Metric": {
            "Namespace": "AWS/AutoScaling",
            "MetricName": "GroupInServiceInstances",
            "Dimensions": [{"Name": "AutoScalingGroupName", "Value": "WorkerASG"}]
          },
          "Stat": "Average"
        },
        "ReturnData": false
      },
      {
        "Id": "e1",
        "Expression": "m1 / m2",
        "Label": "BacklogPerInstance",
        "ReturnData": true
      }
    ]
  }
}

---

Auto Scaling Lifecycle Hooks

Auto Scaling Lifecycle Hooks pause instances during transition states, allowing custom orchestration scripts or external systems to execute initialization or cleanup routines.

Scale-Out: [Pending] ──> [Pending:Wait] ──> (Lifecycle Hook) ──> [Pending:Proceed] ──> [InService]
                               │                                        ▲
                               └──> CompleteLifecycleAction(CONTINUE) ─┘

Scale-In:  [InService] ──> [Terminating:Wait] ──> (Lifecycle Hook) ──> [Terminating:Proceed] ──> [Terminated]
                                  │                                           ▲
                                  └──> CompleteLifecycleAction(CONTINUE) ────┘

Hook Configuration and Transitions

• autoscaling:EC2_INSTANCE_LAUNCHING: The instance boots, attaches EBS volumes, runs user data, and pauses in Pending:Wait. You can install software licenses, execute configuration playbooks via AWS Systems Manager (SSM), or preload application caches.
• autoscaling:EC2_INSTANCE_TERMINATING: The instance is detached from target groups, stops accepting incoming traffic, and pauses in Terminating:Wait. You can flush telemetry buffers, push remaining local transactional logs to Amazon S3, and gracefully complete in-flight transactions.

Heartbeats and Completion Semantics

Lifecycle hooks have a default HeartbeatTimeout (typically 3600 seconds, configurable up to 7200 seconds). If long-running tasks exceed this window, the automation agent must issue a heartbeat extension:

aws autoscaling record-lifecycle-action-heartbeat \
    --lifecycle-hook-name InstanceDrainingHook \
    --auto-scaling-group-name ProductionAppASG \
    --instance-id i-0123456789abcdef0

Once tasks finish, the automation signals completion:

aws autoscaling complete-lifecycle-action \
    --lifecycle-hook-name InstanceDrainingHook \
    --auto-scaling-group-name ProductionAppASG \
    --lifecycle-action-result CONTINUE \
    --instance-id i-0123456789abcdef0

---

Auto Scaling Warm Pools

Applications with large initialization overhead (such as legacy enterprise applications, complex machine learning inference engines, or Windows workloads with massive container layers) often require 15 to 30 minutes to become operational. Standard auto scaling cannot absorb unexpected traffic spikes during this window.

A Warm Pool maintains a cache of pre-initialized EC2 instances attached to the Auto Scaling group:

• Pool States: Instances in the warm pool can be kept in a Stopped state (saving all EC2 hourly compute costs while paying only for attached EBS storage) or a Running state (for applications requiring active memory caches or background sync).
• Scale-Out Acceleration: When the ASG scales out, instances transition from Warmed:Stopped -> Warmed:Pending -> InService in seconds, bypassing the entire OS boot, software package installation, and initialization phase.
• Instance Reuse (reuse-on-scale-in): When scaling in, instances can be returned to the warm pool in a Stopped state rather than terminated, preserving pre-baked state for the next peak.