1.3 Metric Alarms, Anomaly Detection & Composite Alarms

CloudWatch Metric Alarm Anatomy & Evaluation Mathematics

Amazon CloudWatch Metric Alarms monitor metric streams or mathematical expressions, transitioning across three states: OK, ALARM, and INSUFFICIENT_DATA. Key alarm configuration elements include:

• Namespace: Originating service identifier (e.g., AWS/EC2 or CustomApp/Payment).
• MetricName: Property monitored (e.g., CPUUtilization, mem_used_percent, or 5XXError).
• Dimensions: Key-value pairs identifying the target resource (e.g., InstanceId=i-0123456789abcdef0).
• Statistic: Mathematical aggregation across the interval (Average, Sum, Minimum, Maximum, SampleCount, or percentiles like p90, p95, p99).
• Period: Evaluation window duration in seconds (10s, 30s, or multiples of 60s up to 86,400s / 24 hours).
• Threshold & Comparison Operator: Target limit and evaluation logic (e.g., GreaterThanOrEqualToThreshold).

Evaluation Periods and "M out of N" Logic

Alarms evaluate performance using Evaluation Periods ($N$) and Datapoints to Alarm ($M$):

• Standard evaluation ($M = N$, e.g., 3 out of 3): Every consecutive period must breach the threshold before triggering.
• M out of N evaluation ($M < N$, e.g., 3 out of 5): The alarm triggers if at least $M$ datapoints breach the threshold across any of the last $N$ evaluation periods.

Evaluation Window (N = 5 periods)
Period:     [ t-4 ]   [ t-3 ]   [ t-2 ]   [ t-1 ]   [  t  ]
Status:     BREACH     OK      BREACH    BREACH     OK
Evaluation: 3 out of 5 periods breaching (M=3, N=5) ---> State: ALARM

Using $M < N$ prevents alarm flapping from transient, short-lived spikes while ensuring prompt alerting during intermittent degradation.

Missing Data Treatment

When network partitions or crashes interrupt metric streams, missing data treatment governs state evaluation:

---

CloudWatch Anomaly Detection Alarms

Static thresholds struggle with workloads subject to cyclic demand. For instance, an e-commerce platform processing 5,000 requests per second at mid-day but only 200 at night produces false alarms with static limits.

CloudWatch Anomaly Detection applies machine learning to analyze up to two weeks of metric history. The model discovers hourly, daily, and weekly seasonality, filtering operational spikes during training.

The model generates an expected metric band with upper and lower bounds. Operators set the anomaly detection threshold as the number of standard deviations ($k$, typically 1 to 5) the metric may deviate from the band. Alarms trigger when metrics rise above the upper band, fall below the lower band, or exit both boundaries.

---

Composite Alarms & Alert Storm Suppression

During core infrastructure failures—such as a database outage—hundreds of secondary alarms trigger across dependent application servers and load balancers, causing an alert storm.

Composite Alarms combine multiple metric alarms into a single high-level state using boolean rule expressions (ALARM, OK, INSUFFICIENT_DATA):

ALARM("HighCPU-WebFleet") AND ALARM("High5xx-ALB") AND NOT ALARM("DatabaseFailoverInProgress")

Alert Suppression Rules

Composite alarms allow operations teams to establish suppression rules (inhibit logic). When a parent infrastructure failure occurs, dependent child alarms are evaluated within the composite rule, preventing downstream paging alerts. Engineers receive a single critical notification targeting the root cause.

Action Limitations: Composite alarms trigger Amazon SNS notifications, AWS Systems Manager OpsCenter OpsItems, and Amazon EventBridge events. However, composite alarms cannot directly execute EC2 actions (Stop, Reboot, Recover) or Auto Scaling policies; those must remain attached to individual metric alarms.

---

Automated Alarm Actions & EC2 Auto-Recovery

CloudWatch alarms execute automated operational actions:

1. Auto Scaling Actions: Trigger step or target tracking policies to scale EC2 Auto Scaling groups.
2. Notification Actions: Publish formatted payloads to Amazon SNS topics, fanning out to email, SMS, or PagerDuty.
3. EventBridge Event Emission: State transitions emit CloudWatch Alarm State Change events to EventBridge, triggering Lambda functions or Systems Manager runbooks.
4. Amazon EC2 Instance Actions: Execute Stop, Terminate, Reboot, or Recover.

EC2 Auto-Recovery vs. Reboot

Differentiating EC2 status checks is critical:

+-------------------------------------------------------------------------+
| StatusCheckFailed_System (Physical Infrastructure / Hypervisor Failure) |
|   Action: EC2 Recover                                                   |
|   Result: Migrates instance to new physical host in same AZ             |
|   Preserved: Instance ID, Private IP, Elastic IP, Metadata, EBS Volumes |
+-------------------------------------------------------------------------+

+-------------------------------------------------------------------------+
| StatusCheckFailed_Instance (Guest OS / Kernel / Filesystem Failure)     |
|   Action: EC2 Reboot                                                    |
|   Result: Soft reset of the guest operating system                      |
|   Use Case: Unresponsive kernel, corrupted systemd service, OOM freeze  |
+-------------------------------------------------------------------------+

An alarm on StatusCheckFailed_System triggers EC2 Recover, migrating the instance to a new healthy host in the same Availability Zone while preserving its Instance ID, private IP, Elastic IP, and EBS volumes. Conversely, StatusCheckFailed_Instance indicates guest OS kernel panics or filesystem corruption, resolved via EC2 Reboot or manual OS troubleshooting.

---

Alarm Troubleshooting & Failure Modes

• Lingering INSUFFICIENT_DATA: Alarms remain in INSUFFICIENT_DATA when configured dimensions do not match published dimensions exactly. If a metric publishes InstanceId and Environment, an alarm with only InstanceId never receives data.
• Period Mismatches: Evaluating a 1-minute alarm on a 5-minute metric stream introduces gaps, leading to unstable evaluations.
• Percentile Traps: Percentiles (p90, p99) require adequate sample sizes. Calculating p99 on endpoints with under 100 requests per period creates erratic spikes or dead-zone evaluations.