10.2 CloudWatch Dashboards, Composite Alarms & Anomaly Detection

CloudWatch Dashboards: Cross-Account & Cross-Region Architecture

Modern enterprise cloud architectures span hundreds of AWS accounts and multiple geographic Regions. Operating isolated dashboards within each individual account creates visibility silos and slows incident response. Amazon CloudWatch provides centralized, unified operational dashboards capable of rendering telemetry across accounts and Regions on a single pane of glass.

Cross-Account Observability via CloudWatch Observability Access Manager (OAM)

AWS CloudWatch Observability Access Manager (OAM) establishes secure, native cross-account telemetry sharing without requiring complex custom IAM role assumption scripts or third-party forwarding agents.

[ Source Account A (111111111111) ] ──┐
                                      │ (Telemetry Link)
[ Source Account B (222222222222) ] ──┼──> [ Central Monitoring Account (999999999999) ]
                                      │        - Unified Dashboards
[ Source Account C (333333333333) ] ──┘        - Cross-Account Alarms / Metrics

1. Monitoring Account Configuration: In every Region that must be observed, a designated centralized operations account creates one OAM Sink using the regional CreateSink API. A sink resource policy (PutSinkPolicy) defines which source accounts are authorized to attach to the sink. The policy can authorize individual account IDs or an entire organization via aws:PrincipalOrgID:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": ["oam:CreateLink", "oam:UpdateLink"],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:PrincipalOrgID": "o-enterpriseorg123"
        }
      }
    }
  ]
}

1. Source Account Configuration: Each workload account establishes an OAM Link using CreateLink, referencing a sink in the same Region. The link specifies which telemetry types to share, such as AWS::CloudWatch::Metric, AWS::Logs::LogGroup, and AWS::XRay::Trace. Repeat the link in every Region whose telemetry must be shared.
2. Cross-Region Dashboard Composition: OAM itself is a within-Region cross-account feature. After configuring regional sink/link pairs, dashboard widgets can specify their target region; a single dashboard can then display an EC2 metric from us-east-1, an Aurora metric from eu-west-1, and a DynamoDB metric from ap-southeast-1 side-by-side. Alarms and composite alarms remain Region-scoped.

Dashboard Sharing Mechanisms

To share dashboards with stakeholders outside the primary AWS management console, CloudWatch provides three sharing architectures:

• AWS IAM Identity Center (Successor to AWS SSO): Provides enterprise users authenticated access to specific dashboards using corporate Active Directory or Okta credentials without provisioning IAM users in the AWS account.
• Amazon Cognito User Pools: Authenticates external clients or contractors via username/password or third-party SAML/OpenID Connect identity providers.
• Public Dashboard Sharing with Passphrase: Generates a publicly accessible HTTPS URL protected by a shared secret passphrase. Sensitive metric dimensions should be audited prior to enabling public sharing.

---

Dashboard Widget Types & Visualizations

CloudWatch Dashboards are defined as declarative JSON objects containing an array of widget definitions positioned on a 24-column responsive grid layout.

{
  "widgets": [
    {
      "type": "metric",
      "x": 0, "y": 0, "width": 12, "height": 6,
      "properties": {
        "metrics": [
          [ "AWS/ApplicationELB", "TargetResponseTime", "LoadBalancer", "app/prod-alb/123", { "stat": "p99" } ]
        ],
        "period": 60,
        "region": "us-east-1",
        "title": "Production ALB p99 Latency"
      }
    }
  ]
}

Supported Widget Types

• Line & Stacked Area Widgets: Display continuous time-series metrics. Stacked area widgets illustrate resource composition (e.g., memory utilization across multiple ECS task containers).
• Single Value (Number) Widgets: Render the most recent aggregate scalar value with color-coded threshold formatting (e.g., displaying current active EC2 node count or daily error count).
• Bar Charts: Display comparative categorical data across instances, Auto Scaling groups, or microservices.
• Logs Insights Query Widgets: Embed live CloudWatch Logs Insights queries directly into the dashboard. The widget automatically executes the query and renders the resulting tabular or time-binned bar chart, refreshing on dashboard reload.
• Alarm Status Widgets: Display the real-time operational status (OK, ALARM, INSUFFICIENT_DATA) of selected alarms in a compact grid, allowing operations engineers to instantly assess fleet health.
• Text / Markdown Widgets: Embed static runbooks, operational standard operating procedures (SOPs), architecture diagrams, and emergency on-call contact links alongside live telemetry.

---

Static Threshold Alarms vs. Machine Learning Anomaly Detection

Limitations of Static Thresholds

In high-volume consumer-facing applications, traffic fluctuates dramatically following cyclic patterns (e.g., diurnal weekday peaks and late-night lulls). Configuring a static threshold alarm on a metric such as RequestCount or CPUUtilization introduces severe operational drawbacks:

• A static threshold set for peak hours will fail to detect critical outages occurring during low-traffic off-peak hours (false negatives).
• A static threshold set to detect off-peak anomalies will trigger false alarms during standard afternoon traffic spikes (false positives).

CloudWatch Anomaly Detection Architecture

CloudWatch Anomaly Detection applies supervised machine learning algorithms to historical metric data. The model continuously analyzes up to two weeks of metric history to identify:

• Hourly, daily, and weekly seasonality.
• Gradual macroeconomic and operational trends.
• Random noise versus systematic shifts.

Expected Value Band = Model Prediction ± (Band Width × Standard Deviation)

Metric Value
     ^            Upper Threshold Band
     │         . - ~ ~ ~ - .
     │       /               \      [Breach: ALARM]
     │     /   ~~~~~~~~~~~~~   \        * * *
     │    /  /   Actual    \  \      /       \
     │   │  │    Metric     │  │    │         │
     │    \  \             /  /      \       /
     │     \   ~~~~~~~~~~~~~ /        - ~ ~ -
     │       \             /    Lower Threshold Band
     │         ` - ~ ~ ~ - '
     └──────────────────────────────────────────────> Time

Configuring Anomaly Detection Alarms

1. Band Width (Standard Deviation Multiplier): Controls the width of the prediction band. A lower number (e.g., 1) creates a narrow band, increasing alarm sensitivity. A higher number (e.g., 3) creates a wide band, alerting only on extreme, unprecedented deviations.
2. Evaluation Functions:
   • GreaterThanUpperThreshold: Triggers when the metric exceeds the expected ceiling (e.g., detecting unexpected latency or error spikes).
   • LessThanLowerThreshold: Triggers when the metric drops below the expected floor (e.g., detecting an upstream network partition dropping transaction volume).
   • OutsideBand: Triggers when the metric breaches either the upper or lower boundary.
3. Model Customization & Excluded Periods: During abnormal operational events (such as load tests, catastrophic regional outages, or Black Friday sales), DevOps engineers can configure Excluded Time Periods. This instructs the algorithm to ignore data points within those windows so the anomaly detection model is not skewed.

---

Composite Alarms: Architecture & Boolean Logic

The Alert Fatigue & Alarm Storm Problem

When a fundamental infrastructure dependency fails—such as an Amazon RDS database storage volume filling up—it triggers a cascade of individual metric alarms: database storage low, database CPU high, backend container connection pool timeout, Application Load Balancer HTTP 504 count high, and frontend API latency critical. On-call engineers receive dozens of simultaneous pager notifications for a single underlying incident, causing alert fatigue.

Composite Alarm Rule Mechanics

A Composite Alarm evaluates the states of other alarms using Boolean logic (AND, OR, NOT). A composite alarm does not monitor raw metrics directly; it monitors underlying child alarms (which can be standard metric alarms or anomaly detection alarms).

Composite Alarm Rule Syntax:
ALARM("alarm_name") | OK("alarm_name") | INSUFFICIENT_DATA("alarm_name")
Logical Operators: AND, OR, NOT

Production Example: Microservice Outage with Maintenance Suppression

ALARM("alb-target-5xx-high") 
  AND ALARM("ecs-container-cpu-critical") 
  AND NOT ALARM("scheduled-maintenance-active")

This alarm transitions to ALARM only if both the ALB 5xx errors and container CPU are breached simultaneously, but suppresses all notification actions if the operations team has put the environment into scheduled-maintenance-active mode.

Alarm Suppression Rules

CloudWatch Composite Alarms support built-in Suppression Rules. An alarm can designate a Suppressor Alarm:

• When the suppressor alarm is in the ALARM state, the composite alarm is prevented from executing any notification or remediation actions, even if its own Boolean rule evaluates to ALARM.
• WaitPeriod: The duration (in seconds) that the composite alarm waits before evaluating its alarm state after the suppressor alarm transitions. This accommodates propagation delays during deployment rollouts.
• ExtensionPeriod: The duration (in seconds) that the suppression state remains active after the suppressor alarm transitions back to OK. This prevents alarm thrashing while systems stabilize following maintenance or recovery.

---

Automated Operational Remediation Actions

CloudWatch Alarms (both metric and composite) support multi-target automated remediation workflows when transitioning between states (ALARM, OK, INSUFFICIENT_DATA):

[!IMPORTANT] Action Constraint: Composite alarms can execute Amazon SNS notifications, create Systems Manager OpsItems, and, in accounts that already use the service, create SSM Incident Manager incidents. However, composite alarms cannot directly trigger EC2 Auto Scaling policies or EC2 instance recovery/reboot actions; those actions must be triggered by underlying single-metric alarms.

---

DOP-C02 Exam Watchouts & Troubleshooting