CloudWatch Centralized Telemetry

Why centralized telemetry matters on SAP-C02

SAP-C02 Domain 3 scores continuous improvement of existing solutions. Task 3.1 asks you to determine a logging and monitoring strategy, pair alerting with automatic remediation, and improve operational excellence on stacks that already run in production. Task 3.3 adds monitoring toolsets, service level agreements (SLAs), and key performance indicators (KPIs). Independent SAP-C02 study material by OpenExamPrep treats Amazon CloudWatch as the Region-scoped control plane for metrics, logs, and alarms, then connects distributed traces in the next section.

When a production API misses its p99 latency objective, the architect’s first job is to decide whether the signal is missing, aggregated too coarsely, trapped in a member account the platform team cannot query, or never exercised unless a human is online. The exam prefers a centralized telemetry design: consistent namespaces and dimensions, agent-collected in-guest metrics, queryable logs, alarms that represent customer impact, and a monitoring account that can search across the organization without building a one-off extractor for every workload.

Metrics, namespaces, dimensions, and retention

A CloudWatch metric is a time-ordered set of data points uniquely identified by a namespace, a metric name, and zero or more dimensions (name/value pairs). AWS services typically publish into AWS/service namespaces such as AWS/EC2. You must specify a namespace for every custom data point you publish. Classic CloudWatch metrics support up to 30 dimensions. OpenTelemetry metrics ingested into CloudWatch use labels (up to 150) instead of dimensions and are a separate query path (PromQL in Query Studio versus GetMetricData for classic metrics).

Metrics exist only in the Region where they are created. They cannot be deleted; unused metrics expire after 15 months. CloudWatch retains metric data with automatic rollup:

Standard versus high-resolution metrics and alarm periods

Standard resolution stores data at one-minute granularity. High-resolution custom metrics store data at one-second granularity; you can graph or alarm them at 1, 5, 10, 30 seconds, or any multiple of 60 seconds. High-resolution alarms with a 10- or 30-second period cost more than minute-period alarms. AWS service metrics are standard resolution by default. Amazon EC2 basic monitoring publishes about every five minutes; detailed monitoring publishes one-minute metrics. An alarm period must be at least as coarse as the metric resolution: do not set a 60-second alarm on a five-minute basic-monitoring series and expect a clean signal.

A classic exam trap is treating Amazon Elastic Compute Cloud (Amazon EC2) CPUUtilization as a complete health picture. Hypervisor CPU can look fine while the guest is out of memory, swapping, or waiting on a noisy neighbor. Vended AWS/EC2 metrics do not include memory or disk used percent. CPU steal and in-guest memory require the CloudWatch agent (or another in-guest collector). For burstable instances, also watch CPUCreditBalance and CPUSurplusCreditBalance. For Amazon Elastic Block Store (Amazon EBS) gp2 volumes, BurstBalance and VolumeQueueLength often explain latency that CPU charts hide.

Statistics such as Average, Maximum, Sum, SampleCount, and percentiles (for example p95 and p99) describe a period. Percentiles need raw samples; they are not available when values are negative or when you published only a statistic set that does not meet CloudWatch’s equality rules. Alarms compare a metric (or a metric math expression that yields a single time series) to a threshold over evaluation periods and fire only after a sustained state change, not because the alarm merely sits in ALARM.

Metric math for SLOs, not for SEARCH-based alarms

Metric math lets you query multiple metrics and compute a new time series. A standard SAP-C02 pattern is Lambda Errors / Invocations (or HTTP 5xx / request count) as an error rate. Functions such as SUM, AVG, FILL, RATE, IF, ANOMALY_DETECTION_BAND, and SERVICE_QUOTA are written in uppercase. The final expression must be a single time series or an array of time series; a bare scalar such as AVG(m1) is not a valid final result.

The SEARCH function dynamically graphs matching metrics and, in a monitoring account, can find metrics in source accounts. You cannot create an alarm on SEARCH because SEARCH returns multiple time series and a math-based alarm can watch only one series. If the exam stem wants an alarm on “all new ALB 5xx metrics,” SEARCH is the dashboard tool; a metric math expression against a specific load-balancer metric, or an alarm on a well-known AWS/ApplicationELB metric with the right dimensions, is the alerting tool.

Use metric math for KPI translation: availability ≈ successful requests / total requests, SLO burn from error budget, or m1 / SERVICE_QUOTA(m1) to watch usage against a quota. Missing values in arithmetic are treated as 0, which can hide sparse metrics unless you use DATAPOINT_COUNT or FILL carefully. AWS documents warn that FILL on delayed metrics can stick an alarm in OK or ALARM; M of N datapoints is the usual workaround.

Logs, Logs Insights, and high-cardinality contributors

Amazon CloudWatch Logs stores log events in log groups and log streams. Retention is configured per log group; unlike metrics, logs do not silently roll up on a 15-month metric schedule. Ship application logs, AWS Lambda logs, Amazon Elastic Container Service (Amazon ECS) and Amazon Elastic Kubernetes Service (Amazon EKS) container logs, and AWS CloudTrail (often via a centralized trail discussed in the security chapters) into groups with a naming convention the platform team can query.

CloudWatch Logs Insights is the interactive query language. Typical commands are fields, filter, stats, parse, sort, limit, and dedup. Example:

fields @timestamp, @message
| filter @message like /ERROR/
| stats count() by bin(5m)

JSON logs expose nested fields with dot notation. Logs Insights can extract a large number of JSON fields automatically; use parse when a field is not discovered. Field indexes can skip events that cannot match a filter, which matters when the exam mentions scanned volume. In a monitoring account, a single Insights query can target log groups in multiple source accounts at once.

Embedded Metric Format (EMF) lets applications write a structured log event that CloudWatch extracts into metrics, avoiding a separate PutMetricData flood. Contributor Insights builds time series of top-N contributors from logs (top talkers, top users, top URLs). That is the log-side counterpart to a noisy-neighbor hunt when the problem is one tenant or one IP, not one EC2 instance.

Metric alarms, composite alarms, and actions

A metric alarm watches one metric or one math expression. Configure the period, statistic or percentile, threshold, comparison, and how many evaluation periods must breach. Treat-missing-data settings (breaching, not breaching, ignore, missing) change behavior for sparse canaries and intermittent publishers.

A composite alarm watches other alarms with a rule expression such as (ALARM("CPUUtilizationTooHigh") OR ALARM("DiskReadOpsTooHigh")) AND OK("NetworkOutTooHigh"). Operators are AND, OR, and NOT; you can require child states of ALARM, OK, or INSUFFICIENT_DATA. Composite alarms exist to reduce paging: child alarms can have no notification, while the composite notifies Amazon Simple Notification Service (Amazon SNS), invokes AWS Lambda, creates a Systems Manager OpsItem, or starts an Incident Manager incident only when the customer-facing combination is true. Do not create a dependency cycle of composite alarms; AWS documents that cyclic composites stop evaluating.

Alarms can also drive Auto Scaling policies. For operational excellence, prefer an alarm that represents an SLO or KPI (p99 latency, error rate, Synthetics success percent) over paging on every host CPU spike.

CloudWatch agent, Synthetics, dashboards, and the trace map

The CloudWatch agent collects in-guest metrics (memory, disk, swap, netstat, processes, CPU steal where the OS exposes it) and tail-ships log files from Amazon EC2 and on-premises servers. Install it at fleet scale with AWS Systems Manager (Distributor package or the AmazonCloudWatch-ManageAgent Run Command document) and store the JSON agent configuration in Parameter Store so State Manager can enforce it. Without the agent, a noisy-neighbor scenario is often invisible: hypervisor CPUUtilization stays moderate while guest memory, steal, or EBS queue depth explains the SLO miss.

CloudWatch Synthetics runs canaries—Node.js, Python, or Java scripts on a schedule, as often as once per minute—that follow customer routes even when traffic is zero. Canaries create Lambda functions in your account and publish metrics in the CloudWatchSynthetics namespace with a CanaryName dimension (and StepName when step helpers are used). They store screenshots and HAR files as artifacts, can run in a virtual private cloud (VPC), and can enable AWS X-Ray active tracing so the canary appears on the trace map. A failed canary is an early warning, not a substitute for real-user metrics; use it as a child of a composite SLO alarm.

CloudWatch dashboards are operator-designed collections of widgets: metrics, math, alarms, and Logs Insights. They do not automatically discover new microservices. The historical ServiceLens map, which correlated traces, metrics, logs, and alarms, is combined into the X-Ray trace map in the CloudWatch console (Trace Map under X-Ray traces). Application Signals can additionally list services, service level objectives (SLOs), Synthetics canaries, and dependencies. On the exam, pick a dashboard when humans need a curated executive or runbook view; pick the trace map / Application Signals when the question is “which service or edge is the bottleneck, and what logs attach to that node?”

Cross-account observability

CloudWatch cross-account observability links source accounts to a monitoring account in the same Region using Observability Access Manager. The monitoring account creates a sink; source accounts create links that share metrics (all namespaces or a filter), log groups (all or a filter), X-Ray traces, Application Signals services and SLOs, Application Insights applications, and Internet Monitor monitors. From the monitoring account you can graph mixed-account widgets, create alarms on source-account metrics, run Logs Insights across accounts, and view source nodes on the trace map. AWS documents that this sharing has no extra charge for logs and metrics and Application Signals, and that the first trace copy is free; confirm current pricing on the CloudWatch pricing page rather than memorizing a dollar amount.

Do not confuse this with metric streams to a partner, CloudWatch Logs subscription filters to a central Lambda, or an organization CloudTrail trail. Those still matter for archive and security analytics, but they are not the interactive “search any linked account from one console” design the observability feature provides.

SAP-C02 scenario: noisy-neighbor metrics

A payments API runs on a shared-tenancy Auto Scaling group of burstable instances behind an Application Load Balancer. Average CPUUtilization stays near 35 percent, yet p99 latency jumps during another tenant’s batch window. The professional-architect move is not to raise the CPU alarm threshold. Enable detailed monitoring, install the CloudWatch agent for memory, disk, and steal, graph CPUCreditBalance and EBS VolumeQueueLength, put application p99 and Synthetics success percent on the same dashboard, and wrap “SLO breach AND host pressure” in a composite alarm. Use Contributor Insights if logs show one tenant dominating. That telemetry package is what Task 3.1 and Task 3.3 are testing: measurable KPIs, the right toolset, and alarms operators will still trust at 02:00.