Distributed Tracing with AWS X-Ray
Key Takeaways
- An X-Ray trace collects all segments for one request; subsegments time downstream calls, and uninstrumented services appear as inferred nodes from the caller’s subsegment.
- Default sampling is a reservoir of the first request each second plus five percent of additional requests; console sampling rules share that reservoir across instances, and sampling is parent-based at the root.
- Trace and service-graph data are retained for 30 days; annotations are indexed (up to 50 per trace) for filters and groups, while metadata is not indexed.
- X-Ray classifies Error (4xx), Fault (5xx), and Throttle (429); the CloudWatch trace map colors nodes accordingly and is the combined ServiceLens / X-Ray service map.
- Correlate traces with logs by propagating the trace ID into log lines and using Trace Map View logs or Logs Insights; Application Signals attaches SLOs and KPIs to the same service graph.
Why tracing exists beside CloudWatch metrics
Metrics answer “how many and how hot.” They rarely answer “where did this one checkout spend 2.4 seconds?” Task 3.1’s monitoring strategy and Task 3.3’s bottleneck and SLA skills both assume you can follow a request across Amazon API Gateway, compute, and data stores. AWS X-Ray receives segments from instrumented resources, groups segments that share a request into a trace, and builds a service graph visualized as the trace map in the Amazon CloudWatch console.
AWS documentation states that the X-Ray service map and the CloudWatch ServiceLens map are combined into that trace map (choose Trace Map under X-Ray traces). CloudWatch Application Signals can list services, clients, Synthetics canaries, and dependencies, show health from service level objectives (SLOs), and drill into correlated X-Ray traces. On SAP-C02, “turn on X-Ray” is incomplete unless you also decide sampling, what you index, and how logs join the trace.
Segments, subsegments, traces, and headers
A segment records work on one compute resource: host identity, request and response, timing, and issues. Segment documents can be up to 64 kB. A segment may contain subsegments for downstream Amazon DynamoDB calls, HTTP APIs, SQL, or arbitrary functions. If the downstream service does not send its own segment, X-Ray creates an inferred segment from the caller’s subsegment so DynamoDB still appears on the map. If both sides are instrumented, the downstream segment replaces the inferred node for that service, while the edge still uses the upstream subsegment (round-trip latency including network).
A trace ID ties every segment for one request. The first X-Ray-integrated service adds the X-Amzn-Trace-Id header (Root, optional Parent, Sampled). Load balancers may supply X-Forwarded-For; AWS warns that client IPs from that header can be forged. Applications can strip incoming trace headers if untrusted clients should not dictate sampling. Trace data and service-graph data are retained for 30 days—much shorter than 15-month metric rollups—so long-term SLA reporting must export aggregates to CloudWatch metrics or a data lake, not rely on raw traces forever.
Sampling: reservoir, rate, and parent-based decisions
Reservoir, rate, and console-managed rules
To keep tracing representative and affordable, the X-Ray SDK samples. By default it records the first request each second (the reservoir) and five percent of additional requests (the rate). Local JSON sampling on many instances adds reservoirs together, which over-samples. Rules defined in the X-Ray / CloudWatch console let the service share the reservoir and assign per-instance quotas, and you can change rules without redeploying.
Rules have a priority from 1 to 9999 (lowest number wins), plus matchers for service name, service type, host, HTTP method, URL path, and resource ARN. Example: a high-priority rule with reservoir 1 and rate 100 percent on PUT /checkout/* during an incident, and a low rate on health checks. Sampling is parent-based: the root (API Gateway, load balancer, or first instrumented service) decides, and downstream services honor Sampled=1 or 0. A strict rule on “Service B” never applies if Service A always called it first. Put the rule on the entry point, or on workers that start new traces.
AWS Distro for OpenTelemetry (ADOT) and the X-Ray SDK use the CloudWatch agent as a sampling proxy (default TCP 2000). API Gateway, AWS AppSync, and AWS Step Functions support active tracing with the same rule model.
Annotations, metadata, groups, and error classes
Annotations are indexed key-value pairs for filter expressions and GetTraceSummaries. X-Ray indexes up to 50 annotations per trace. Use them for tenant ID, HTTP status class, or a synthetic “checkout_stage” that operations will search. Metadata can be objects and lists but is not indexed—store payloads you might open on a single trace, not keys you will filter on.
Groups wrap a filter expression, produce their own graph and summaries, and publish CloudWatch metrics every minute for matching traces. Updating a group’s filter does not rewrite old traces. Groups are billed by retrieved matching traces; use them to turn an SLA filter (“response time > 2 seconds AND service = checkout”) into a KPI metric you can alarm.
X-Ray classifies downstream and application failures as Error (400-series), Fault (500-series), and Throttle (429). The CloudWatch trace map outlines nodes in yellow, red, and purple respectively. The map can show up to 10,000 nodes and supports cross-account tracing when observability links exist. Choose a node or edge, then use metrics, alerts, and response-time distribution tabs; View logs opens associated CloudWatch logs when the node supports that integration.
| Concept | What it is | Exam use |
|---|---|---|
| Segment / subsegment | Timed work on a resource / downstream call | Find which hop ate the p99 |
| Default sampling | 1 req/s reservoir + 5% thereafter | Do not 100% sample health checks |
| Annotations vs metadata | Indexed filters vs unindexed detail | Tenant SLA search needs annotations |
| Group | Filter + CloudWatch metrics | KPI/SLA burn from traces |
| Trace map (ServiceLens combined) | Topology + correlated metrics/logs | Bottleneck vs curated dashboard |
Correlating traces with logs and SLAs
Correlation fails when logs have no trace ID. Instrument frameworks to log Root from X-Amzn-Trace-Id (or the OpenTelemetry trace id). Operators then run Logs Insights filter @message like /1-5759e988-.../ or use View logs from the selected trace-map node. Synthetics canaries with X-Ray active tracing appear on the map so a failed canary is a trace, not only a CloudWatchSynthetics metric.
CloudWatch RUM (real user monitoring) captures client-session performance and can send traces so the path starts at the browser rather than at API Gateway. Application Signals is the current place to attach SLOs to discovered services and jump into traces when an SLO burns. Translate a business SLA such as “checkout p99 under 400 ms during the month” into: (1) a CloudWatch metric math KPI on the ALB or service latency percentile, (2) an X-Ray group for traces above that threshold, (3) dashboards for humans, (4) composite alarms that page on SLO burn rather than on every 5xx. AWS does not publish an exam-level pass-rate percentage; do not confuse certification folklore with workload SLAs you must invent from the scenario.
SAP-C02 bottleneck analysis
A retailer sees checkout p99 at 2.4 seconds against a 400 ms SLO. CloudWatch shows Lambda duration p99 of 180 ms—so the function is not the whole story. Enable active tracing on API Gateway and Lambda, SDK-instrument DynamoDB and the payments HTTP client, raise sampling on POST /checkout only, and open the trace map. If the DynamoDB inferred node is red and the subsegment is 2.1 seconds on PutItem with throttle flags, the remediation is capacity, keys, or backoff—not “add Lambda memory.” If the edge to payments is slow but the payments segment is short, the time is in the network or queue. That distinction is exactly what segments versus subsegments are for.
Do not use CloudTrail as an application profiler; CloudTrail is an account API audit trail. Do not set reservoir and rate so high that you trace load-balancer health checks. Do not store tenant IDs only in metadata if you must filter traces for a regulated tenant’s SLA report. After you find the hop, the next chapter’s Systems Manager tools patch and automate the fleet; tracing told you where, not how to patch.
Checkout p99 is 2.4 seconds against a 400 ms SLO. Lambda duration p99 is 180 ms. The architect needs to identify the hop that dominates latency across API Gateway, Lambda, DynamoDB, and an external payments API. What is the most appropriate next design?
Operations must filter traces for a single tenant, build a 30-day SLA report of slow checkouts, and alarm when that population grows. Tenant identifiers are currently stuffed into unindexed metadata. What should they implement?
A trace map node for the payments integration shows red (server faults). Engineers need the matching application log lines for the same request, not a bulk export of all logs in the Region. What correlation approach should they use?