11.3 Synthetic Monitoring with CloudWatch Synthetics Canaries

Synthetic Monitoring vs. Passive Observability

Traditional infrastructure and application monitoring is inherently passive: metrics, logs, and traces are generated only when real end users interact with the system. While vital, passive observability suffers from three critical blind spots in enterprise environments:

1. Low-Traffic Masking: During off-peak hours, weekends, or scheduled maintenance windows, user traffic drops significantly. A severe infrastructure outage (such as an expired TLS certificate, DNS misconfiguration, or broken database connection) may generate zero alerts simply because no users are attempting to connect.
2. The 200 OK False Positive Trap: An application frontend may successfully return HTTP 200 OK with an empty HTML body, or fail to render because a client-side JavaScript bundle failed to load from a CDN. Server-side metrics show a 100% success rate, while 100% of human users experience a broken application.
3. Post-Deployment Verification Latency: During blue/green or canary deployments, relying on real user error rates to detect bugs forces customers to act as involuntary software testers.

Amazon CloudWatch Synthetics solves these dilemmas through proactive, continuous testing. Synthetics canaries are modular, automated scripts that run on a continuous schedule (e.g., every 1, 5, or 15 minutes, 24 hours a day, 7 days a week). Canaries simulate the exact actions of a real user—clicking buttons, navigating checkout flows, filling forms, and validating API responses—to verify availability, latency, and functional correctness before end users are impacted.

---

Canary Blueprints & Runtime Engines

Under the hood, CloudWatch Synthetics packages canary scripts as specialized AWS Lambda functions running in a managed runtime environment. Synthetics supports two primary headless browser and automation runtimes:

• Node.js: Runs Chromium via Puppeteer (managed runtime syn-nodejs-puppeteer-*).
• Python: Runs Chromium via Selenium WebDriver (managed runtime syn-python-selenium-*).

Out-of-the-Box Canary Blueprints

For standard monitoring tasks, AWS provides pre-built Canary Blueprints that require minimal configuration:

Custom Script Anatomy: synthetics.executeStep()

For advanced multi-step workflows, engineers author custom scripts using the Synthetics SDK. The fundamental building block is synthetics.executeStep(), which segments a complex journey into distinct, individually timed phases:

const synthetics = require('Synthetics');
const log = require('SyntheticsLogger');

const checkoutFlowCanary = async function () {
    const page = await synthetics.getPage();

    // Step 1: Navigate to Home and Verify Title
    await synthetics.executeStep('NavigateToHome', async function () {
        await page.goto('https://shop.example.com', { waitUntil: ['domcontentloaded', 'networkidle0'] });
        await synthetics.takeScreenshot('home_page_loaded', 'loaded');
        const title = await page.title();
        if (!title.includes('Online Store')) {
            throw new Error(`Unexpected page title: ${title}`);
        }
    });

    // Step 2: Search for Item
    await synthetics.executeStep('SearchCatalog', async function () {
        await page.type('#search-input', 'wireless keyboard');
        await Promise.all([
            page.waitForNavigation({ waitUntil: 'networkidle0' }),
            page.click('#search-button')
        ]);
        await synthetics.takeScreenshot('search_results', 'displayed');
    });

    // Step 3: API Inventory Check
    await synthetics.executeStep('VerifyInventoryAPI', async function () {
        const response = await synthetics.executeHttp({
            hostname: 'api.example.com',
            path: '/v1/inventory/item-1234',
            method: 'GET',
            headers: { 'Authorization': 'Bearer <token>' }
        });
        if (response.statusCode !== 200) {
            throw new Error(`Inventory API returned HTTP ${response.statusCode}`);
        }
    });
};

exports.handler = async () => {
    return await checkoutFlowCanary();
};

---

Canary Artifacts, S3 Lifecycle & CloudWatch Metrics

Every time a canary executes, CloudWatch Synthetics collects a rich set of diagnostic artifacts and stores them in a designated Amazon S3 bucket under the prefix canary/<region>/<canary-name>/<timestamp>/.

Diagnostic Artifact Types

• Step Screenshots: High-resolution PNG images captured at each step (or automatically captured on failure). These allow engineers to visually inspect UI layout failures, modal dialog traps, or unexpected error banners.
• HTTP Archive (HAR) Files: A complete, detailed record of every HTTP/HTTPS network transaction executed by the browser during the run. HAR files capture DNS lookup duration, TCP connection time, SSL/TLS handshake latency, Time to First Byte (TTFB), content download time, HTTP request/response headers, and response sizes for every image, script, and API call.
• Execution and Console Logs: Captures all browser console warnings, JavaScript errors, network failures, and explicit logs emitted via SyntheticsLogger.

[!CAUTION] S3 Cost Explosion Watchout: A single GUI canary running once per minute generates 1,440 executions per day, producing thousands of PNG screenshots and HAR files daily. Without automated lifecycle management, S3 storage costs escalate rapidly. Always configure an Amazon S3 Lifecycle Rule on the canary artifact bucket to transition objects to S3 Standard-IA or expire/delete artifacts after 30 to 90 days.

CloudWatch Synthetics Metrics

Canary execution results are published to the CloudWatchSynthetics namespace with the dimension CanaryName:

• SuccessPercent: Percentage of canary runs that succeeded (100 or 0 for a single run; average over an evaluation window).
• Duration: Total execution time in milliseconds.
• Failed: Count of failed canary runs.
• 2xx, 4xx, 5xx: Count of HTTP status codes returned by endpoints probed during the run.

---

VPC Canaries: Monitoring Private Internal Microservices

By default, CloudWatch Synthetics canaries run in a managed VPC operated by AWS and can only probe publicly accessible internet endpoints. However, in enterprise microservice architectures, core internal APIs and administration portals reside in private VPC subnets behind internal Application Load Balancers or private API Gateways with no internet ingress.

VPC Canary Architecture & Networking Prerequisites

To probe private endpoints, you configure the canary as a VPC Canary:

1. The canary is configured with a VPC ID, two or more private Subnet IDs, and one or more Security Group IDs.
2. The underlying Lambda execution platform provisions Elastic Network Interfaces (ENIs) directly inside the designated private subnets.
3. The canary executes its requests from an IP address within the private subnet IP space.

Private Subnet (AZ-a)
┌─────────────────────────────────────────────────────────┐
│ Canary ENI ──(HTTP :8080)──> Internal Microservice ALB  │
│     │                                                   │
│     ├─(HTTPS :443)──> S3 Gateway VPC Endpoint (Artifacts)│
│     │                                                   │
│     └─(HTTPS :443)──> CloudWatch VPC Endpoints          │
│                       - com.amazonaws.<reg>.monitoring  │
│                       - com.amazonaws.<reg>.logs        │
└─────────────────────────────────────────────────────────┘

The VPC Endpoint vs. NAT Gateway Requirement

A frequent DOP-C02 exam scenario involves a VPC canary that successfully probes the private internal microservice, but the canary run itself fails with a timeout or error status.

Root Cause: When running inside a customer VPC, the canary is bound to the VPC's routing rules. In addition to reaching the target microservice, the canary must communicate with AWS public service endpoints to:

• Write execution logs to Amazon CloudWatch Logs
• Publish metric data points to Amazon CloudWatch
• Upload screenshots and HAR files to Amazon S3

If the private subnet lacks internet egress and lacks VPC endpoints, the canary times out attempting to upload its artifacts and fails.

Remediation Options:

• Option A (Air-Gapped / PrivateLink): Provision a Gateway VPC Endpoint for Amazon S3 (free of charge) and Interface VPC Endpoints (AWS PrivateLink) for CloudWatch Monitoring (monitoring.<region>.amazonaws.com) and CloudWatch Logs (logs.<region>.amazonaws.com). Ensure the canary's security group allows outbound HTTPS (port 443) to these endpoints.
• Option B (NAT Egress): Ensure the private subnets have a valid route (0.0.0.0/0) targeting an Amazon NAT Gateway located in a public subnet with an attached Internet Gateway.

---

Automated Deployment Guardrails & CodeDeploy Integration

In modern CI/CD pipelines, CloudWatch Synthetics canaries serve as automated deployment safety gates that trigger automated rollbacks during canary or blue/green traffic shifting.

CodeDeploy Traffic Shifting & Rollback Triggers

When deploying application updates using AWS CodeDeploy (for ECS, Lambda, or EC2/On-Premises), traffic shifting is executed progressively:

• CodeDeployDefault.LambdaCanary10Percent5Minutes (shifts 10% of traffic, waits 5 minutes, then shifts remaining 90%).
• CodeDeployDefault.LambdaLinear10PercentEvery1Minute (shifts 10% every minute over 10 minutes).
• CodeDeployDefault.ECSLinear10PercentEvery3Minutes.

To automate rollback on failure:

1. Configure a CloudWatch Synthetics canary to run every 1 minute against the application endpoint.
2. Create a CloudWatch Alarm on the canary's SuccessPercent metric (SuccessPercent < 100 for 1 evaluation period of 1 minute) or Failed >= 1.
3. In the AWS CodeDeploy Deployment Group configuration, navigate to Deployment Alarms and attach the CloudWatch Synthetics alarm, enabling Roll back when alarms are triggered.
4. During traffic shifting (when 10% of user traffic is hitting the new replacement version), if the synthetic canary encounters an HTTP 5xx error, broken DOM selector, or breached latency threshold, the alarm enters ALARM state.
5. CodeDeploy immediately aborts the deployment, terminates traffic shifting, reverts 100% of traffic back to the original stable version, and flags the deployment as Failed.

CodeDeploy Lifecycle Hooks with Synthetics

For pre-traffic verification, AWS CodeDeploy supports lifecycle event hooks:

• Lambda Deployments: BeforeAllowTraffic hook invokes a validation Lambda function that triggers a one-off Synthetics canary execution against the newly deployed target revision before any production traffic shifts.
• ECS Blue/Green Deployments: The replacement task set is registered to a test listener port (e.g., port 8443) on the ALB. The canary executes its validation test against port 8443 during the AfterAllowTestTraffic hook. If the canary passes, traffic shifts to port 443; if it fails, CodeDeploy triggers an automated rollback before production users ever touch the new build.

---

Distributed Tracing Integration with AWS X-Ray

When a synthetic canary fails or detects elevated latency, pinpointing the root cause across a distributed microservice architecture can be difficult. CloudWatch Synthetics natively integrates with AWS X-Ray:

1. Active Tracing Configuration: Set ActiveTracing: true in the canary configuration.
2. Trace Propagation: When active tracing is enabled, the Synthetics runtime automatically injects the X-Amzn-Trace-Id HTTP header into every synthetic HTTP request it originates.
3. Downstream Trace Correlation: As the synthetic request traverses the architecture—passing through an Application Load Balancer, Amazon API Gateway, ECS containers, AWS Lambda functions, and downstream DynamoDB or Aurora databases—each instrumented service propagates the trace ID.
4. Service Map Visualization: DevOps engineers open the AWS X-Ray console or CloudWatch ServiceLens to view the complete synthetic transaction call tree. If the canary's Duration metric spikes from 200 ms to 4,000 ms, the X-Ray service map visually highlights the exact downstream microservice or SQL query causing the latency bottleneck.

---

DOP-C02 Exam Watchouts & Troubleshooting