2.2 Load Testing, Performance Benchmarking & Synthetic Canaries

Functional testing verifies whether software produces correct outputs for given inputs; non-functional testing verifies how the system behaves under operational stress. On the AWS DOP-C02 exam, performance testing and synthetic monitoring are tested as automated quality gates that guard production availability and latency Service Level Objectives (SLOs).

DevOps engineers must understand how to generate large-scale distributed load using managed container architectures, how to evaluate latency percentiles against strict performance budgets in staging, and how to use synthetic canaries as automated circuit breakers during progressive rollouts.

---

Distributed Load Testing on AWS Architecture

A single test client running JMeter or Locust on an EC2 instance or local developer machine quickly runs into CPU, memory, or network socket exhaustion (ephemeral port limits) when simulating tens of thousands of concurrent users. To solve this, AWS provides the Distributed Load Testing on AWS implementation—an open-source AWS Solutions implementation.

                  [API Gateway / Web UI]
                             │
                             ▼
                   [AWS Step Functions]
                             │
              ┌──────────────┴──────────────┐
              ▼                             ▼
       [Amazon DynamoDB]            [Amazon S3 Bucket]
     (Test State & Config)          (Scripts & Artifacts)
              │                             │
              └──────────────┬──────────────┘
                             ▼
                 [Amazon ECS on AWS Fargate]
                 ┌─────────────────────────┐
                 │ Fargate Task (Worker 1) │ ──┐
                 │ Fargate Task (Worker 2) │ ──┼──> [Target Application]
                 │ Fargate Task (Worker N) │ ──┘
                 └─────────────────────────┘
                             │
                             ▼
                 [Amazon CloudWatch Metrics]
                 (Real-Time Concurrency/Latency)

Architectural Components

• Orchestration Layer: AWS Step Functions orchestrates the end-to-end load test lifecycle: validating test parameters, launching containers, monitoring progress, and aggregating final test reports.
• Execution Layer: Amazon ECS on AWS Fargate executes containerized test workers. The solution can deploy regional stacks and simulate tens of thousands of concurrent users across multiple Regions, but capacity depends on the test script, per-task CPU and memory, Fargate quotas, and calibration. It does not promise a fixed request rate or task count.
• Supported Test Types:
  • Apache JMeter: Upload JMX test plans for complex HTTP/S scenarios.
  • Grafana k6: Upload JavaScript k6 test scripts.
  • Locust: Upload Python test scripts with dynamic user behavior.
  • Single HTTP Endpoint: Configure a simple endpoint test without supplying a script. The solution runs these test types through the Taurus automation framework; Taurus is the orchestration layer rather than a fourth user-authored test type.
• Storage & Telemetry: Test scenarios and raw results are stored in Amazon S3; test execution status is tracked in Amazon DynamoDB; real-time response times, throughput, and HTTP status codes are pushed to Amazon CloudWatch Metrics.

---

Performance Benchmarking & Latency Budgets in Staging

To prevent performance regressions from reaching production, teams integrate automated performance benchmarking into staging pipeline stages. Staging environments must maintain architectural parity with production (e.g., equivalent database provisioned IOPS, similar caching tiers, and identical container resource allocations).

Why Averages Mask Catastrophic Failures: The Percentile Rule

Exam scenarios consistently emphasize evaluating latency percentiles rather than arithmetic averages:

• Average Latency (Mean): Masks high-latency tail events. If 99 requests take 10ms and 1 request takes 10,000ms, the average is ~109ms. The system appears healthy while 1 in 100 users experiences complete failure.
• P90 / P95 Latency: Measures the experience of the 90th or 95th percentile of users, highlighting micro-stalls and thread starvation.
• P99 / P99.9 Latency (Tail Latency): Crucial for microservice architectures. In an e-commerce checkout page that issues 50 statistically independent backend calls, a one-percent per-call tail event appears at least once in about 39.5% of page requests: 1 - 0.99^50. Real dependencies can also be correlated, so teams measure the end-to-end distribution instead of inferring it from an average. Tail latency directly drives user-perceived abandonment.

Automated Latency Gating in Pipelines

An automated performance gate script running inside AWS CodeBuild or AWS Step Functions queries CloudWatch or parses Locust/JMeter output against defined latency budgets:

#!/usr/bin/env bash
set -euo pipefail

echo "Evaluating Staging Performance Benchmarks against SLOs..."

# Extract latency metrics from test execution summary (in milliseconds)
P95_LATENCY=$(jq '.metrics.p95' load-test-summary.json)
P99_LATENCY=$(jq '.metrics.p99' load-test-summary.json)
ERROR_RATE=$(jq '.metrics.error_rate' load-test-summary.json)

MAX_ALLOWED_P95=150   # 150ms
MAX_ALLOWED_P99=300   # 300ms
MAX_ERROR_RATE=0.001  # 0.1%

if (( $(echo "$P99_LATENCY > $MAX_ALLOWED_P99" | bc -l) )); then
  echo "ERROR: P99 latency ($P99_LATENCY ms) exceeded budget ($MAX_ALLOWED_P99 ms)!" >&2
  exit 1
fi

if (( $(echo "$ERROR_RATE > $MAX_ERROR_RATE" | bc -l) )); then
  echo "ERROR: Error rate ($ERROR_RATE) exceeded allowable threshold ($MAX_ERROR_RATE)!" >&2
  exit 1
fi

echo "All performance benchmarks satisfied. Proceeding to production promotion."
exit 0

If any threshold is violated, the script exits with status 1, failing the CodeBuild action and blocking progression to production.

---

Amazon CloudWatch Synthetics Canaries

While load testing evaluates capacity under synthetic stress, synthetic monitoring continuously verifies application availability and user experience from the outside in. Amazon CloudWatch Synthetics allows you to create canaries—configurable, automated scripts that run on a schedule (e.g., every 1 or 5 minutes) to monitor endpoints and APIs.

Runtime Environments & Blueprints

Canaries run inside AWS Lambda micro-runtimes using headless browser frameworks:

• Node.js (Puppeteer): Headless Chromium automation for JavaScript-heavy web applications.
• Python (Selenium Webdriver): Python automation for web and API validation.

CloudWatch Synthetics provides built-in blueprints:

1. Heartbeat Monitoring: Performs periodic HTTP/S GET or POST calls to an endpoint, tracking status code, response time, and DNS latency.
2. API Canaries: Executes multi-step RESTful API sequences, sending JSON payloads, authenticating via OAuth/Cognito tokens, and validating response schema and headers.
3. GUI / Workflow Canaries: Fully automates end-user web interactions—clicking buttons, filling text fields, navigating checkouts, waiting for DOM elements, capturing user-perceived performance metrics, and taking full-page screenshots.

Canary Artifacts & Storage

Every canary run generates rich diagnostic artifacts stored in a dedicated Amazon S3 bucket and CloudWatch Logs:

• Screenshots: PNG captures of the browser at designated interaction steps or upon failure.
• HAR (HTTP Archive) Files: Complete network logs detailing every asset loaded, HTTP request/response headers, and timing breakdowns.
• Execution Logs: Console outputs and stack traces from the Puppeteer/Selenium script.

Critical Architecture: Canaries Inside a VPC

By default, canaries run in an AWS-managed VPC with internet access. However, testing private staging environments, internal Application Load Balancers, or private API Gateways requires configuring the canary to run inside your private VPC subnets.

[!CAUTION] The In-VPC Canary Network Trap: When a canary is attached to a private VPC subnet, it uses an Elastic Network Interface (ENI). It can no longer access public AWS APIs by default. The canary must write screenshots and HAR logs to Amazon S3 and write telemetry to CloudWatch Logs. If your VPC lacks an Amazon S3 Gateway VPC Endpoint and a CloudWatch Logs Interface Endpoint (or NAT Gateway routing), the canary will fail with connection timeout errors, even if your application endpoint responded perfectly!

---

Integrating Canaries as CodeDeploy Deployment Gates

One of the most heavily tested patterns on the DOP-C02 exam is using CloudWatch Synthetics canaries as automated circuit breakers during AWS CodeDeploy progressive deployments.

Traffic Shifting Modes in CodeDeploy

When deploying to AWS Lambda, Amazon ECS, or EC2, CodeDeploy supports progressive traffic shifting:

• Canary Deployments: A small percentage of traffic is routed to the new version for a specified time window, followed by a complete shift if healthy (e.g., Canary10Percent5Minutes or Canary10Percent15Minutes).
• Linear Deployments: Traffic shifts in equal increments over time until 100% is reached (e.g., Linear10PercentEvery1Minute or Linear10PercentEvery3Minutes).

Automated Rollback Integration Flow

1. Targeted Testing via Hooks: During deployment, CodeDeploy executes lifecycle event hooks. For Lambda and ECS, the AfterAllowTestTraffic hook allows validation scripts or test traffic to hit the replacement revision before any production user traffic is routed.
2. Canary Monitoring During Traffic Shifting: As live production traffic shifts (e.g., 10% on the new revision), the CloudWatch Synthetics canary continuously executes synthetic transactions against the service.
3. Alarm Evaluation: A CloudWatch Alarm monitors the canary metrics published to the CloudWatchSynthetics namespace under the CanaryName dimension (for example, Failed >= 1 or SuccessPercent < 100).
4. Instant Rollback: The CodeDeploy Deployment Group is configured with alarmConfiguration. If the canary alarm enters ALARM state at any point during the traffic shifting window:
   • CodeDeploy immediately cancels the deployment.
   • 100% of traffic is instantly redirected back to the original healthy revision.
   • Replacement tasks or instances are terminated.
   • CodePipeline marks the deployment stage as Failed.

---

Comparison: Testing & Monitoring Paradigms

---

Exam Watchouts & Operational Pitfalls

[!IMPORTANT] Canary Artifact S3 Bucket Permissions: The IAM execution role assigned to a CloudWatch Synthetics canary must possess explicit permissions to write to the designated artifact S3 bucket (s3:PutObject, s3:GetBucketLocation). If custom KMS encryption is enabled on the bucket, the role must also have kms:GenerateDataKey and kms:Decrypt.

[!WARNING] Pre-Existing Alarms in CodeDeploy: If a CloudWatch alarm attached to a CodeDeploy deployment group is already in the ALARM state when the deployment begins, CodeDeploy will either fail the deployment immediately upon starting or ignore the alarm depending on configuration. Always ensure alarms are in OK status prior to initiating progressive deployments.

[!NOTE] Canary Test Data Idempotency: Synthetic canaries executing against staging or production systems must be strictly idempotent. If your canary simulates an e-commerce order checkout, it must either use dedicated synthetic test accounts with dummy payment gateways or automatically invoke a cancellation API to prevent phantom inventory reservations.