1.1 Configuring Workload Observability

Workload Observability Across Compute, Serverless, and AI/ML

Production cloud operations require full-stack visibility across compute, serverless, and AI/ML workloads. In AWS, observability relies on metrics, logs, and traces. Effective monitoring configures workload-specific telemetry reflecting true operational health.

For Amazon EC2 compute workloads, the AWS hypervisor monitors virtual hardware metrics (CPU, basic disk, network) without host-installed agents. However, guest operating system internals—memory utilization, file system space, and network sockets—require the unified CloudWatch Agent.

For serverless architectures, AWS Lambda automatically emits operational metrics to Amazon CloudWatch:

• Invocations: Total execution requests triggered by event sources.
• Errors: Invocations resulting in unhandled exceptions, runtime crashes, or timeouts.
• Duration: Execution elapsed wall-clock time in milliseconds.
• Throttles: Invocations rejected due to exceeding reserved concurrency or account limits.
• ConcurrentExecutions: Simultaneous active execution environments.

Amazon API Gateway monitors API traffic via Count (request volume), 4XXError (client errors), 5XXError (backend failures or gateway timeouts), Latency (total response time from client request to delivery), and IntegrationLatency (time spent waiting for the backend integration to respond). Comparing Latency against IntegrationLatency isolates whether delays originate in backend code or API Gateway overhead.

For AI/ML workloads on Amazon SageMaker real-time inference endpoints (AWS/SageMaker namespace), primary operational metrics include Invocations, ModelLatency (inference processing duration inside the container), OverheadLatency (request payload handling and network transmission), and hardware accelerators GPUUtilization and GPUMemoryUtilization.

---

CloudWatch Metrics Architecture & Storage Retention Tiers

CloudWatch Metrics stores timestamped time-series data defined by a MetricName, Namespace (e.g., AWS/EC2 or custom App/Billing), and up to 30 Dimensions (identifying key-value pairs).

EC2 supports two monitoring frequencies:

1. Standard Monitoring: 5-minute metric publication included at no additional charge.
2. Detailed Monitoring: 1-minute metric publication enabled per instance or launch template for an additional fee, allowing Auto Scaling groups to respond quickly to traffic surges.

Custom Metrics & High-Resolution Publishing

Workloads emit custom metrics using the PutMetricData API, specifying namespace, metric name, dimensions, timestamp, value, and unit (such as Seconds, Percent, or Bytes). Setting --storage-resolution 1 configures high-resolution metrics with 1-second intervals (standard metrics use 60-second intervals).

CloudWatch retains data across four distinct duration tiers:

---

CloudWatch Logs & CloudWatch Logs Insights

Amazon CloudWatch Logs organizes log ingestion through two core components:

• Log Groups: Define retention rules (1 day to 10 years, or Never Expire), AWS KMS customer managed key (CMK) encryption, and access policies.
• Log Streams: Discrete sequences of log events from the same source (such as an EC2 instance, ECS task, or Lambda container).

Metric Filters and Transformations

Metric filters scan incoming logs in real time against filter patterns:

• JSON pattern: { $.statusCode = 500 || $.statusCode = 502 }
• Space-delimited pattern: [ip, user, timestamp, request, status_code = 5*, response_size]

The metric transformation maps matched tokens to a custom metric. Administrators must set the Default Value parameter to 0. Without a default value, filters emit nothing when logs contain no matching events, creating sparse metrics that leave alarms in INSUFFICIENT_DATA or stale states.

CloudWatch Logs Insights

Logs Insights provides an interactive query syntax to analyze logs across groups:

• fields: Selects fields to display.
• filter: Evaluates boolean or regex conditions.
• stats: Calculates aggregations grouped by time bins.
• sort and limit: Orders and caps result sets.

fields @timestamp, @message, status_code, response_time
| filter status_code >= 500
| stats count() as error_count by bin(5m)
| sort error_count desc
| limit 20

---

AWS CloudTrail: Governance, Data Events & CloudTrail Lake

AWS CloudTrail audits API calls across accounts:

1. Management Events: Record control plane operations (e.g., RunInstances, CreateBucket). Captured by default across all Regions in a 90-day free event history.
2. Data Events: Record high-volume data plane operations (e.g., S3 GetObject/PutObject, Lambda InvokeFunction). Disabled by default; billed per event.

Multi-Region trails record events across all Regions and capture global services (IAM, STS, CloudFront) in us-east-1. Log File Integrity Validation generates SHA-256 digest files signed with RSA keys, enabling cryptographic tampering verification via aws cloudtrail validate-logs. CloudTrail Lake provides a managed SQL data lake for querying audit logs with configurable retention up to 10 years (3,653 days).

---

Amazon Managed Service for Prometheus (AMP)

Amazon Managed Service for Prometheus (AMP) delivers serverless, scalable, Prometheus-compatible monitoring for containerized workloads. The AWS Distro for OpenTelemetry (ADOT) collector runs as an EKS DaemonSet, scraping Prometheus endpoints and forwarding metrics to AMP via remote_write with AWS SigV4 authentication. AMP evaluates alerting and recording rules using native PromQL, routing alerts to Amazon SNS topics.

---

Operational Best Practices

• Batch Custom Metrics: Batch up to 1,000 metrics or 1 MB per PutMetricData call, or use Embedded Metric Format (EMF) via logs to avoid API throttling.
• Log Retention Governance: Define explicit log retention policies (such as 30 or 90 days) in Infrastructure as Code to eliminate unbounded storage costs.