1.2 CloudWatch Agent Architecture & Deployment

CloudWatch Agent Architecture & Hypervisor Visibility Boundary

Operating infrastructure across Amazon EC2, Amazon ECS, and Amazon EKS requires telemetry at both hypervisor and guest operating system layers. Early AWS setups relied on separate tools: the legacy CloudWatch Logs agent (awslogs) and python/perl scripts (mon-scripts). AWS unified these into the modern Amazon CloudWatch Agent, which runs on Linux and Windows to collect system metrics and log streams through a single daemon.

Understanding the hypervisor boundary is essential for CloudOps:

+-------------------------------------------------------------+
| GUEST OPERATING SYSTEM (Requires CloudWatch Agent)          |
|  - Available & Used Memory (mem_used_percent)               |
|  - Filesystem & Inode Utilization (disk_used_percent)       |
|  - Swap Usage (swap_used_percent)                           |
|  - Active TCP Sockets & State Counters (netstat_*)          |
|  - Process-Level CPU & RSS Memory (procstat plugin)         |
+-------------------------------------------------------------+
==================== HYPERVISOR BOUNDARY ======================
+-------------------------------------------------------------+
| AWS HYPERVISOR (Nitro / Xen - Default EC2 Metrics)          |
|  - CPU Utilization (CPUUtilization)                         |
|  - Hypervisor Network Traffic (NetworkIn, NetworkOut)       |
|  - Instance Store Disk I/O (DiskReadBytes, DiskWriteBytes)  |
|  - Physical Hardware Status (StatusCheckFailed_System)      |
+-------------------------------------------------------------+

The hypervisor manages physical CPUs, virtualized NICs, and storage attachments, providing metrics like CPUUtilization with zero in-guest software. However, the hypervisor cannot inspect guest kernel memory pages, virtual memory allocations, or filesystem directory structures. Memory and disk metrics remain invisible without the CloudWatch Agent.

---

Guest Operating System Metrics Collection on EC2

The CloudWatch Agent collects five core categories of guest metrics:

1. Memory Metrics: The hypervisor cannot detect memory exhaustion. The agent captures mem_used_percent, mem_available, mem_used, and mem_total. Auto-scaling and paging alarms typically target mem_used_percent.
2. Disk and Filesystem Metrics: Gathers storage utilization per mount point and filesystem type, including disk_used_percent, disk_used, disk_free, and disk_inodes_used. Inode monitoring prevents file-creation failures when storage blocks remain available.
3. Swap Metrics: Virtual memory paging counters include swap_used_percent and swap_used. High swap usage coupled with elevated disk I/O reveals severe memory exhaustion and thrashing.
4. Network Socket Metrics (netstat): Gathers TCP connection states, such as netstat_tcp_established, netstat_tcp_time_wait, and netstat_tcp_close_wait, highlighting socket exhaustion or unclosed database connections.
5. Process Metrics (procstat): The procstat plugin monitors individual system processes using pid_file, exe name, or regex pattern. It publishes metrics like procstat_cpu_usage and procstat_memory_rss for daemons such as NGINX, Apache, or MySQL.

---

Centralized Configuration Management with Systems Manager

The CloudWatch Agent relies on a single JSON configuration: amazon-cloudwatch-agent.json. The configuration schema divides into three primary stanzas:

• "agent": Controls operational parameters, collection intervals, and runtime OS user (run_as_user).
• "metrics": Configures metric plugins under "metrics_collected", defining collection intervals, units, and aggregation dimensions (such as InstanceId and AutoScalingGroupName).
• "logs": Specifies log files under "logs_collected", mapping source file paths to CloudWatch log groups and streams.

Configuration Creation & Parameter Store Integration

Administrators generate baseline configurations using the interactive wizard:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

In enterprise environments, configurations are stored in AWS Systems Manager (SSM) Parameter Store as standard String parameters (e.g., AmazonCloudWatch-linux or AmazonCloudWatch-windows).

Fleet-Wide Deployment via SSM Run Command

Deploying or updating the agent across hundreds of EC2 instances executes without SSH access using SSM Run Command with the document AmazonCloudWatch-ManageAgent:

• action: configure
• mode: ec2
• optionalConfigurationSource: ssm
• optionalConfigurationLocation: AmazonCloudWatch-linux
• optionalRestart: yes

IAM Role Prerequisite: Every target EC2 instance requires an IAM instance profile with the managed policy CloudWatchAgentServerPolicy (granting permissions for cloudwatch:PutMetricData, logs:PutLogEvents, logs:CreateLogStream, and ssm:GetParameter).

---

Container Observability: Amazon ECS and Amazon EKS

Containerized architectures require specialized observability patterns:

Amazon ECS Container Insights

Amazon ECS Container Insights delivers monitoring for ECS clusters, services, and tasks. It is activated at the cluster level via CLI or console:

aws ecs update-cluster-settings --cluster ProductionCluster --settings name=containerInsights,value=enabled

ECS automatically collects task-level CPU and memory utilization, network I/O, and storage metrics without manual agent configuration. Metrics are emitted as Embedded Metric Format (EMF) logs into the log group /aws/ecs/containerinsights/<cluster-name>/performance. For application logs, task definitions configure the awslogs driver, streaming stdout/stderr to CloudWatch Logs.

Amazon EKS Container Insights with Enhanced Observability

In Kubernetes environments on Amazon EKS:

1. CloudWatch Agent DaemonSet: Deployed across worker nodes alongside the AWS Distro for OpenTelemetry (ADOT) collector within the amazon-cloudwatch namespace. Enhanced observability captures pod-level CPU/memory utilization, control plane performance (API server latency, etcd metrics), and persistent volume storage metrics (pod_volume_fs_used_percent) via the EBS CSI driver.
2. AWS for Fluent Bit DaemonSet: High-performance log processor deployed as a Kubernetes DaemonSet. Fluent Bit parses container logs, enriches them with Kubernetes metadata (namespace, pod name), and routes them to dedicated CloudWatch log groups:
   • /aws/eks/<cluster-name>/application: Application container logs
   • /aws/eks/<cluster-name>/dataplane: Kubelet, containerd, and node system logs
   • /aws/eks/<cluster-name>/host: Operating system audit and auth logs

---

Agent Management & Common Troubleshooting Traps

Administrators manage the agent locally with amazon-cloudwatch-agent-ctl:

• Start agent with SSM config:

  sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c ssm:AmazonCloudWatch-linux
  

• Check runtime status:

  sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status
  

• Diagnostic log path: /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log

Common Configuration Traps

1. Private Subnet Connectivity: Instances in private subnets without NAT require Interface VPC Endpoints (AWS PrivateLink) for CloudWatch (monitoring), CloudWatch Logs (logs), and Systems Manager (ssm, ec2messages, ssmmessages).
2. Missing IAM Permissions: Omitting CloudWatchAgentServerPolicy produces AccessDenied errors when publishing metrics or fetching SSM parameters.
3. Invalid Filesystem Paths: Specifying non-existent mount points in the disk plugin aborts startup or prevents disk metric emission.