9.2 Unified CloudWatch Agent & Custom Metric Collection

Unified CloudWatch Agent Architecture and Capabilities

When Amazon EC2 instances run workloads in AWS, the native CloudWatch hypervisor metrics (AWS/EC2 namespace) provide visibility into the host boundary: CPUUtilization, NetworkIn, NetworkOut, DiskReadOps, and DiskWriteOps (for instance store volumes).

However, the hypervisor cannot inspect guest operating system internals due to virtualization isolation boundaries. The hypervisor has zero visibility into:

• RAM utilization (active, buffered, cached, free memory)
• Filesystem disk space utilization and inode consumption
• Swap file and pagefile utilization
• Operating system process performance (CPU and memory per process)
• Local application log files and Windows Event Viewer logs

The Unified CloudWatch Agent is a high-performance Go-based daemon that installs inside the guest OS on both Linux and Windows Server instances, as well as on-premises physical and virtual servers. It replaces legacy, deprecated tools (such as the legacy Python-based awslogsd agent and Perl monitoring scripts) with a single, consolidated telemetry collector.

---

Declarative Configuration: amazon-cloudwatch-agent.json

The behavior of the Unified CloudWatch Agent is controlled entirely by a JSON configuration file. The configuration is organized into three primary top-level blocks:

1. agent: Defines global runtime properties (collection intervals, execution user, debug logging).
2. metrics: Configures OS metric plugins, collection frequencies, custom dimensions, procstat, and embedded telemetry listeners (statsd and collectd).
3. logs: Configures log file harvesting, Windows Event Log channels, timestamp extraction, and multi-line parsing rules.

Production-Grade Configuration Example

{
  "agent": {
    "metrics_collection_interval": 60,
    "run_as_user": "cwagent"
  },
  "metrics": {
    "namespace": "CWAgent",
    "metrics_collected": {
      "mem": {
        "measurement": [
          "mem_used_percent",
          "mem_available_percent",
          "mem_used",
          "mem_available"
        ],
        "metrics_collection_interval": 60
      },
      "disk": {
        "measurement": [
          "used_percent",
          "inodes_free"
        ],
        "metrics_collection_interval": 60,
        "resources": [
          "/",
          "/var/log",
          "/data"
        ]
      },
      "swap": {
        "measurement": [
          "swap_used_percent"
        ]
      },
      "netstat": {
        "measurement": [
          "tcp_established",
          "tcp_time_wait"
        ],
        "metrics_collection_interval": 60
      },
      "procstat": [
        {
          "pattern": "nginx",
          "measurement": [
            "cpu_usage",
            "memory_rss"
          ]
        },
        {
          "exe": "payment-service",
          "measurement": [
            "cpu_usage",
            "memory_vms"
          ]
        }
      ],
      "statsd": {
        "service_address": ":8125",
        "metrics_collection_interval": 10,
        "metrics_aggregation_interval": 60
      }
    },
    "aggregation_dimensions": [
      ["AutoScalingGroupName"],
      ["InstanceId", "InstanceType"]
    ],
    "append_dimensions": {
      "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
      "ImageId": "${aws:ImageId}",
      "InstanceId": "${aws:InstanceId}",
      "InstanceType": "${aws:InstanceType}"
    }
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/application/*.log",
            "log_group_name": "/aws/ec2/production/application",
            "log_stream_name": "{instance_id}",
            "multi_line_start_pattern": "{datetime_format}",
            "timestamp_format": "%Y-%m-%d %H:%M:%S"
          }
        ]
      }
    }
  }
}

Advanced Metric Capabilities: procstat and statsd

• Process Monitoring with procstat: Monitors individual operating system processes. Processes can be matched using exe (executable name), pattern (command line regex matching), or pid_file (reading process ID from /var/run/*.pid). Metrics emitted include procstat_cpu_usage, procstat_memory_rss (resident set size), and procstat_num_threads.
• Embedded StatsD Listener: The agent can expose a local UDP or TCP socket (default port 8125). Applications running on the instance can send custom metrics using standard StatsD formatting (statsd.increment("checkout.success")). The agent buffers and aggregates these counters locally before publishing to CloudWatch, drastically reducing PutMetricData API calls.

---

Fleet-Wide Deployment and Automation with AWS Systems Manager (SSM)

In enterprise DevOps environments, manually connecting to instances via SSH or RDP to install and configure software is strictly prohibited. Fleet-wide agent lifecycle management is orchestrated entirely through AWS Systems Manager.

Centralized Configuration via SSM Parameter Store

Instead of baking configuration files into Amazon Machine Images (AMIs) or hardcoding them on local disks, store the agent configuration centrally in AWS Systems Manager Parameter Store:

• Parameter Name: /configuration/cloudwatch-agent/linux-production
• Type: String
• Value: The raw JSON configuration content.

Step-by-Step Automated Deployment Workflow

1. Attach IAM Instance Profile
   ├── CloudWatchAgentServerPolicy
   └── AmazonSSMManagedInstanceCore
       │
2. Store Agent JSON in SSM Parameter Store
   └── /enterprise/cwagent/linux-config
       │
3. Install Agent via SSM Run Command / State Manager
   └── Document: AWS-ConfigureAWSPackage (Package: AmazonCloudWatchAgent)
       │
4. Configure & Start Agent via SSM Run Command
   └── Document: AmazonCloudWatch-ManageAgent (Action: configure, Mode: ec2)

1. IAM Permissions: The EC2 instance profile must include:

   • CloudWatchAgentServerPolicy: Grants permissions to publish metrics (cloudwatch:PutMetricData), create log streams, and put log events.
   • AmazonSSMManagedInstanceCore: Grants permissions for the SSM Agent to communicate with Systems Manager.
   • Inline or custom policy allowing ssm:GetParameter on the specific Parameter Store configuration ARN.

2. Package Installation: Use SSM Run Command or SSM State Manager with document AWS-ConfigureAWSPackage:

   • action: Install
   • name: AmazonCloudWatchAgent

3. Agent Configuration & Startup: Use SSM document AmazonCloudWatch-ManageAgent:

   • action: configure
   • mode: ec2
   • optional_configuration_source: ssm
   • optional_configuration_location: /configuration/cloudwatch-agent/linux-production
   • optional_restart: yes

4. Continuous Drift Management with State Manager: Create an SSM State Manager Association targeting instances by tag (e.g., Environment=Production). The association periodically enforces that the agent package is installed and running with the designated Parameter Store configuration, eliminating configuration drift across dynamic Auto Scaling groups.

---

Windows Telemetry: Performance Counters and Event Logs

On Windows Server instances, the Unified CloudWatch Agent hooks directly into the Windows Management Instrumentation (WMI) subsystem and the Windows Event Log service.

Windows Performance Counters

In the metrics_collected block under windows_events and Windows performance counter sections, administrators collect standard OS counters:

• Memory: % Committed Bytes In Use, Available MBytes
• LogicalDisk: % Free Space, Free Megabytes for volumes C:, D:, etc.
• Paging File: % Usage
• System: Processor Queue Length

Windows Event Log Ingestion

The logs_collected section captures critical Windows Event channels:

{
  "logs": {
    "logs_collected": {
      "windows_events": {
        "collect_list": [
          {
            "event_name": "System",
            "event_levels": ["ERROR", "CRITICAL"],
            "log_group_name": "/aws/ec2/windows/system",
            "log_stream_name": "{instance_id}"
          },
          {
            "event_name": "Security",
            "event_levels": ["INFORMATION", "WARNING", "ERROR"],
            "log_group_name": "/aws/ec2/windows/security",
            "log_stream_name": "{instance_id}"
          }
        ]
      }
    }
  }
}

---

Troubleshooting Agent Permissions and Network Connectivity

Local Verification Commands

On a Linux host, check agent runtime status using the bundled CLI utility:

# Check agent status
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status

# Inspect local agent log
sudo tail -n 100 /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log

# Manually apply configuration from local file for testing
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
    -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json