9.3 Centralized Log Aggregation, Retention & Cross-Account Subscriptions

CloudWatch Logs Architecture: Groups, Streams, and Retention Lifecycles

Amazon CloudWatch Logs is a highly available, scalable service for centralizing operating system, application, and AWS service logs (such as Lambda execution logs, VPC Flow Logs, and API Gateway execution logs).

Log Organizational Hierarchy

• Log Event: A record containing an exact timestamp (milliseconds since epoch) and a raw UTF-8 string message payload (maximum size: 1,024 KB / 1 MB per event; a single PutLogEvents batch is also capped at 1 MB).
• Log Stream: A sequence of log events emitted by the same source instance, container task, or Lambda execution environment. Log streams represent an ephemeral unit of concurrency.
• Log Group: A logical administrative collection of log streams that share the same access control policies, monitoring configurations, metric filters, subscription filters, and retention rules.

Ingestion Quotas and Retention Policies

By default, CloudWatch Logs retains log events indefinitely (Never Expire). In enterprise organizations, this creates mounting storage costs for ephemeral diagnostic logs that lose business value after days or weeks.

• Retention Windows: The only accepted values are 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1096, 1827, 2192, 2557, 2922, 3288, and 3653 days, or Never Expire (achieved by deleting the retention policy).
• Cost-Optimization Architecture: Set log group retention to a short operational window (e.g., 14 to 30 days) for active CloudWatch Logs Insights querying and metric alarms. For long-term audit compliance (e.g., PCI-DSS or HIPAA requiring 7-year retention), export logs to Amazon S3 and transition them to S3 Glacier Flexible Retrieval or Glacier Deep Archive via S3 Lifecycle rules.

---

Metric Filters: Real-Time Log Parsing and Numerical Extraction

Metric filters monitor log events as they are ingested into CloudWatch Logs in near-real time. They evaluate incoming log records against a filter pattern and publish custom metric data points without persisting or modifying the underlying log events.

Filter Pattern Syntax: JSON vs. Space-Delimited

CloudWatch Logs supports two distinct filter syntaxes:

1. JSON Metric Filter Patterns: Used when applications emit structured JSON logs. Supports boolean logic, numerical comparisons, and nested attribute inspection:

{ ($.statusCode >= 400 && $.statusCode < 500) || $.errorType = "ValidationException" }

1. Space-Delimited Metric Filter Patterns: Used for legacy unstructured or space-separated formats (such as Apache, NGINX, or custom text logs):

[ip, user, username, timestamp, request, statusCode = 5*, bytes, responseTime > 1000]

Metric Transformation and the DefaultValue Rule

When a metric filter matches a log event, CloudWatch publishes a metric data point according to its Metric Transformation:

• Metric Namespace: The target custom namespace (e.g., CustomApp/WebErrors).
• Metric Name: The name of the metric (e.g., 4xxErrorCount, ExecutionLatency).
• Metric Value: The numerical value to publish. Can be a static integer (e.g., 1 for counting events) or a dynamic extracted token (e.g., $.durationMs or $responseTime).
• Default Value: Specifies the value published when a reporting period contains no matching log events.

[!IMPORTANT] DOP-C02 Exam Trap: If you create a CloudWatch Alarm on a metric filter that counts errors (e.g., 5xxErrorCount) and leave DefaultValue blank (null), the metric will publish no data points during periods where zero errors occur. The alarm will transition to INSUFFICIENT_DATA or fail to clear. Setting DefaultValue: 0 ensures that CloudWatch publishes 0 during error-free periods, allowing the alarm to evaluate properly and transition back to OK.

---

Subscription Filters: Real-Time Event Streaming Destinations

While metric filters extract scalar numbers, Subscription Filters extract the complete, raw log event payload and stream it in near-real time to external processing systems.

Subscription Filter Limits and Destinations

Each CloudWatch Log Group supports up to five subscription filters. Subscription filters push log data to three supported AWS targets:

Note: Do not confuse Subscription Filters with CreateExportTask. CreateExportTask is an asynchronous batch API that dumps logs from CloudWatch Logs to Amazon S3. It cannot stream in real-time, can take hours to complete, and is limited to one active export task per account per region.

---

Enterprise Cross-Account Log Centralization Architecture

Large enterprises operate multi-account AWS Organizations with strict governance boundaries. Standard security architecture mandates consolidating all application, security, and audit logs from hundreds of Member Accounts into a single, hardened Central Logging Account.

[ Member Account (222222222222) ]
  CloudWatch Log Group
      │
      ▼ (Subscription Filter)
[ Central Logging Account (111111111111) ]
  CloudWatch Logs Destination (CentralLogDestination)
      │  (Access Policy: Allows Member Account 222222222222)
      ▼
  IAM Role (Assumed by CloudWatch Logs Service)
      │  (Policy: kinesis:PutRecord* on Kinesis Stream)
      ▼
  Amazon Kinesis Data Stream
      │
      ▼
  Amazon Kinesis Data Firehose ──> Amazon S3 / OpenSearch

Step 1: Central Logging Account Configuration

1. Kinesis Stream: Provision an Amazon Kinesis Data Stream (e.g., CentralSecurityLogStream) in the central account (111111111111).
2. IAM Execution Role: Create an IAM role (CWLogsToKinesisRole) that the CloudWatch Logs service assumes to write records into the stream:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "logs.us-east-1.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Permissions Policy attached to the role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["kinesis:PutRecord", "kinesis:PutRecords"],
      "Resource": "arn:aws:kinesis:us-east-1:111111111111:stream/CentralSecurityLogStream"
    }
  ]
}

1. CloudWatch Logs Destination: Create the destination resource using PutDestination:

aws logs put-destination \
    --destination-name "CentralLogDestination" \
    --target-arn "arn:aws:kinesis:us-east-1:111111111111:stream/CentralSecurityLogStream" \
    --role-arn "arn:aws:iam::111111111111:role/CWLogsToKinesisRole"

1. Destination Access Policy: Attach a resource policy (PutDestinationPolicy) granting member accounts permission to subscribe to the destination. You can grant access to individual accounts or an entire AWS Organization:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowOrganizationMemberAccounts",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "logs:PutSubscriptionFilter",
      "Resource": "arn:aws:logs:us-east-1:111111111111:destination:CentralLogDestination",
      "Condition": {
        "StringEquals": {
          "aws:PrincipalOrgID": "o-abcdef1234"
        }
      }
    }
  ]
}

Step 2: Member Account Configuration

In each member account (222222222222), configure a Subscription Filter on the local Log Group pointing to the central destination ARN:

aws logs put-subscription-filter \
    --log-group-name "/aws/ec2/production/application" \
    --filter-name "StreamToCentralSecurity" \
    --filter-pattern "" \
    --destination-arn "arn:aws:logs:us-east-1:111111111111:destination:CentralLogDestination"

Note: Setting --filter-pattern "" captures all log events. Passing a specific pattern filters events before cross-account transmission.

---

KMS CMK Encryption at Rest for CloudWatch Logs

By default, CloudWatch Logs encrypts stored log data using AWS-managed keys. For regulatory compliance, organizations must enforce customer control using AWS KMS Customer Managed Keys (CMKs).

Associating a KMS Key with a Log Group

Associate a KMS CMK via CLI or CloudFormation:

aws logs associate-kms-key \
    --log-group-name "/aws/ec2/production/application" \
    --kms-key-id "arn:aws:kms:us-east-1:111111111111:key/12345678-1234-1234-1234-123456789012"

KMS Key Policy Delegation Requirements

[!CAUTION] Critical Key Policy Rule: The KMS CMK key policy must explicitly grant the regional CloudWatch Logs service principal permissions to perform cryptographic operations. CloudWatch Logs uses an Encryption Context that contains the ARN of the log group.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowCloudWatchLogsEncryption",
      "Effect": "Allow",
      "Principal": {
        "Service": "logs.us-east-1.amazonaws.com"
      },
      "Action": [
        "kms:Encrypt*",
        "kms:Decrypt*",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:DescribeKey"
      ],
      "Resource": "*",
      "Condition": {
        "ArnEquals": {
          "kms:EncryptionContext:aws:logs:arn": "arn:aws:logs:us-east-1:111111111111:log-group:/aws/ec2/production/application"
        }
      }
    }
  ]
}

---

DOP-C02 Troubleshooting & Operational Watchouts

Cross-Account Subscription Debugging Sequence

When cross-account subscriptions fail, verify in this strict sequence:

1. Region Symmetry: The member account Log Group, the central CloudWatch Logs Destination, and the Kinesis Data Stream must all reside in the exact same AWS Region (e.g., us-east-1). CloudWatch Logs Destinations do not support cross-region subscription delivery.
2. Destination Trust Policy: The IAM role referenced in PutDestination must have a trust policy allowing logs.<region>.amazonaws.com (or logs.amazonaws.com).
3. Destination Resource Policy: The PutDestinationPolicy must explicitly allow the member account's AWS account ID (or Organization ID) to call logs:PutSubscriptionFilter against the destination resource.