2.4 Troubleshooting Security Telemetry & Log Ingestion

Systematic Troubleshooting Methodology for Security Telemetry

Security telemetry pipelines fail for predictable architectural reasons: IAM authorization failures, service principal mismatches in resource policies, network line-of-sight blockages, and incompatible cryptographic configurations. For the AWS Certified Security – Specialty examination, you must possess the ability to rapidly isolate the root cause when logs fail to deliver.

[ Log Source ] ──> (1. IAM / Trust) ──> (2. Network / Endpoints) ──> (3. Service Principal) ──> (4. KMS Encryption) ──> [ Destination ]

When a telemetry stream ceases or fails to initialize, evaluate these four layers systematically:

1. Identity & Execution Role: Does the entity producing the logs have permissions to call PutLogEvents or assume the logging role?
2. Network Path: Can the client establish TLS sessions to logs.<region>.amazonaws.com through NAT or VPC Endpoints?
3. Resource / Bucket Policy: Does the receiving S3 bucket policy trust the exact AWS service principal and satisfy all condition keys?
4. Cryptographic Key Policy: Does the destination KMS key policy grant kms:GenerateDataKey* and kms:DescribeKey to the delivering service with the proper encryption context?

---

Diagnosing CloudWatch Unified Agent Failures

The Amazon CloudWatch Unified Agent is installed on EC2 instances and on-premises servers to collect system metrics and log files (e.g., /var/log/secure, /var/log/audit/audit.log, Windows Event Logs). When log streams fail to appear in CloudWatch Logs, investigate the following failure modes:

1. Missing or Inadequate IAM Permissions

The EC2 instance profile must have an attached IAM policy granting CloudWatch Logs write operations. Attach the AWS-managed policy CloudWatchAgentServerPolicy or a scoped custom policy containing:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogStreams"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    }
  ]
}

2. Host-Level File Permissions

The Unified Agent typically runs as a dedicated non-root service account (cwagent). If target operating system log files (such as /var/log/audit/audit.log) have file permissions set to 0600 owned strictly by root:root, the cwagent process cannot read the file. The local agent log (/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log) will report open /var/log/audit/audit.log: permission denied.

3. Network Line-of-Sight & VPC Endpoints

EC2 instances residing in private subnets without an internet gateway or NAT gateway cannot reach the public CloudWatch Logs regional API. You must provision an Interface VPC Endpoint (com.amazonaws.<region>.logs). Common pitfalls include:

• Security Group on the VPC Endpoint: The security group attached to the Interface Endpoint must permit inbound TCP port 443 from the private subnet CIDR.
• Private DNS: Private DNS must be enabled on the VPC Endpoint so that SDK and agent calls to logs.<region>.amazonaws.com resolve automatically to the endpoint's private IP addresses.

---

API Gateway Execution & Access Logging Failures

Amazon API Gateway supports two types of logging:

• Access Logging: Logs client caller identity, request path, IP address, and response latency.
• Execution Logging: Logs detailed internal execution steps, authorizer execution, request/response transformation, and integration backend errors.

The Global Account Settings Trap

A frequent point of failure is that enabling execution or access logging on an individual API stage appears to succeed in the console, but no log groups or log streams are ever created in CloudWatch Logs.

Root Cause: API Gateway requires an IAM role configured globally at the API Gateway Account Settings level for that region. API Gateway cannot assume a role specified per-stage; it relies on a regional account-level role.

[ API Gateway Console / API ]
   └── Settings
         └── CloudWatch log role ARN: arn:aws:iam::111122223333:role/APIGatewayLoggingRole

The IAM role must have a trust relationship allowing apigateway.amazonaws.com:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "apigateway.amazonaws.com" },
      "Action": "sts:AssumeRole"
    }
  ]
}

And it must have the managed policy AmazonAPIGatewayPushToCloudWatchLogs attached, granting logs:CreateLogGroup, logs:CreateLogStream, logs:PutLogEvents, and logs:DescribeLogStreams.

Full Request/Response Data Tracing Security Risk

When enabling Execution Logging, administrators can enable Log full requests/responses data. While invaluable for debugging, this logs entire request bodies, query strings, and headers into plaintext CloudWatch Logs. If requests contain passwords, API keys, or credit card numbers, this creates an unmasked compliance violation.

---

AWS Lambda Execution Role Permissions

When an AWS Lambda function executes, it streams execution outputs (stdout, stderr, and runtime errors) to a default log group named /aws/lambda/<function-name>.

Required IAM Actions

The Lambda function's execution role must contain:

• logs:CreateLogGroup
• logs:CreateLogStream
• logs:PutLogEvents These are included in the AWS-managed policy AWSLambdaBasicExecutionRole.

Encrypted Log Groups with Customer Managed Keys (CMK)

If the CloudWatch Log Group is encrypted with an AWS KMS Customer Managed Key, who needs KMS permissions? This is a classic exam question.

• The Lambda execution role does NOT require KMS permissions to write logs. The Lambda runtime passes log data to the CloudWatch Logs service.
• The CloudWatch Logs Service Principal (logs.<region>.amazonaws.com) MUST be granted KMS permissions in the KMS Key Policy:

{
  "Sid": "AllowCloudWatchLogsKeyAccess",
  "Effect": "Allow",
  "Principal": { "Service": "logs.us-east-1.amazonaws.com" },
  "Action": [
    "kms:Encrypt*",
    "kms:Decrypt*",
    "kms:ReEncrypt*",
    "kms:GenerateDataKey*",
    "kms:DescribeKey"
  ],
  "Resource": "*",
  "Condition": {
    "ArnLike": {
      "kms:EncryptionContext:aws:logs:arn": "arn:aws:logs:us-east-1:111122223333:log-group:*"
    }
  }
}

---

S3 Bucket Policy & KMS Key Policies: CloudTrail vs. VPC Flow Logs

When log services deliver directly to Amazon S3, you must use the correct service principal. Using the wrong service principal in the bucket policy causes delivery to fail with Access Denied.

Exam Trap: A common distractor on the exam proposes configuring vpc-flow-logs.amazonaws.com in an S3 bucket policy. This will fail! When VPC Flow Logs writes to Amazon S3, AWS uses the Log Delivery service, whose service principal is delivery.logs.amazonaws.com.

Confused Deputy Prevention with aws:SourceArn

To prevent an adversary from abusing an AWS service to write unauthorized data to your audit bucket, always include aws:SourceArn or aws:SourceAccount in your bucket policy condition blocks.

---

CloudFront Logging Debugging: Standard vs. Real-Time Logs

Amazon CloudFront provides two distinct logging mechanisms with fundamentally different operational constraints.

Standard Access Logs (S3 Delivery)

Standard logs are delivered as compressed gzip files within minutes to an hour or so. The original S3 delivery is now labelled standard logging (legacy); the newer standard logging (v2) can also send the same records to CloudWatch Logs or Amazon Data Firehose.

The two bucket traps in legacy S3 delivery:

• ACLs must be enabled. CloudFront grants the awslogsdelivery account FULL_CONTROL through the bucket ACL. New buckets default to S3 Object Ownership Bucket owner enforced, which disables ACLs, so logs never arrive until ACLs are enabled on that bucket.
• KMS key choice. SSE-S3 always works. SSE-KMS works only with a customer managed key whose key policy allows the delivery.logs.amazonaws.com service principal to call kms:GenerateDataKey* (plus kms:Decrypt if S3 Bucket Keys are enabled). The AWS managed key aws/s3 cannot be used, because its policy can't be edited.

Real-Time Logs (Kinesis Data Streams Delivery)

Real-time logs deliver configurable access data within seconds to Amazon Kinesis Data Streams.

• Requires an IAM role that CloudFront assumes to write records.
• The IAM role trust policy must grant sts:AssumeRole to cloudfront.amazonaws.com.
• The IAM permissions policy must grant kinesis:PutRecord and kinesis:PutRecords on the stream ARN.
• If the Kinesis data stream uses SSE-KMS, the IAM role that CloudFront assumes must also be allowed to use that key (for example, kms:GenerateDataKey).