2.5 Analyzing, Normalizing & Correlating Security Logs

2.5 Analyzing, Normalizing & Correlating Security Logs

Collecting logs is only half of detection. SCS-C03 Skills 1.2.4 and 1.2.5 test whether you can query logs to answer an investigation question and normalize, parse, and correlate logs from different sources. The exam names CloudWatch Logs Insights, Amazon Athena, Security Hub findings, Amazon OpenSearch Service, AWS Lambda, and Amazon Managed Grafana. The skill is choosing the right tool for where the data lives, how fast you need answers, and who will look at them.

---

Choosing an Analysis Tool

---

CloudWatch Logs Insights

Logs Insights runs interactive queries across one or more log groups, which makes it the fastest option when CloudTrail, VPC Flow Logs, Route 53 Resolver query logs, Lambda logs, or application logs already stream to CloudWatch Logs. Queries can be written in the Logs Insights query language, OpenSearch Piped Processing Language (PPL), or SQL. You pay for the data scanned, so narrow the time range and log groups first.

A typical investigation query finds which principals are generating access denials in a CloudTrail log group:

fields @timestamp, eventSource, eventName, userIdentity.arn, sourceIPAddress
| filter errorCode = "AccessDenied" or errorCode = "UnauthorizedOperation"
| stats count(*) as denials by userIdentity.arn, eventName
| sort denials desc
| limit 20

Useful features include saved queries, adding query results to CloudWatch dashboards, and the pattern command, which clusters similar log lines so unusual messages stand out. Logs Insights only sees data in CloudWatch Logs, so logs that go straight to S3 need Athena instead.

---

Amazon Athena for Logs in S3

Organization trails, VPC Flow Logs, ALB and CloudFront access logs, and AWS WAF logs commonly land in a central log archive bucket. Athena queries them in place with SQL. Two practices keep investigations fast and affordable:

• Partition the tables (by account, Region, and date) and use partition projection, so a query for one day in one account doesn't scan years of logs.
• Use workgroups to enforce an encrypted query-results location, per-query data-scan limits, and separate access for the security team.

SELECT eventtime, useridentity.arn, eventname, sourceipaddress
FROM cloudtrail_logs
WHERE eventsource = 'iam.amazonaws.com'
  AND eventname IN ('CreateAccessKey', 'AttachUserPolicy', 'PutUserPolicy')
  AND timestamp BETWEEN '2026/09/01' AND '2026/09/07'
ORDER BY eventtime;

Here timestamp is the partition column that CloudTrail's generated table uses. Amazon Security Lake stores normalized OCSF data in S3 and registers it in the AWS Glue Data Catalog, so a subscriber with query access can run the same kind of SQL across CloudTrail, VPC Flow Logs, Route 53, and Security Hub data at once.

---

Security Hub CSPM Findings as an Analysis Source

Security Hub CSPM collects findings from GuardDuty, Inspector, Macie, IAM Access Analyzer, AWS Config, and partner products. Insights group related findings by an attribute, such as resource ID or AWS account, so you can see, for example, which EC2 instances have the most critical findings. With cross-Region aggregation and a delegated administrator, one account can analyze findings for the whole organization. Findings also flow to Amazon EventBridge for automated response and can be sent to Security Lake for long-term SQL analysis.

---

Normalizing, Parsing & Correlating

Logs from different sources use different field names, time formats, and identifiers. Correlation needs a common shape.

• OCSF normalization: Security Lake converts supported AWS sources to the Open Cybersecurity Schema Framework automatically. Custom sources must be written as OCSF records in Parquet format.
• Lambda transformation: A CloudWatch Logs subscription filter can send events to Lambda, or an Amazon Data Firehose stream can invoke Lambda to transform records before delivery. Typical jobs include parsing an application's text log into JSON, converting timestamps to UTC, adding account names or asset owners, and mapping fields to OCSF. Send failed records to a dead-letter destination so parsing errors don't silently drop evidence.
• Amazon OpenSearch Service: OpenSearch Ingestion pipelines load logs from S3, Security Lake, or HTTP sources. The Security Analytics feature applies detectors built from Sigma rules to log types such as CloudTrail, VPC Flow Logs, and DNS, and correlation rules link findings across log types (for example, a suspicious DNS query followed by an outbound connection from the same host). Direct query (zero-ETL) integrations can also query data in S3, CloudWatch Logs, and Security Lake without copying it into an index. Protect the domain with VPC access, fine-grained access control, and encryption.
• Amazon Managed Grafana: A managed Grafana workspace builds shared dashboards over CloudWatch, OpenSearch, Athena, and other data sources. Users sign in through IAM Identity Center or SAML, and the workspace uses an IAM role to read each data source, so dashboards can span accounts without handing out console access.

---

Specialty Exam Pitfalls

1. Choosing Logs Insights for data in S3: Logs Insights queries CloudWatch Logs only. For an organization trail delivered only to S3, use Athena or Security Lake.
2. Unpartitioned Athena tables: Full-table scans over years of CloudTrail logs are slow and expensive; partition projection limits the scan.
3. Correlating without normalization: Joining events on inconsistent field names or time zones produces false conclusions. Normalize to OCSF or a common schema first.
4. Treating dashboards as detection: Grafana and OpenSearch dashboards show data; alerts, detectors, or EventBridge rules are what notify responders.