2.2 Amazon Security Lake & OCSF Log Normalization

The Enterprise Telemetry Fragmentation Problem

In large-scale enterprise environments spanning dozens of AWS accounts, multiple regions, and hybrid on-premises data centers, security teams confront severe telemetry fragmentation. Each logging system produces records in disparate, proprietary formats:

• CloudTrail emits nested JSON structures.
• VPC Flow Logs emit space-delimited text strings.
• Route 53 Resolver logs emit distinct JSON schemas.
• GuardDuty findings use the Amazon GuardDuty finding schema.
• Security Hub findings use the AWS Security Finding Format (ASFF).
• On-premises firewalls and identity providers generate syslog, CEF, or custom key-value pairs.

Historically, Security Operations Center (SOC) teams had to build, maintain, and scale complex Extract, Transform, Load (ETL) pipelines using AWS Lambda, Amazon Kinesis, and AWS Glue to parse, re-index, and normalize these streams before ingesting them into a Security Information and Event Management (SIEM) tool. These pipelines introduce significant compute costs, processing latency, and operational fragility.

Amazon Security Lake solves this challenge by automatically centralizing security data from multiple AWS accounts and regions into a purpose-built security data lake, normalizing all telemetry into the open-source Open Cybersecurity Schema Framework (OCSF).

---

The Open Cybersecurity Schema Framework (OCSF)

OCSF is an open-source, vendor-agnostic standard initiated by AWS and leading cybersecurity vendors. It defines a standardized, hierarchical taxonomy for security events, eliminating the need for custom schema parsing.

[OCSF Root]
   └── Categories (e.g., Network Activity, System Activity, Finding)
         └── Event Classes (e.g., DNS Activity, Network Activity, Security Finding)
               └── Standardized Attributes (e.g., src_endpoint, dst_endpoint, actor, severity_id)

Core OCSF Concepts

• Categories: High-level groupings of event domains (e.g., Category 1: System Activity, Category 2: Findings, Category 3: Identity & Access Management, Category 4: Network Activity).
• Event Classes: Specific event implementations within a category. For example, Event Class 4001: Network Activity represents connection flows, while Event Class 4003: DNS Activity represents DNS queries and resolutions.
• Standardized Data Types and Dictionaries: OCSF standardizes IP addresses (ip), port numbers (port), MAC addresses (mac), hostnames (hostname), and status codes. For example, whether a connection was allowed or denied is mapped to standard disposition_id integers (e.g., 1: Allowed, 2: Blocked).
• Unified Severity: Threat severity across different security products is normalized to a severity_id integer (1 Informational, 2 Low, 3 Medium, 4 High, 5 Critical, 6 Fatal, plus 0 Unknown and 99 Other), enabling consistent correlation across GuardDuty, third-party endpoint detection and response (EDR) agents, and network firewalls.

---

Amazon Security Lake Core Architecture

Security Lake is deployed across an entire AWS Organization via a designated Delegated Administrator Account (typically the enterprise Security Tooling or Central Security account).

[ AWS Accounts & Regions ]
   ├── CloudTrail Management & Data Events
   ├── VPC Flow Logs
   ├── Route 53 Resolver Query Logs   === (Native Extraction) ===> [ Amazon Security Lake ]
   ├── Amazon GuardDuty Findings                                       │
   └── AWS Security Hub Findings                                       ▼
                                                             [ Normalization Engine ]
                                                                       │
                                                                       ▼
                                                             [ Apache Parquet in S3 ]
                                                             (Partitioned by Source,
                                                              Region, Account, Date)

Automatically Collected Native Sources

When enabled, Security Lake collects its natively supported AWS sources without agents or custom Lambda triggers:

1. AWS CloudTrail Management Events
2. AWS CloudTrail Data Events (for Amazon S3 and AWS Lambda)
3. Amazon VPC Flow Logs
4. Amazon Route 53 Resolver Query Logs
5. AWS Security Hub CSPM Findings (which carry GuardDuty, Inspector, Macie, and IAM Access Analyzer findings, converted from ASFF to OCSF)
6. Amazon EKS Audit Logs
7. AWS WAFv2 Logs (web ACL traffic logs, which is how edge telemetry reaches an OCSF data lake)

Storage Mechanics: Apache Parquet & Partitioning

Security Lake stores all ingested and normalized data in Amazon S3 using Apache Parquet format. Parquet is an open-source, columnar storage format optimized for analytical queries:

• Columnar Layout: Queries that examine specific columns (such as src_endpoint.ip or actor.user.name) only read the relevant columns from disk, avoiding costly full-row table scans.
• Compression: Parquet is a compressed columnar format, so the stored volume is far smaller than the equivalent raw JSON.
• Deterministic Partitioning: Data is partitioned by source, Region, account ID, and event day, so object prefixes end in a pattern such as region=us-east-1/accountId=111122223333/eventDay=20260929/ (custom sources are written under an ext/ prefix). Partition pruning allows analytical query engines to scan only the exact temporal and structural slices required.

---

Multi-Region Roll-Up Architecture

Global enterprises operating across multiple AWS regions face architectural trade-offs between centralized visibility and cross-region data transfer costs. Security Lake resolves this via Regional Roll-ups.

Roll-Up Mechanics

• Contributing Regions: Regions where workloads run and telemetry originates (e.g., eu-west-1, ap-southeast-1, us-west-2).
• Roll-up Regions: Designated target regions (e.g., us-east-1) that automatically receive replicated security data from contributing regions.

Security Lake uses Amazon S3 cross-region replication under the hood to consolidate telemetry into the roll-up region. This enables security analysts and SIEM systems to execute queries against a single regional endpoint while complying with corporate retention and cross-region egress budgets.

Exam Tip: Data residency compliance laws (such as GDPR in Europe) may prohibit transferring security logs outside their originating jurisdiction. In such scenarios, do NOT configure roll-up replication for regulated regions; leave them as standalone regional data stores and query them locally.

---

Querying Security Lake via AWS Lake Formation & Amazon Athena

Security Lake registers all normalized Parquet tables in the AWS Glue Data Catalog and enforces data access controls through AWS Lake Formation.

Security analysts can immediately run standard SQL queries in Amazon Athena against OCSF tables without configuring table DDLs or repair scripts:

-- Investigate high-volume blocked outbound traffic across all VPCs
SELECT 
    time_dt,
    src_endpoint.ip AS source_ip,
    src_endpoint.port AS source_port,
    dst_endpoint.ip AS destination_ip,
    dst_endpoint.port AS destination_port,
    traffic.bytes AS bytes_transferred,
    disposition
FROM 
    amazon_security_lake_glue_db_us_east_1.amazon_security_lake_table_us_east_1_vpc_flow_2_0
WHERE 
    disposition = 'Blocked'
    AND traffic.bytes > 10000000
    AND eventDay = '20260929'
ORDER BY 
    traffic.bytes DESC
LIMIT 100;

Because the underlying data is Parquet, Athena scans only the requested columns and matching date partitions, dramatically accelerating query execution while minimizing query costs (charged at $5.00 per TB scanned).

---

Subscriber Integration Models: Query Access vs. Data Access

Security Lake decouples log collection from security consumption. Downstream consumers (e.g., Splunk, Datadog, Snowflake, OpenSearch, internal SOC tools) register as Subscribers.

Security Lake supports two subscriber consumption models:

[ Security Lake S3 Bucket ]
        │
        ├── (Direct Query via Athena) ─────────> [ Query Subscriber (Snowflake / BI) ]
        │                                          (Lake Formation Permissions)
        │
        ├── New Parquet Object Created
        │        │
        │        ▼
        ├──> [ Amazon SQS Queue ] ─────────────> [ Data Subscriber (Splunk / Datadog) ]
        │        │                                 (Polls SQS, assumes cross-account
        └────────┴── Fetches Parquet File ──────── IAM role with ExternalId) 

---

Ingesting Custom & Third-Party Log Sources

Enterprise security requires holistic visibility across cloud and on-premises infrastructure. Security Lake supports ingesting Custom Sources (such as Palo Alto firewall logs, Okta authentication logs, or bespoke container application logs):

1. Define Source: The administrator registers a custom source in Security Lake, specifying the source name and OCSF event class.
2. S3 Ingestion Location: Security Lake provisions a dedicated S3 bucket prefix and an IAM execution role.
3. Transformation: The customer's ingestion pipeline (e.g., AWS Lambda, AWS Glue ETL, or Amazon Data Firehose) transforms raw log events into OCSF-compliant Apache Parquet format and writes the objects to the custom source prefix.
4. Cataloging: Security Lake invokes an AWS Glue Crawler to update partition definitions in the Glue Data Catalog, making custom data instantly queryable alongside native AWS telemetry.