2.3 Security Data Storage, Data Streams & ILM

In an enterprise SIEM, security telemetry arrives as a continuous, high-volume, timestamped sequence of events. Because security audit records, network flows, and endpoint process executions represent historical facts, they are fundamentally immutable once recorded. Managing millions of time-series events using traditional, static Elasticsearch indices required manual index rotation, custom alias management, and complex reindexing scripts. Modern Elastic Stack architectures replace this operational complexity with Elastic Data Streams and Index Lifecycle Management (ILM).

---

1. Elastic Data Streams Architecture

A Data Stream is a named, logical abstraction that allows an analyst or ingest client to treat time-series data as a single continuous resource, while Elasticsearch automatically partitions the underlying data across multiple hidden physical indices known as backing indices.

               Logical Data Stream Interface
             (logs-endpoint.events.process-production)
                                |
       +------------------------+------------------------+
       |                        |                        |
       v (Read Queries)         v (Read Queries)         v (Read & Write)
+------------------+   +------------------+   +------------------+
|  Backing Index   |   |  Backing Index   |   |  Backing Index   |
| .ds-...-000001   |   | .ds-...-000002   |   | .ds-...-000003   |
|   (Read-Only)    |   |   (Read-Only)    |   |  (Write Index)   |
+------------------+   +------------------+   +------------------+

Core Properties of Data Streams

1. Append-Only Operations: Data streams are purpose-built for time-series logging. They strictly prohibit direct document updates (_update) and document replacement with static IDs (PUT <stream>/_doc/<id>). Ingested events must be submitted using the create operation (op_type=create), guaranteeing the forensic immutability of security audit logs. Deletions can only be executed by targeting specific backing indices directly or via _delete_by_query.
2. Backing Indices Naming Convention: Underlying physical indices are hidden indices named according to the deterministic pattern: .ds-<data-stream-name>-<yyyy.MM.dd>-<generation> The generation is a 6-digit zero-padded integer (e.g., 000001, 000002) that increments sequentially upon every rollover.
3. Single Active Write Index: Exactly one backing index serves as the active write index (the backing index with the highest generation number). All incoming write requests directed to the data stream name are routed exclusively to this index. Read requests query across all backing indices associated with the stream.
4. Mandatory @timestamp Field: Every document indexed into a data stream must contain an @timestamp field mapped to the date or date_nanos field type.

---

2. Data Stream Naming Taxonomy: <type>-<dataset>-<namespace>

Elastic enforces a standardized, three-part naming convention for data streams across all integrations and security data sources:

$\text{Stream Name} = \text{type} - \text{dataset} - \text{namespace}$

The Three Components

• type: High-level generic data category. In SIEM, this is almost universally logs for events, or metrics for performance telemetry.
• dataset: Identifies the specific source system, application, or event format. Examples include windows.sysmon_operational, suricata.eve, aws.cloudtrail, cisco.asa, and endpoint.events.process.
• namespace: An organization-defined partition representing an environment, business division, cloud tenant, or regulatory classification (such as production, corp, dmz, pci_dss, or dev).

Example Stream Name

logs-endpoint.events.process-production

• type: logs
• dataset: endpoint.events.process
• namespace: production

SecOps Benefits of the Taxonomy

1. Granular Role-Based Access Control (RBAC): SOC administrators can grant tier-1 analysts read privileges to logs-*-production while restricting sensitive audit streams like logs-aws.cloudtrail-compliance to senior forensic investigators.
2. Dataset-Specific Retention: High-volume, short-retention network telemetry (e.g., logs-network.traffic-*) can be assigned an ILM policy that purges data after 30 days, while authentication audit records (e.g., logs-system.auth-*) are retained for 365 days.
3. Modular Template Inheritance: Index templates match wildcards across types, datasets, or namespaces, ensuring uniform mappings across disparate security tools.

---

3. Composable Index Templates and Component Templates

In modern Elasticsearch, indices and data streams are configured using Composable Index Templates (_index_template), which assemble reusable building blocks called Component Templates (_component_template).

+--------------------------------------------------------------+
|               Component Templates (_component_template)      |
|  +-------------------------+    +-------------------------+  |
|  |    Settings Template    |    |    Mappings Template    |  |
|  | (shards, replicas, ILM) |    |   (ECS field mappings)  |  |
|  +-------------------------+    +-------------------------+  |
+------------------------------+-------------------------------+
                               |
                               v (Inherited into)
+--------------------------------------------------------------+
|              Composable Index Template (_index_template)     |
|  - Index Pattern: logs-windows.*-production                  |
|  - Data Stream: {} (Enables Data Stream Creation)            |
|  - Priority: 200                                             |
+------------------------------+-------------------------------+
                               |
                               v (Automatically Configures)
+--------------------------------------------------------------+
|                        Data Stream                           |
|       logs-windows.sysmon_operational-production             |
+--------------------------------------------------------------+

Component Templates

Component templates modularize configuration settings across the cluster:

• Settings Component Templates: Define cluster-level behavior such as number_of_shards: 2, number_of_replicas: 1, refresh_interval: "5s", and the assigned ILM policy name (index.lifecycle.name: "security-logs-ilm-policy").
• Mappings Component Templates: Define the Elastic Common Schema (ECS) field definitions, explicit field types (keyword, ip, date), and dynamic templates for nested fields.

Composable Index Templates

A composable index template binds component templates together:

1. index_patterns: Specifies wildcard patterns matching incoming index names (e.g., ["logs-windows.*-*"]).
2. data_stream: {}: An empty JSON object block. Its presence explicitly instructs Elasticsearch to create a Data Stream rather than a standard index when a matching write request is received.
3. composed_of: An ordered array listing the names of component templates to apply.
4. priority: An integer defining template precedence. If a data stream matches multiple templates (e.g., logs-*-* with priority 100 and logs-windows.*-* with priority 200), the template with the highest priority wins.

---

4. Index Lifecycle Management (ILM) Deep-Dive

Index Lifecycle Management (ILM) automates the progression of backing indices through up to five distinct lifecycle phases to balance high-speed search performance against infrastructure storage expenditure.

Phase 1: Hot Phase and Rollover Triggers

The Hot phase handles active write operations. The active backing index remains in the Hot phase until at least one configured rollover condition is met:

• max_primary_shard_size: The recommended primary rollover trigger for security data. Elastic's default ILM policies for integrations roll over at 50 GB per primary shard (or 30 days), which keeps shards within the recommended 10 GB to 50 GB range.
• max_age: Maximum elapsed time before forcing a rollover (e.g., 30d). Ensures that low-volume datasets roll over regularly and transition through the lifecycle rather than remaining in the Hot tier indefinitely.
• max_docs: Triggers rollover after a fixed number of documents have been indexed.
• max_size: Evaluates total index size across both primary and replica shards.

When rollover executes:

1. A new backing index is created with an incremented generation number (e.g., .ds-...-000002).
2. The new backing index becomes the active write index.
3. The previous backing index becomes read-only and begins its lifecycle progression toward subsequent phases.

Phase 2: Warm Phase and Segment Optimization

In the Warm phase, telemetry is read-only and queried for secondary investigations and baseline calculations. Two critical actions execute here:

• forcemerge (max_num_segments: 1): Merges all smaller Lucene segments into a single segment file. This permanently purges deleted document tombstones, optimizes BKD trees and inverted indices, and substantially reduces JVM heap overhead.
• shrink: Consolidates primary shard count (e.g., shrinking a 4-shard index down to 1 shard), further reducing cluster-wide shard overhead.

Phase 3 & 4: Cold and Frozen Tiers with Searchable Snapshots

Searchable Snapshots eliminate the requirement to retain full primary and replica shard copies on expensive local block storage:

• Fully Mounted Indices (Cold Tier): The index is restored from the snapshot repository onto local data_cold disks as a complete copy. Searches run against local data, and no replica shards are needed because the snapshot itself protects the data.
• Partially Mounted Indices (Frozen Tier): Only a shared local cache is kept on data_frozen nodes. When a search needs data that is not cached, the node fetches it from the object store, so searches are slower but local storage needs are very small.

Phase 5: Delete Phase

Once an index's age exceeds the configured retention window (e.g., min_age: 365d), ILM executes the delete action, permanently removing the backing index and releasing cloud storage allocations.

---

5. The Critical min_age Calculation Rule (Exam Trap)

A frequent source of confusion on the Elastic SIEM exam is how the min_age parameter is calculated.

Step-by-Step Scenario Analysis

1. Day 0 (October 1st): A new backing index .ds-logs-network-default-2026.10.01-000001 is created in the Hot phase.
2. Day 15 (October 15th): The index reaches 50 GB and satisfies max_primary_shard_size: 50GB. The index rolls over on October 15th.
3. Warm Phase Configuration: The assigned ILM policy defines the Warm phase with min_age: 10d.
4. Transition Date: The backing index does not enter the Warm phase on October 10th (10 days from creation). It enters the Warm phase on October 25th (October 15th rollover time + 10 days min_age).

If an index does not roll over (e.g., a static standalone index), min_age is calculated from the index creation time. But for all Data Streams governed by rollover, rollover time is the reference timestamp.