5.1 SIEM Architecture, Components, and Deployment Models

Evolutionary Heritage: Convergence of SIM and SEM

Modern enterprise security operations rely on Security Information and Event Management (SIEM) as the central nervous system for threat detection, investigation, and compliance orchestration. Understanding contemporary SIEM architecture requires examining its historical evolution from two previously distinct technological paradigms that emerged in the late 1990s and early 2000s:

1. Security Information Management (SIM): Engineered primarily for historical log archival, regulatory compliance reporting (such as Sarbanes-Oxley, HIPAA, and early PCI DSS mandates), and batch-oriented analysis. SIM platforms ingested raw log files across servers, databases, and network appliances, loading them into relational databases or flat-file archives. Queries were retrospective, report-oriented, and slow, taking minutes or hours to process massive historical datasets.
2. Security Event Management (SEM): Engineered for real-time streaming analysis, in-memory event correlation, and immediate operational alerting. SEM systems focused on perimeter defenses—such as firewalls, routers, and Network Intrusion Detection Systems (NIDS)—processing event streams through rule-based correlation engines to alert SOC operators to active attacks. However, SEM solutions lacked long-term indexing capacity and forensic depth.

In 2005, industry analysts Mark Nicolett and Amrit Williams coined the term SIEM, defining the convergence of SIM's comprehensive data retention and forensic reporting with SEM's low-latency stream processing and real-time correlation. Today, SIEM platforms ingest terabytes of diverse telemetry daily, serving as the primary workbench for SOC Tier 1 triage, Tier 2 deep-dive investigation, and Tier 3 threat hunting.

---

Modern SIEM Functional Architecture & Pipeline Components

A production-grade SIEM platform operates as an end-to-end data pipeline structured into six discrete, interdependent functional layers:

[Raw Telemetry Sources]
  -> Ingestion & Collection (Syslog, Agents, APIs, NetFlow)
    -> Pre-Processing & Normalization (Parsing, Field Extraction, CEF/LEEF/ECS, UTC Sync)
      -> Correlation & Analytics Engine (Rule-based, Stateful Windows, Statistical, UEBA)
        -> Tiered Data Storage (Hot NVMe, Warm SAS, Cold Archive, Object Data Lakes)
          -> Presentation & Dashboards (Analyst Workbench, Metric Visualizations)
            -> Automation & Ticketing (SOAR Playbooks, Bi-directional ITSM APIs)

1. Data Ingestion and Collection Layer

The ingestion layer acquires telemetry across heterogeneous enterprise assets using diverse collection mechanisms:

• Agent-Based Collection: Lightweight software agents (e.g., Splunk Universal Forwarder, Elastic Agent, NXLog, Winlogbeat) installed directly on operating systems. Depending on the product and configuration, agents can tail files, subscribe to Windows events, buffer locally during outages, filter at the edge, and forward over an authenticated encrypted transport.
• Agentless Collection: Network-based transport mechanisms for appliances that do not support endpoint agents. Common mechanisms include legacy BSD-style syslog commonly sent over UDP 514, RFC 5424-formatted messages carried over configured transports such as RFC 5425 TLS (commonly TCP 6514), SNMP traps, and authenticated WMI/WinRM collection for Windows environments.
• API-Based Cloud Ingestion: Polling serverless REST APIs or subscribing to cloud message queues (such as AWS SQS/CloudWatch, Azure Event Hubs, and Google Cloud Pub/Sub) to retrieve SaaS audit logs, identity telemetry (Microsoft Entra ID, Okta), and cloud infrastructure activity.
• Network Telemetry: Ingesting unsampled or sampled flow records (NetFlow v5/v9, IPFIX, sFlow) to maintain Layer 3 and Layer 4 visibility across core switching fabrics and software-defined perimeters.

2. Pre-Processing, Parsing, and Normalization Engine

Raw log entries arrive as unformatted text strings. The pre-processing engine executes three critical tasks:

• De-duplication and Filtering: Dropping repetitive, low-value telemetry (such as benign health checks or interface flapping) at the collector edge to conserve indexing licenses and storage.
• Parsing and Field Extraction: Utilizing regular expressions (regex), delimiter-based parsing, or grok patterns to extract unstructured data into distinct key-value pairs (e.g., extracting SrcIP=192.168.10.45, DestPort=443, User=svc_backup).
• Taxonomy Normalization: Mapping vendor-proprietary field names into standardized schema frameworks, such as Common Event Format (CEF), Log Extended Event Format (LEEF), Elastic Common Schema (ECS), or the Open Cybersecurity Schema Framework (OCSF). This enables a single correlation rule to evaluate authentication failures across Windows, Linux, Cisco firewalls, and cloud identity providers simultaneously.
• Timestamp and UTC Synchronization: Normalizing divergent local timestamps into standardized ISO 8601 UTC formats with millisecond or microsecond precision, preventing sequence errors caused by daylight saving shifts or multi-region timezones.

3. Correlation and Analytics Engine

The correlation engine continuously evaluates normalized event streams to detect multi-stage attack patterns:

• Rule-Based Deterministic Logic: Boolean logic combining event attributes across sliding time windows (e.g., matching a port scan event followed within 120 seconds by an exploit signature and an anomalous service installation).
• Statistical and Threshold Correlation: Triggering when an event frequency exceeds a predefined static threshold or a dynamic standard deviation baseline (e.g., more than 5 failed logins within 60 seconds).
• User and Entity Behavior Analytics (UEBA): Machine learning models that build baseline profiles for users, peer groups, and service accounts, flagging deviations such as impossible travel velocity, anomalous data access volumes, or off-hours administrative activity.

4. Tiered Data Storage Subsystem

To optimize search performance against hardware and cloud costs, enterprise SIEM architectures may use tiered storage. The ranges below are illustrative design choices, not universal definitions:

• Hot Storage Tier: Deployed on high-performance NVMe solid-state drives. Contains newly ingested and fully indexed data from the past 7 to 30 days. Optimized for sub-second query latency during real-time SOC investigations.
• Warm Storage Tier: Deployed on cost-effective SAS SSDs or clustered storage nodes. Retains indexed data from 31 to 90 days. Query speeds are moderately slower but fully accessible without manual administrator intervention.
• Cold Storage Tier: Deployed on spinning hard disks or warm cloud object storage (e.g., AWS S3 Standard-IA). Retains compressed data from 91 to 365 days. Index structures may be detached; searching requires targeted time-slice scoping.
• Frozen / Archive Tier: Uses lower-cost object or archive storage for data outside the interactive window. Retention follows the organization's legal, contractual, security, and evidentiary requirements; retrieval method and delay depend on the platform. HIPAA does not impose one universal technical-log retention period.

5. Presentation, Dashboards, and SOC Analyst Console

The graphical user interface (GUI) provides real-time situational awareness. Key capabilities include high-priority alert queues, investigation workbenches, interactive relationship graphs, geographic mapping of external IP connections, and executive KPI dashboards tracking metrics such as Mean Time to Detect (MTTD) and Mean Time to Remediate (MTTR).

6. Case Management and SOAR Integration

Modern SIEMs interface directly with IT Service Management (ITSM) platforms (such as ServiceNow or Jira) and Security Orchestration, Automation, and Response (SOAR) tools. Upon rule detonation, the SIEM automatically generates a tracked incident ticket, attaches enriched contextual artifacts (IP reputation, user department, asset criticality), and executes automated containment playbooks (e.g., isolating a compromised workstation via EDR or disabling an Active Directory user account).

---

SIEM Deployment Models: Comparative Evaluation

Organizations select SIEM architectures based on data sovereignty requirements, regulatory compliance, operational engineering maturity, and budgetary models (CapEx vs. OpEx).

---

Sizing Considerations: EPS, Storage, Overhead, and Bandwidth Formulas

Accurate capacity planning prevents catastrophic performance degradation, dropped packets, and unexpected storage exhaustion.

Core Sizing Metrics

• Events Per Second (EPS): The primary measure of SIEM ingestion velocity.
• Average Event Size: The uncompressed byte length of a single log message (typically ranging from 350 bytes for simple syslog messages to 1,200+ bytes for verbose Windows XML Security events).
• Indexing Overhead ($O_{idx}$): The additional disk space consumed by search indexes, inverted indexes, and metadata tables, typically adding 20% to 50% ($0.20$ to $0.50$) to the raw payload size.
• Storage Compression Ratio ($C$): The factor to which raw text logs are reduced when written to persistent disk (typically achieving 2:1 to 5:1 compression, representing a compressed size factor of $0.50$ down to $0.20$).

Mathematical Sizing Formulas

$\text{Average EPS} = \frac{\text{Total Daily Event Count}}{86,400\text{ seconds}}$

$\text{Planning Peak EPS} = \text{Average EPS} \times \text{Observed or Design Peak Factor}$

A factor such as 2.5 can be a worked assumption, but production sizing should derive it from measured bursts, onboarding plans, failure recovery, and headroom.

$\text{Raw Daily Volume (GB/day)} = \frac{\text{Average EPS} \times 86,400\text{ sec} \times \text{Average Event Size (bytes)}}{1,073,741,824\text{ bytes/GB}}$

$\text{Net Daily Storage Required (GB/day)} = \text{Raw Daily Volume (GB/day)} \times (1 + O_{idx}) \times C$

$\text{Total Retention Storage (GB)} = \text{Net Daily Storage} \times \text{Retention Period (Days)}$

$\text{Sustained Payload Bandwidth (Mbps)} = \frac{\text{Average EPS} \times \text{Average Event Size (bytes)} \times 8 \times (1 - \text{WAN Compression Reduction}) \times (1 + \text{Protocol Overhead})}{1,000,000}$

Step-by-Step Worked Sizing Scenario

Consider an enterprise SOC sizing an on-premises distributed SIEM deployment under the following specifications:

• Sustained Ingestion Velocity: 5,000 EPS (Average)
• Average Raw Event Size: 500 bytes
• Indexing Overhead ($O_{idx}$): 30% ($0.30$)
• Storage Compression Ratio: 2.5:1 (meaning compressed data occupies $1 / 2.5 = 0.40$ of original volume)
• Hot/Warm High-Performance Retention: 90 days
• Cold Compliance Retention: 275 days (Total 365 days retention)
• Network Protocol Overhead: 10% ($0.10$)
• WAN Collector Compression: 30% reduction ($1 - 0.30 = 0.70$)

Step 1: Calculate Raw Daily Ingestion Volume

$\text{Raw GB/day} = \frac{5,000 \times 86,400 \times 500}{1,073,741,824} = \frac{216,000,000,000}{1,073,741,824} \approx 201.17\text{ GB/day}$

Step 2: Calculate Net Daily Disk Storage

Applying the 30% indexing overhead and 2.5:1 ($0.40$) compression: $\text{Net GB/day} = 201.17 \times (1 + 0.30) \times 0.40 = 201.17 \times 1.30 \times 0.40 \approx 104.61\text{ GB/day}$

Step 3: Calculate Tiered Storage Allocations

• Hot/Warm Storage (90 Days): $104.61\text{ GB/day} \times 90\text{ days} = 9,414.9\text{ GB} \approx 9.20\text{ TB (Usable NVMe/SSD Storage)}$
• Cold Storage (275 Days): $104.61\text{ GB/day} \times 275\text{ days} = 28,767.75\text{ GB} \approx 28.09\text{ TB (Usable Object/Archive Storage)}$
• Total Annual Usable Storage: $9.20\text{ TB} + 28.09\text{ TB} = 37.29\text{ TB}$

Step 4: Calculate WAN Ingestion Bandwidth

$\text{Sustained Bandwidth} = \frac{5,000 \times 500 \times 8 \times 0.70 \times 1.10}{1,000,000} = 15.4\text{ Mbps}$

This is a sustained payload estimate under the stated compression and overhead assumptions. If the design also assumes a 2.5× peak, the corresponding estimate is $15.4 \times 2.5 = 38.5\text{ Mbps}$ before additional engineering headroom. Capacity planning must also account for compression variability, retransmissions, acknowledgements, concurrent traffic, outage catch-up, and measured peak behavior.