2.2 Telemetry Ingestion Architecture & Data Pipelines

Ingestion pipelines form the critical acquisition and transformation backbone of any Security Information and Event Management (SIEM) deployment. If raw telemetry is delayed, dropped due to queue saturation, or improperly mapped, downstream detection rules fail silently. Modern Elastic SIEM deployments employ a multi-tiered ingestion architecture that balances lightweight endpoint shipping, centralized policy orchestration, resilient intermediate buffering, and high-performance pre-indexing transformations.

---

1. The Telemetry Collection Ecosystem: Beats vs. Elastic Agent

Historically, the Elastic Stack utilized specialized, standalone shippers known as Beats. While legacy environments still run Beats, modern deployments center on the Unified Elastic Agent managed by Fleet.

Standalone Beats Shippers

Beats are lightweight, single-purpose data forwarders written in Go. They install directly on servers and network devices to capture specific telemetry streams:

• Filebeat: High-throughput log forwarding. Employs a harvest registry to track file read offsets and guarantees at-least-once delivery with backpressure handling. Includes pre-packaged modules for common security data sources (Suricata, Zeek, Cisco ASA/FTD, Palo Alto Networks, Okta, AWS CloudTrail) that package pre-configured Ingest Pipelines, ECS field mappings, and Kibana dashboards.
• Winlogbeat: Windows Event Log collection engine. Interacts directly with the native Windows Event Log API to read standard channels (Security, System, Application) and specialized diagnostic channels (Microsoft-Windows-Sysmon/Operational, Microsoft-Windows-PowerShell/Operational, Microsoft-Windows-Windows Defender/Operational). Supports channel filtering, event ID whitelisting, and localized event XML parsing.
• Packetbeat: Passive network packet analyzer. Uses libpcap or Npcap to capture wire traffic on endpoints or network taps without proxying connections. Dissects protocols including DNS, HTTP, TLS (extracting SNI, cipher suites, certificate fingerprints), SMB, and flow statistics, outputting normalized ECS network documents.
• Auditbeat: Host audit framework and integrity monitoring agent for Linux and macOS. Interfaces with the Linux auditd kernel module to capture process execution, syscalls, and user logins. Incorporates a File Integrity Monitoring (FIM) module that computes SHA-256 hashes and monitors metadata changes on sensitive directories (/etc, /bin, /usr/bin).

Unified Elastic Agent

While Beats are highly efficient, maintaining separate configuration files (filebeat.yml, winlogbeat.yml, auditbeat.yml) and orchestrating binary upgrades across tens of thousands of endpoints creates significant operational overhead. The Elastic Agent unifies this ecosystem into a single binary.

Under the hood, Elastic Agent acts as a managed supervisor daemon that controls sub-processes for log collection, metrics, OSquery inspection, and Elastic Defend (endpoint security and EDR capabilities). Rather than configuring local YAML files, administrators manage Elastic Agent entirely through Kibana using centralized Agent Policies.

---

2. Fleet Server Architecture and Communication Model

Fleet Server is the centralized control plane that orchestrates Elastic Agents across an enterprise infrastructure. It runs as the Fleet Server integration inside an Elastic Agent, either self-managed or hosted by Elastic Cloud.

+-------------------------------------------------------------+
|                        Kibana UI                            |
|          (Agent Policy Creation, Integration Config)        |
+------------------------------+------------------------------+
                               |
                               v (REST API)
+-------------------------------------------------------------+
|                    Elasticsearch Cluster                    |
|               (Stores .fleet-* Indices & Telemetry)         |
+------------------------------+------------------------------+
                               ^
                               | (Internal API)
+------------------------------v------------------------------+
|                        Fleet Server                         |
|               (HTTPS Control Plane, Port 8220)              |
+------------------------------+------------------------------+
                               ^
                               | Outbound HTTPS Long-Polling
          +--------------------+--------------------+
          |                                         |
+---------+---------+                     +---------+---------+
|   Elastic Agent   |                     |   Elastic Agent   |
| (Windows Endpoint)|                     |  (Linux Server)   |
+-------------------+                     +-------------------+

Communication Mechanism

1. Outbound-Only Connectivity: Elastic Agents initiate an outbound HTTPS connection to Fleet Server (default port 8220). Endpoints never listen on inbound network ports, eliminating firewall attack surface on managed hosts.
2. Long-Polling Check-In: Agents maintain an active HTTP long-polling connection with Fleet Server. Fleet Server holds the request open. When a security engineer modifies an Agent Policy in Kibana (such as enabling Sysmon ingestion or adjusting Elastic Defend malware blocking), Fleet Server pushes the configuration delta over the established connection within seconds.
3. Status Heartbeats: Agents check in regularly with their health (healthy, degraded or failed components, updating) and integration status, which Fleet uses to show statuses such as Healthy, Unhealthy, Updating and Offline, which records them in the .fleet-* system indices.
4. Stateless Scalability: Fleet Server is completely stateless. Multiple Fleet Server instances can be deployed behind a standard enterprise load balancer (e.g., F5, HAProxy, AWS ALB). If a Fleet Server instance crashes, connected agents seamlessly reconnect to alternate instances without interruption.
5. Enrollment Tokens and Scoped API Keys: Agents enroll into Fleet using an enrollment token associated with a specific Agent Policy. Upon successful validation, Fleet Server issues unique Elasticsearch API keys to each agent with least-privilege permissions constrained strictly to the data streams required by its assigned integrations.

---

3. Logstash: Ingestion Buffering and Complex ETL

While Elastic Agent can stream telemetry directly into Elasticsearch, enterprise SOCs often position Logstash as an intermediate aggregation and processing layer between edge shippers and the Elasticsearch cluster.

When to Deploy Logstash in SecOps

1. Surge Protection and Buffering: During security incidents (such as automated brute-force attacks or malware propagation), event volumes can spike by 500% to 1000%. If data nodes experience indexing backpressure, directly connected agents may buffer data locally or drop events. Logstash provides Persistent Queues (PQ) (queue.type: persisted), buffering incoming telemetry on local disk to ensure zero event loss while Elasticsearch recovers.
2. Multi-Source Enrichment: Logstash pipelines can enrich events by executing external queries against relational databases (jdbc_streaming), Memcached key-value stores, or threat intelligence platforms during transit.
3. Central Protocol Aggregation: Logstash can act as a central listener for many inputs (syslog over UDP/TCP, Beats, Kafka, HTTP) and route events to several destinations. Elastic Agent can also receive syslog and NetFlow directly through integrations such as Custom UDP/TCP logs and NetFlow Records. Logstash is therefore a choice for central buffering and complex routing, not a requirement for these protocols.
4. Dead-Letter Queues (DLQ): When Elasticsearch rejects an index request due to mapping conflicts (such as an IP field receiving a string that cannot be parsed), Logstash can write the rejected document to an on-disk dead-letter queue (dead_letter_queue.enable: true), allowing engineers to resolve the mapping conflict and replay the event.

---

4. Elasticsearch Ingest Pipelines and Key Processors

Ingest Pipelines execute pre-indexing document transformations directly on Elasticsearch nodes assigned the ingest role. Pipelines consist of an ordered series of processors that execute sequentially before a document is written into a Lucene segment.

Critical Ingest Processors for SecOps

dissect vs. grok: Performance Trade-offs

A fundamental optimization in security data pipelines is choosing between dissect and grok:

• dissect (Delimiter Parsing): Operates by matching fixed string delimiters. It does not compile regular expressions or perform backtracking. As a result, dissect is typically much faster than grok and consumes less CPU. It should always be used when log formats have a consistent structure (such as Palo Alto Networks CSV logs or standard CEF headers).
• grok (Regex Parsing): Necessary when log lines have variable structures, nested optional fields, or unstructured text (such as custom application audit logs or Linux daemon messages). Because regex backtracking can cause severe CPU saturation under high ingestion volumes, grok patterns must be carefully bounded (avoiding greedy .* patterns).

{
  "description": "Example Security Ingest Pipeline with ECS Normalization",
  "processors": [
    {
      "dissect": {
        "field": "message",
        "pattern": "%{syslog_timestamp} %{host.name} %{process.name}[%{process.pid}]: %{event_message}",
        "ignore_failure": false
      }
    },
    {
      "date": {
        "field": "syslog_timestamp",
        "target_field": "@timestamp",
        "formats": ["MMM d HH:mm:ss", "MMM dd HH:mm:ss"],
        "timezone": "UTC"
      }
    },
    {
      "geoip": {
        "field": "source.ip",
        "target_field": "source.geo",
        "ignore_missing": true
      }
    }
  ]
}

---

5. Pipeline Error Handling and Ingestion Failure Troubleshooting

In production SIEM pipelines, malformed logs, schema mismatches, and unexpected characters will inevitably trigger processor exceptions. If unhandled, the pipeline fails, the document is not indexed, and the client receives an error for that document. The security event is dropped unless the shipper retries or stores it elsewhere.

Handling Failures with on_failure

Elasticsearch Ingest Pipelines support two levels of on_failure error handling:

1. Processor-Level on_failure: Attached directly to an individual processor. If that specific processor errors (e.g., a convert processor fails because a port field contains non-numeric text), the processor-level fallback executes without failing the rest of the pipeline.
2. Pipeline-Level on_failure: Defined as a global array of processors at the root of the pipeline. If any processor in the pipeline fails and lacks a processor-level fallback, execution immediately jumps to the pipeline-level on_failure block.

Best Practice SecOps Error-Handling Pattern

A resilient security ingest pipeline must:

• Capture the exact error string into error.message.
• Append a failure tag (such as _dissectfailure or _grokparsefailure) to tags.
• Retain the unparsed raw log payload in event.original.
• Route the failed document into an error index or dead-letter data stream (e.g., logs-ingest_failures-default) rather than discarding it.

{
  "description": "Ingest Pipeline with Robust on_failure Catch-All",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{COMMON_PATTERN}"]
      }
    }
  ],
  "on_failure": [
    {
      "set": {
        "field": "error.message",
        "value": "{{ _ingest.on_failure_message }}"
      }
    },
    {
      "append": {
        "field": "tags",
        "value": ["_pipeline_failure", "_grokparsefailure"]
      }
    },
    {
      "set": {
        "field": "_index",
        "value": "logs-pipeline_error-default"
      }
    }
  ]
}

Debugging with the Simulate API

Security engineers can validate pipeline logic, benchmark processor latency, and test error handling using the Ingest Simulate API (POST _ingest/pipeline/<pipeline-id>/_simulate) with real log payloads before deploying pipelines to production.

---

6. Architectural Comparison: Agent vs. Beats vs. Logstash