9.2 Pipeline Component Placement & Queue Management

Pipeline Component Placement & Queue Management

In an enterprise Splunk deployment, data processing pipelines are not executed on a single host. Instead, pipelines are distributed strategically across architectural tiers—spanning Universal Forwarders (UFs), Heavy Forwarders (HFs), and Indexers (Search Peers).

Deciding where specific processing steps occur has profound implications for network bandwidth, CPU consumption, data latency, and storage throughput. Furthermore, because data moves between pipeline stages through bounded in-memory queues, administrators must understand how flow control and backpressure operate to diagnose ingest bottlenecks and prevent data flow disruption.

---

Architectural Distribution Across Deployment Tiers

Splunk classifies data traversing the network into two distinct formats: uncooked data (raw, unparsed byte streams) and cooked data (events that have already undergone character decoding, line breaking, timestamp extraction, and typing).

+-------------------------------------------------------------------------+
| UNIVERSAL FORWARDER (Endpoint Tier)                                     |
|  - Executes: INPUT PIPELINE ONLY                                        |
|  - Emits: Uncooked Data Chunks via TCP 9997 (s2s)                       |
+-------------------------------------------------------------------------+
                                     |
                  Raw Uncooked Chunks (Port 9997)
                                     |
                                     v
+-------------------------------------------------------------------------+
| HEAVY FORWARDER (Optional Intermediate Tier)                            |
|  - Executes: INPUT, PARSING, MERGING, & TYPING PIPELINES                |
|  - Performs: Regex filtering (nullQueue), masking, PII scrubbing        |
|  - Emits: Cooked Data via TCP 9997 or Syslog/JSON to Third-Party        |
+-------------------------------------------------------------------------+
                                     |
                   Pre-Processed Cooked Events (Port 9997)
                                     |
                                     v
+-------------------------------------------------------------------------+
| INDEXER / SEARCH PEER (Storage Tier)                                    |
|  - Receives Cooked Data: Directly enqueues into INDEXING PIPELINE       |
|  - Receives Uncooked Data: Executes PARSING, MERGING, TYPING & INDEXING |
|  - Writes: journal.zst rawdata and *.tsidx inverted index files         |
+-------------------------------------------------------------------------+

1. The Universal Forwarder (UF)

The Universal Forwarder is a purpose-built client engineered for a small system footprint.

• Pipelines Executed: Input Pipeline only (pipeline = input).
• Capabilities: It runs file monitors, Windows inputs, scripted inputs, and network inputs. It assigns initial metadata (host, source, sourcetype, and target index) based on inputs.conf.
• Limitations: The UF binary does not contain the Python runtime, Splunk Web, or the regex parsing and transformation libraries required for line breaking and timestamp extraction. For normal data it does not apply parsing-phase props.conf settings such as TIME_FORMAT or TRANSFORMS. It does apply input-time settings (CHARSET, NO_BINARY_CHECK, [source::] source type assignment), EVENT_BREAKER, and structured-data INDEXED_EXTRACTIONS.
• Output: It packages raw byte streams into internal transmission chunks and pushes them through outputQueue / tcpOutQueue, sending uncooked data across TCP port 9997 using the Splunk-to-Splunk (s2s) protocol.

2. The Heavy Forwarder (HF)

The Heavy Forwarder is a full installation of Splunk Enterprise with Splunk Web typically disabled (splunk disable web).

• Pipelines Executed: Input, Parsing, Merging, and Typing Pipelines.
• Capabilities: Because it contains the complete Splunk processing libraries, the Heavy Forwarder can perform character set decoding (utf8Processor), execute regex line breaking (LINE_BREAKER), extract timestamps into _time, and evaluate transforms.conf rules.
• Common Enterprise Roles:
  • Data Scrubbing and Masking: Redacting sensitive customer data (credit cards, passwords) before events leave a secure network segment.
  • Noise Filtering: Discarding unwanted debug logs by routing them to nullQueue, saving network bandwidth and index licensing costs.
  • Content-Based Routing: Inspecting event contents and directing specific streams to distinct indexer clusters or geographic regions.
  • Third-Party Forwarding: Converting event streams into formatted syslog, raw text, or JSON to feed external SIEMs, Kafka clusters, or object storage repositories.
• Output: When sending to Splunk indexers, the Heavy Forwarder transmits cooked data over TCP port 9997. The events arrive pre-parsed, line-broken, and timestamped.

3. The Indexer (Search Peer)

The Indexer is responsible for final data persistence and index generation, but its pipeline workload depends entirely on whether incoming data is cooked or uncooked:

• Ingesting from Universal Forwarders (Uncooked Data): Because the UF only ran the Input phase, the Indexer must execute the Parsing, Merging, Typing, and Indexing pipelines before the data can be written to disk.
• Ingesting from Heavy Forwarders (Cooked Data): Because the Heavy Forwarder already completed parsing, merging, and typing, the Indexer bypasses those three pipelines entirely. The inbound TCP receiver hands the pre-processed event records directly into indexQueue for the Indexing Pipeline, which writes the rawdata journal and generates .tsidx files.
• Ingesting Local Data: When an Indexer monitors its own local log files (e.g., $SPLUNK_HOME/var/log/splunk/splunkd.log), it executes all five pipelines locally: Input, Parsing, Merging, Typing, and Indexing.

Tier Pipeline Placement Summary

---

In-Memory Queues & Flow Control Architecture

Between every pair of pipeline stages sits a dedicated, thread-safe, in-memory FIFO queue. Queues serve as buffers that decouple producer threads from consumer threads, absorbing transient spikes in data volume without dropping events.

+------------------+     [inputQueue]     +-------------------+
|  Input Pipeline  | -------------------> |  Parsing Pipeline |
+------------------+                      +-------------------+
                                                    |
                                               [aggQueue]
                                                    |
                                                    v
+------------------+     [typingQueue]    +-------------------+
| Indexing Pipeline| <------------------- |  Typing Pipeline  |
+------------------+                      +-------------------+
        |
   [indexQueue]
        |
        v
+------------------+
| Hot Bucket Disk  |
+------------------+

Complete Queue Inventory

1. Input queues: Individual inputs have their own in-memory queues (network and scripted inputs can add a persistentQueueSize on disk).
2. parsingQueue: Feeds the parsing pipeline. Data from inputs, and unparsed data received from forwarders, enters here.
3. aggQueue (Aggregation Queue): Connects the parsing pipeline to the merging pipeline. It buffers discrete lines awaiting multi-line assembly and timestamp extraction.
4. typingQueue: Connects the merging pipeline to the typing pipeline. It holds fully assembled events awaiting punctuation calculation, regex transforms, index routing, and masking.
5. indexQueue: Connects the typing pipeline to the indexing pipeline. It buffers fully prepared events awaiting compression into the rawdata journal (journal.zst) and inverted index tokenization (.tsidx).
6. tcpOutQueue (Output Queue): Present on forwarders. It buffers events awaiting socket transmission across the network to downstream indexers or intermediate forwarders.

---

The Mechanics of Upstream Backpressure

Every in-memory queue in Splunk has a bounded capacity. The [queue] stanza in server.conf sets a default maxSize of 500 KB, and individual queues can be sized separately. Bounded queues are essential for system stability: if queues were unbounded, an ingestion bottleneck would cause memory consumption to balloon until the operating system terminated splunkd via Out-Of-Memory (OOM) errors.

However, bounded queues introduce backpressure. When a downstream consumer thread slows down or stalls, the queue feeding it fills to capacity. Once a queue reaches a fill ratio of 1.00 (100% full), the upstream producer thread cannot write new data and must block. This blocking condition then cascades backward through every preceding queue until it reaches the initial data source.

Backpressure Cascade Scenario: Saturated Storage Subsystem

Consider what happens when an indexer's underlying storage volume experiences severe I/O contention (e.g., slow RAID arrays or saturated cloud storage IOPS):

[1. Disk Write Bottleneck]  --> IOPS saturated; the indexer cannot write fast enough
            |
[2. indexQueue Fills (1.0)] --> typing pipeline blocked; cannot push to indexQueue
            |
[3. typingQueue Fills]      --> merging pipeline blocked; cannot push to typingQueue
            |
[4. aggQueue Fills]         --> parsing pipeline blocked; cannot push to aggQueue
            |
[5. parsingQueue Fills]     --> network receiver blocked; cannot push to parsingQueue
            |
[6. TCP Zero Window Sent]   --> TCP socket buffer fills; TCP window drops to 0
            |
[7. Forwarder tcpOutQueue]  --> Forwarder outputQueue fills to 100%
            |
[8. Ingestion Halts]        --> tailingProcessor stops reading log files from disk

1. Storage Bottleneck: The indexer disk cannot write fast enough, so the indexing pipeline slows down.
2. indexQueue Saturation: Incoming events accumulate until indexQueue reaches its maximum capacity (current_size = max_size, fill ratio = 1.00).
3. Typing Pipeline Blocks: The typing pipeline attempts to push an event into indexQueue. Because the queue is full, the thread blocks.
4. typingQueue Saturation: Events waiting for transforms back up until typingQueue fills to 1.00.
5. Merging Pipeline Blocks: The aggregator cannot push into typingQueue and blocks, causing aggQueue to fill.
6. Parsing Pipeline Blocks: The line breaker cannot push into aggQueue and blocks, causing parsingQueue to fill.
7. Network Socket Stalls: The TCP receiver thread on port 9997 cannot push into parsingQueue. Incoming network packets fill the operating system's TCP socket receive buffer.
8. TCP Window Exhaustion: The indexer's TCP stack sends a TCP Zero Window packet to the upstream forwarder, signaling that its network buffers are completely full.
9. Forwarder tcpOutQueue Saturated: The forwarder can no longer transmit packets across port 9997. Outbound events fill the forwarder's local tcpOutQueue.
10. Data Safely Retained on Disk: With tcpOutQueue full, the forwarder's tailingProcessor halts reading monitored log files. Crucially, no events are lost. Data remains safe inside the original log files on the client machine until the indexer's queues clear and processing resumes.

---

Queue Health Monitoring & Troubleshooting

When ingestion slows down or forwarder queues stall, administrators must identify which specific queue in the pipeline is blocked.

1. Identifying Blocked Queues in metrics.log

Every Splunk instance writes queue statistics to metrics.log (indexed into _internal) about every 30 seconds, one group=queue event per queue. A full queue is reported with blocked=true:

... INFO  Metrics - group=queue, name=indexqueue, blocked=true, max_size_kb=500, current_size_kb=499, ...

Diagnostic SPL Searches in _internal

# Which queues have been blocked, and where?
index=_internal source=*metrics.log group=queue blocked=true
| stats count by host, name
| sort - count

# Queue fill percentage over time
index=_internal source=*metrics.log group=queue
| eval fill_pct = round(current_size_kb / max_size_kb * 100, 2)
| timechart max(fill_pct) by name

2. Using the Monitoring Console (MC)

The Splunk Monitoring Console provides pre-built dashboards that visualize pipeline health without requiring manual SPL:

• Navigate to Monitoring Console > Indexing > Performance > Indexing Performance: Instance.
• Review the Queue Fill Ratio panel: This displays an area chart showing the percentage fill (0% to 100%) for parsingQueue, aggQueue, typingQueue, and indexQueue over time.

3. Isolating Root Causes by Queue Location

Because backpressure cascades backwards, the true root cause of an ingestion bottleneck is always the furthest downstream queue that is 100% full:

---

Queue Tuning & Multi-Pipeline Scaling in server.conf

When ingestion pipelines experience micro-bursts, administrators can tune queue sizes and parallelize pipelines in $SPLUNK_HOME/etc/system/local/server.conf.

1. Adjusting Queue Sizes (maxSize)

Queue capacities are defined under [queue] or queue-specific stanzas in server.conf:

[queue]
# Global default queue size (default is usually 500KB)
maxSize = 10MB

[queue=parsingQueue]
maxSize = 20MB

[queue=aggQueue]
maxSize = 20MB

[queue=typingQueue]
maxSize = 20MB

[queue=indexQueue]
maxSize = 50MB

[!WARNING] The Queue Tuning Fallacy: Increasing maxSize does not make Splunk process data faster. A larger queue merely provides a bigger shock absorber. If the underlying disk or regex cannot keep up with continuous incoming volume, a 50 MB queue will eventually fill up just like a 500 KB queue—it will simply take a few minutes longer to trigger backpressure while consuming significantly more server RAM.

2. Scaling with Multiple Pipeline Sets (parallelIngestionPipelines)

Historically, Splunk processed data through a single set of pipeline threads. On modern enterprise servers equipped with 24, 32, or 64 CPU cores, a single pipeline thread set can become CPU-bound on a single core while other cores remain idle.

To use more cores for ingestion, administrators can configure multiple parallel pipeline sets in server.conf:

[general]
parallelIngestionPipelines = 2

How Pipeline Sets Operate:

• With parallelIngestionPipelines = 2, Splunk creates two independent pipeline sets, each with its own parsing, merging, typing, and indexing pipelines and queues.
• New inputs are assigned to pipeline sets by pipelineSetSelectionPolicy, which is round_robin, weighted_random, or blocked_queue_count.
• The default is 1, which Splunk calls optimal for most installations. More pipeline sets use more CPU cores, leaving fewer for searching.
• Each pipeline set enforces some limits independently, including maxHotBuckets and maxHotSpanSecs in indexes.conf. A TCP or UDP stream (such as syslog) uses only one pipeline.