7.2 Forwarder Types: Universal Forwarder vs. Heavy Forwarder

Forwarder Types: Universal Forwarder vs. Heavy Forwarder

Quick Summary: Selecting between a Universal Forwarder (UF) and a Heavy Forwarder (HF) is a fundamental architectural decision. The Universal Forwarder is a purpose-built, lightweight agent optimized for endpoint data collection without local parsing overhead. The Heavy Forwarder is a full Splunk Enterprise installation dedicated to ingestion workflows, equipped with complete parsing pipelines, modular input execution engines, and data transformation capabilities.

Architectural Footprint & Core Binaries

The fundamental difference between forwarder types stems from their underlying binary packages and runtime architecture:

1. The Universal Forwarder (UF)

The Universal Forwarder is distributed as an independent, stripped-down software package (splunkforwarder).

• Resource Footprint: The UF is designed to have a small CPU and memory footprint on production servers. It packages data into blocks without running index-time regular expressions, so its overhead is far lower than a full Splunk Enterprise instance.
• Omitted Subsystems: The UF completely omits the Splunk Web server (port 8000 does not exist), the indexing database engine, and the parsing pipeline queues.
• No Bundled Python: The UF does not ship with a Python interpreter. Scripted inputs on a UF must be executables or shell/batch/PowerShell scripts, or must call an interpreter that is already installed on the host. Python-based add-ons that need Splunk's bundled Python belong on a heavy forwarder.

2. The Heavy Forwarder (HF)

The Heavy Forwarder is created by installing the full Splunk Enterprise binary package (splunk) and configuring it to forward all data rather than index locally.

• Resource Footprint: It needs the disk, memory, and CPU of a full Splunk Enterprise install, which is why it is used for specific jobs rather than on every endpoint.
• Full Ingestion Pipeline: Possesses the complete processing pipeline, including parsingQueue and typingQueue. It can execute complex transformations, evaluate regular expressions against event payloads, and run heavy data manipulation scripts.
• Runtime Environment: Includes Splunk's bundled Python runtime and can host modular inputs and add-ons. Splunk DB Connect, for example, runs on a full Splunk Enterprise instance with a Java runtime installed on the host.
• Administrative Best Practice: By default, Splunk Enterprise enables Splunk Web. On a dedicated Heavy Forwarder, administrators should always disable the web interface via the CLI (splunk disable web) or in web.conf ([settings] startwebserver = 0) to conserve memory and eliminate an unnecessary network attack surface.

---

Ingestion Pipeline Mechanics: Unparsed vs. Parsed Streams

Understanding how data travels through pipeline queues explains why the two forwarders behave differently:

Pipeline Queues Overview

Splunk moves data between processing stages through bounded in-memory queues:

1. Input stage: inputs read bytes from files, ports, scripts, or APIs and attach host, source, sourcetype, and index.
2. parsingQueue → parsing pipeline: character-set conversion to UTF-8, line breaking (LINE_BREAKER), and header processing.
3. aggQueue → merging pipeline: line merging (SHOULD_LINEMERGE) and timestamp extraction (TIME_PREFIX, TIME_FORMAT).
4. typingQueue → typing pipeline: regex replacement (TRANSFORMS-*, SEDCMD-*) and punct annotation.
5. indexQueue → indexing pipeline: writes events to buckets or, on a forwarder, hands them to the TCP output (tcpout) queue.

Processing Path on a Universal Forwarder

On a Universal Forwarder, data flows directly from the inputQueue to the outputQueue:

• The UF captures raw byte streams from monitored files or network ports.
• It attaches basic transport metadata headers (source, sourcetype, host, index).
• It encapsulates data chunks into the Splunk-to-Splunk (s2s) protocol stream.
• Crucial Rule: For ordinary unstructured data, the UF does no event line breaking, no timestamp extraction, and no regex transforms. It forwards an unparsed stream, and the receiving indexer (or heavy forwarder) runs the parsing, merging, and typing pipelines. The exceptions are narrow: EVENT_BREAKER (to help load balancing), structured-data INDEXED_EXTRACTIONS, input-time settings such as CHARSET, and force_local_processing.

Processing Path on a Heavy Forwarder

A Heavy Forwarder runs data through the input, then parsingQueue -> aggQueue (merging) -> typingQueue -> its output (tcpout) queue:

• The HF can break raw streams into discrete events using LINE_BREAKER in props.conf.
• It parses and normalizes timestamps using TIME_FORMAT.
• It executes regex transformations, such as routing unwanted events to nullQueue in transforms.conf or executing SEDCMD to overwrite sensitive data strings.
• Cooked vs. Parsed: sendCookedData = true is the default for both forwarder types. It means data travels in Splunk's own format with metadata attached. From a UF the data is cooked but unparsed. From an HF it is cooked and parsed, so the receiving indexer skips the parsing, merging, and typing pipelines and only indexes the events. That moves CPU work from the indexers to the heavy forwarder.

---

Detailed Comparison Matrix

The table below summarizes the architectural, operational, and functional distinctions between Universal Forwarders and Heavy Forwarders:

---

When to Deploy: Practical Selection Scenarios

Choosing the correct forwarder depends on network topology, regulatory constraints, and data collection methods:

Scenario 1: Ubiquitous Endpoint Data Collection -> Universal Forwarder

For most server monitoring (web servers, database hosts, virtual machines, cloud instances), deploy the Universal Forwarder.

• Rationale: Low memory and CPU consumption keeps the impact on production workloads small. Standardizing on UFs simplifies patch management and eliminates configuration drift across thousands of endpoints. Parsing is handled centrally by the indexer tier.

Scenario 2: Regulatory Data Anonymization at the Perimeter -> Heavy Forwarder

When an organization must collect logs containing Protected Health Information (PHI) under HIPAA, or primary account numbers (PAN) under PCI-DSS, but security policy forbids unencrypted or unmasked sensitive data from crossing network boundaries.

• Rationale: The Heavy Forwarder must be placed within the secure source zone. It utilizes props.conf and SEDCMD to overwrite sensitive strings (e.g. replacing credit card numbers with XXXX-XXXX-XXXX-####) before the events enter the network stream toward the centralized indexers.

Scenario 3: Pre-Transit Data Reduction & Event Filtering -> Heavy Forwarder

A remote branch office generates 200 GB/day of noisy debug logs across a bandwidth-constrained WAN link. However, the security team only requires error, warning, and fatal events (approximately 10 GB/day).

• Rationale: Deploy a Heavy Forwarder locally at the branch office. Configure transforms.conf to evaluate event payloads and route all INFO and DEBUG events to nullQueue. The remaining 10 GB/day of high-value events are forwarded across the WAN, reducing bandwidth consumption by 95%.

Scenario 4: Intermediate Consolidation & DMZ Gateway -> Heavy Forwarder

Hundreds of legacy appliances (firewalls, routers, switches) stream raw UDP syslog data on port 514, but firewall rules prohibit hundreds of individual connections into the core network where indexers reside.

• Rationale: Deploy a Heavy Forwarder in the DMZ or network boundary. The HF collects local syslog streams, bundles them into an encrypted, load-balanced Splunk-to-Splunk connection over TCP 9997, and routes all traffic through a single pinhole in the internal firewall.

---

Administrative Traps & Operational Misconceptions

Misconception 1: The "Heavy Forwarders Consume License Quota" Myth

A widespread administrative misconception is that running a Heavy Forwarder burns daily indexing license capacity.

• The Reality: Splunk licensing measures only data written to index storage buckets on indexers. Because a pure Heavy Forwarder forwards all events without storing them locally, it consumes zero bytes of license quota. Forwarding 1 TB/day through an HF incurs no license fee until the data reaches the indexer cluster.

Misconception 2: Attempting Regex Event Filtering on a Universal Forwarder

Administrators frequently create props.conf and transforms.conf on Universal Forwarders with rules like REGEX = (?i)debug and DEST_KEY = queue FORMAT = nullQueue, expecting noisy events to be dropped at the agent.

• The Reality: Universal Forwarders do not run the typing pipeline for normal data, so they ignore TRANSFORMS-* settings such as DEST_KEY = queue / FORMAT = nullQueue. The events continue to stream across the network in full volume. On a UF, filtering is limited to input-level controls such as file whitelist/blacklist settings in inputs.conf (and event-code allow/deny lists on Windows event log inputs).

Misconception 3: Over-Parsing on Heavy Forwarders vs. Indexer Scaling

Some administrators deploy Heavy Forwarders everywhere under the theory that pre-parsing data offloads work from indexers.

• The Reality: Distributing parsing across hundreds of Heavy Forwarders creates immense configuration complexity, makes troubleshooting parsing rules difficult, and drastically increases the memory and maintenance footprint of every server. In modern architectures, parsing belongs centrally on indexers unless specific boundary masking, WAN reduction, or protocol bridging demands an HF.

Misconception 4: "More Forwarder Threads Will Fix Uneven Indexer Load"

When one indexer runs hot while others idle, administrators often reach for undocumented tuning.

• The Reality: Forwarders switch indexers on the autoLBFrequency timer (default 30 seconds) or after autoLBVolume bytes, choosing the next indexer at random from the target group. A universal forwarder can only switch safely at an event boundary. Without EVENT_BREAKER_ENABLE/EVENT_BREAKER for the sourcetype, it can stay "stuck" on one indexer while it streams a large file. A heavy forwarder knows event boundaries because it parses the data. Extra pipeline sets (parallelIngestionPipelines in server.conf) add throughput at the cost of CPU. They are not a load-balancing fix, and you should increase them only with care.