10.1 Forwarder Setup & outputs.conf Load Balancing

Forwarder Setup & outputs.conf Load Balancing

Quick Summary: In distributed Splunk Enterprise deployments, forwarders collect data at the edge and stream it to the indexing tier. The configuration file outputs.conf dictates how forwarders establish network sessions, distribute data volumes across peer indexers, adapt dynamically to auto-scaling clusters via Indexer Discovery, and encrypt in-flight wire streams using TLS/SSL.

---

Architecture of Forwarder Data Egress

Forwarders—both Universal Forwarders (UFs) and Heavy Forwarders (HFs)—act as the primary data collection agents in an enterprise Splunk deployment. While inputs are defined in inputs.conf, the egress transmission of captured data is governed exclusively by outputs.conf.

The forwarding tier uses the proprietary Splunk-to-Splunk (s2s) protocol, which operates by default over TCP port 9997. The primary objective of the forwarding architecture is to deliver balanced ingestion across the indexer tier without overwhelming any single node, causing pipeline starvation, or truncating individual events during network handoffs.

+-----------------------------------------------------------------------------------+
|                            FORWARDER (UF or HF)                                   |
|  +--------------------+    +--------------------+    +-------------------------+  |
|  |    inputs.conf     | -> | Ingestion Pipeline | -> |      outputs.conf       |  |
|  | (Files, TCP, Win)  |    | (Memory Queue)     |    | (Routing & Load Balance)|  |
|  +--------------------+    +--------------------+    +-------------------------+  |
+-----------------------------------------------------------------------------------+
                                          |
                  Splunk-to-Splunk Protocol (s2s: TCP 9997)
                  Automatic Load Balancing (30s / Volume Switches)
                                          |
         +--------------------------------+--------------------------------+
         |                                |                                |
         v                                v                                v
+------------------+             +------------------+             +------------------+
|  INDEXER PEER 1  |             |  INDEXER PEER 2  |             |  INDEXER PEER 3  |
|  inputs.conf:    |             |  inputs.conf:    |             |  inputs.conf:    |
|  [splunktcp://   |             |  [splunktcp://   |             |  [splunktcp://   |
|          9997]   |             |          9997]   |             |          9997]   |
+------------------+             +------------------+             +------------------+

---

Complete Anatomy of outputs.conf

The outputs.conf configuration file is structured hierarchically. Global default parameters are declared in the [tcpout] stanza, while specific destinations, load-balancing groups, and connection behaviors are organized into named target groups using [tcpout:<target_group>] stanzas.

1. The Global [tcpout] Stanza

The [tcpout] stanza specifies default forwarder behavior and defines which target group or groups receive incoming data streams when an input does not specify an explicit destination:

[tcpout]
defaultGroup = primary_indexers
indexAndForward = false
disabled = false

• defaultGroup = <target_group1>, <target_group2>, ...: Identifies one or more comma-separated target group stanzas that receive forwarded data by default. If multiple groups are declared here, Splunk clones the data, transmitting identical streams to each group in parallel.
• indexAndForward = false|true: Applies almost exclusively to Heavy Forwarders and full Splunk Enterprise instances. When set to false (the default), the instance passes data directly to its egress queues without writing events to local index storage buckets. When set to true, the instance indexes the data locally into its own hot buckets and forwards a copy to the specified output target group.
• disabled = false|true: When set to true, completely deactivates the TCP forwarding engine, preventing any egress connections on port 9997.

2. The Named Target Group Stanza [tcpout:<target_group>]

A target group represents a logical pool of indexers. The forwarder load-balances across all reachable servers listed in the group:

[tcpout:primary_indexers]
server = idx01.corp.internal:9997, idx02.corp.internal:9997, idx03.corp.internal:9997
autoLBFrequency = 30
autoLBVolume = 0
compressed = false
dnsResolutionInterval = 300
maxQueueSize = 20MB
useACK = true

• server = <host1>:<port1>, <host2>:<port2>, ...: A comma-separated list of receiving indexer endpoints. Each host entry must define the fully qualified domain name (FQDN) or IP address and the receiving s2s TCP port (typically 9997).
• compressed = false|true: Specifies whether data chunks are compressed (using zlib) on the forwarder prior to transmission. While setting this to true reduces WAN bandwidth consumption, it increases CPU overhead on both the forwarder and the indexer. Modern high-bandwidth enterprise LANs generally keep compression disabled.
• dnsResolutionInterval = <integer> (default 300): The base interval, in seconds, for re-resolving the host names in server. The runtime interval is extended by 30 seconds for each additional indexer in the list.

3. Receiving Configuration on the Indexing Tier (inputs.conf)

For an indexer to accept s2s data streams sent from forwarders, the indexer must define a network receiver stanza in its local inputs.conf:

[splunktcp://9997]
connection_host = dns

When [splunktcp://9997] is enabled, the indexer opens TCP port 9997 and binds it to its inputQueue, passing raw byte streams into the parsing and typing pipelines.

Core outputs.conf Attribute Reference

---

Automatic Load Balancing Mechanics

Unlike traditional network load balancers (F5, AWS ALB, NGINX) that terminate HTTP requests or distribute individual packets, Splunk utilizes client-side, connection-based load balancing built directly into the forwarder binary.

How Splunk Distributes Streams Across Indexers

1. The forwarder resolves the indexers listed in the target group's server setting, re-resolving DNS every dnsResolutionInterval (300 seconds by default, plus 30 seconds per additional indexer).
2. It sends data to one indexer at a time over a single connection.
3. When the time or volume threshold is reached, it switches to another indexer, chosen at random from the list, at a point where it is safe to switch.
4. Load balancing is always automatic. The current outputs.conf.spec has no separate autoLB switch to turn on.

Forwarder egress timeline (autoLBFrequency = 30s):
[00:00-00:30]  ==> Indexer 2   (switch)
[00:30-01:00]  ==> Indexer 3   (switch: next indexer chosen randomly)
[01:00-01:30]  ==> Indexer 1   (switch)
[01:30-02:00]  ==> Indexer 3

Time-Based Load Balancing (autoLBFrequency)

By default, Splunk load balancing is controlled by time via autoLBFrequency:

• Default value: 30 seconds (autoLBFrequency = 30).
• Every 30 seconds, the forwarder switches to another indexer in the group, chosen at random, once it reaches a safe switching point.
• Tuning Consideration: Setting autoLBFrequency too low (e.g., 5 seconds) causes excessive TCP socket open/close overhead, TLS handshake churn, and pipeline serialization pauses. Setting it too high (e.g., 600 seconds) causes data to pool unevenly across indexers, creating disk utilization imbalances and hot bucket skew across the indexer cluster.

Volume-Based Load Balancing (autoLBVolume)

In environments characterized by bursty, high-volume ingestion—such as database extraction dumps, batch application exports, or sudden security incident floods—a forwarder can transmit tens of gigabytes within a 30-second window. Under standard time-based load balancing, the entire multi-gigabyte burst would be sent to a single indexer, causing that indexer's inputQueue to fill while neighboring indexers sit idle.

To prevent burst-induced indexer starvation, administrators configure volume-based load balancing:

[tcpout:primary_indexers]
server = idx01:9997, idx02:9997, idx03:9997
autoLBFrequency = 30
autoLBVolume = 10485760

• autoLBVolume = <bytes>: The volume of data to send to one indexer before a new indexer is randomly selected. 0 (the default) turns volume-based switching off.
• Example: autoLBVolume = 10485760 (10 MB). The forwarder switches indexers as soon as it transmits 10 MB, even if only 2 seconds have elapsed since the connection opened.
• Rule of Operation: The forwarder checks autoLBVolume first. If that volume has not been reached but autoLBFrequency has, it switches on time.

---

Event-Boundary Switching & Stream Integrity

A forwarder must not switch indexers in the middle of an event, or half of a multi-line event would land on each indexer.

Heavy Forwarders

A heavy forwarder has already parsed the data into events, so it can switch between events whenever the timer or volume threshold fires.

Universal Forwarders and EVENT_BREAKER

A universal forwarder sends unparsed data and does not know where events end. Without help, it waits for a point where switching is safe, such as the end of a file or a pause in the data. When it reads a large file continuously, it can therefore stay "stuck" on one indexer long after autoLBFrequency has passed, which leaves indexers unevenly loaded.

The fix is to tell the UF where events end for that source type, in props.conf on the universal forwarder:

[my_app:log]
EVENT_BREAKER_ENABLE = true
EVENT_BREAKER = ([\r\n]+)\d{4}-\d{2}-\d{2}

With EVENT_BREAKER the UF's lightweight chunked line breaker splits the stream at event boundaries so it can switch indexers on schedule. forceTimebasedAutoLB = true in outputs.conf forces switches on the timer instead. Splunk recommends EVENT_BREAKER_ENABLE/EVENT_BREAKER for universal forwarders.

---

Dynamic Indexer Discovery

In modern cloud architectures, indexer clusters scale horizontally to match variable analytical workloads. Adding or decommissioning peer indexers requires updating the server list in outputs.conf across thousands of forwarders. Manually managing static IP addresses or deploying configuration updates via the Deployment Server introduces configuration latency, temporary load imbalances, and administrative overhead.

To solve this challenge, Splunk provides Dynamic Indexer Discovery.

+----------------------------------------------------------------+
|                    CLUSTER MANAGER (CM)                        |
|  - server.conf: [indexer_discovery]                            |
|  - Maintains real-time registry of all active peer indexers    |
+----------------------------------------------------------------+
                               ^ (TCP 8089 HTTPS REST)
                               | Forwarder polls CM (interval set by CM)
                               | Returns JSON list of active peers
                               v
+----------------------------------------------------------------+
|                   UNIVERSAL FORWARDER (UF)                     |
|  - outputs.conf: indexerDiscovery = cluster_manager_pool       |
|  - Dynamically updates internal load-balancing pool            |
+----------------------------------------------------------------+
         |                                                |
         v (Streams over TCP 9997)                        v (Streams over TCP 9997)
+--------------------------------+               +--------------------------------+
|      ACTIVE INDEXER PEER 1     |               |      ACTIVE INDEXER PEER 2     |
+--------------------------------+               +--------------------------------+

Architectural Workflow of Indexer Discovery

1. Peer Registration: Each peer reports its receiving port and status to the cluster manager over the management port (TCP 8089).
2. Dynamic Polling: Forwarders are configured with the cluster manager's URI rather than a static list of indexers. They poll the manager over TCP 8089 at an interval the manager sets from its polling_rate and the number of connected forwarders.
3. Peer List Response: The Cluster Manager validates the forwarder's shared secret key (pass4SymmKey) and returns the current list of healthy, active indexer peers.
4. Local Pool Update: The forwarder dynamically updates its internal load-balancing table. If an indexer enters maintenance mode or goes offline, the Cluster Manager removes it from the list; the forwarder drops that indexer from its rotation during the next poll, routing traffic only to surviving peers.

Configuration on the Cluster Manager (server.conf)

Indexer discovery is enabled on the cluster manager:

[indexer_discovery]
pass4SymmKey = EnterpriseDiscoverySecretKey2026
polling_rate = 10
indexerWeightByDiskCapacity = false

• pass4SymmKey: Shared secret that forwarders must present. Use the same value in each forwarder's [indexer_discovery:<name>] stanza.
• polling_rate = <1-10> (default 10): Not a number of seconds. It is a rate factor that, together with the number of connected forwarders, sets how often forwarders poll: (number_of_forwarders / polling_rate + 30 seconds).
• indexerWeightByDiskCapacity = true: Tells forwarders to use weighted load balancing, sending more data to peers with larger total disk capacity.

Configuration on the Forwarder (outputs.conf)

On the forwarder, the target group points to an indexer discovery stanza instead of a static server list:

[tcpout]
defaultGroup = discovered_cluster

[tcpout:discovered_cluster]
indexerDiscovery = cluster_manager_pool
useACK = true

[indexer_discovery:cluster_manager_pool]
manager_uri = https://cm01.corp.internal:8089
pass4SymmKey = EnterpriseDiscoverySecretKey2026

• indexerDiscovery = <name>: Binds the [tcpout:<group>] stanza to the discovery definition.
• manager_uri: The cluster manager's management URI. master_uri is the deprecated name for the same setting.
• pass4SymmKey: Must match the value in the cluster manager's [indexer_discovery] stanza.
• polling_interval (default 5 seconds) is only how long the forwarder waits after startup before its first fetch. After that, the cluster manager's response sets the polling interval.

---

Forwarder Transport Security: TLS/SSL Encryption

In enterprise environments governed by security standards such as PCI-DSS, HIPAA, or ISO 27001, transmitting log data in cleartext across the network is prohibited. Splunk enables administrators to secure the s2s protocol (TCP 9997) using Transport Layer Security (TLS/SSL).

+-----------------------+                         +-----------------------+
|      FORWARDER        |                         |        INDEXER        |
|  - outputs.conf:      |                         |  - inputs.conf:       |
|    clientCert         |   TLS Encrypted s2s     |    [splunktcp-ssl:    |
|    sslPassword        | ----------------------> |             9997]     |
|    sslVerifyServerCert|       (TCP 9997)        |    [SSL] serverCert   |
| server.conf CA bundle |                         | server.conf CA bundle |
+-----------------------+                         +-----------------------+

Configuring TLS/SSL in outputs.conf

To enable TLS on the forwarder, configure certificate settings in the target group stanza of outputs.conf:

[tcpout:secure_cluster]
server = idx01.corp.internal:9997, idx02.corp.internal:9997
useACK = true

# TLS settings
clientCert = $SPLUNK_HOME/etc/auth/mycerts/forwarder_combined.pem
sslPassword = <private key password>
sslVerifyServerCert = true
sslCommonNameToCheck = idx01.corp.internal, idx02.corp.internal

The trusted CA bundle is set once per instance in server.conf:

[sslConfig]
sslRootCAPath = $SPLUNK_HOME/etc/auth/mycerts/enterprise_root_ca.pem

Attribute Breakdown:

1. clientCert = <path>: The forwarder's certificate in PEM format, usually with its private key. The old name sslCertPath is deprecated. Setting clientCert alone turns on SSL for the connection.
2. sslPassword: The password for the private key.
3. sslVerifyServerCert = true|false (default false): When true, the forwarder checks that the indexer's certificate is valid and signed by a trusted CA. Both the common name and the alternate name are then checked for a match.
4. sslCommonNameToCheck / sslAltNameToCheck: Optional lists of names the indexer's certificate must contain. They work only when sslVerifyServerCert = true.
5. sslRootCAPath in outputs.conf is deprecated. Use server.conf [sslConfig] sslRootCAPath for the CA bundle.

Indexer Configuration for TLS Reception (inputs.conf)

On the receiving indexers:

[splunktcp-ssl:9997]
disabled = 0

[SSL]
serverCert = $SPLUNK_HOME/etc/auth/mycerts/indexer_combined.pem
sslPassword = <private key password>
requireClientCert = true

• [splunktcp-ssl:<port>] opens a TLS-only Splunk-to-Splunk receiving port. Note the stanza form: there is no //.
• [SSL] holds the server certificate and its password. requireClientCert = true enforces mutual TLS, so each forwarder must present a certificate signed by a CA the indexer trusts. (rootCA here is deprecated in favor of server.conf sslRootCAPath.)

---

Administrative Pitfalls & Troubleshooting

Pitfall 1: Over-Aggressive Load Balancing Frequency

• Symptom: High CPU utilization on forwarders and indexers, frequent TcpOutputFd - Connection reset messages in splunkd.log, and delayed data indexing.
• Root Cause: Configuring autoLBFrequency to an excessively low value, such as 5 seconds. Because each connection switch requires socket teardown, TCP three-way handshake, and TLS negotiation, aggressive switching wastes significant network and CPU cycles on session management rather than data transmission.
• Remedy: Keep the 30-second default for autoLBFrequency. If large files pin a universal forwarder to one indexer, set EVENT_BREAKER_ENABLE/EVENT_BREAKER for that source type. Consider autoLBVolume for bursty senders.

Pitfall 2: Static server Entries in Clustered Environments

• Symptom: Data is directed to decommissioned indexers, generating connection errors; newly provisioned indexers receive zero data.
• Root Cause: Relying on static server lists in outputs.conf when operating an elastic or clustered indexer tier.
• Remedy: Implement dynamic Indexer Discovery. Forwarders retrieve the real-time cluster membership from the Cluster Manager, automatically adapting to node additions and maintenance states.

Pitfall 3: Certificate Verification Failures (sslVerifyServerCert)

• Symptom: Forwarders display SSL certificate validation failed: certificate has expired or common name does not match in $SPLUNK_HOME/var/log/splunk/splunkd.log, and forwarding stops completely.
• Root Cause: Indexer certificates were renewed without updating the SAN extension, or local system clock skew caused the forwarder to evaluate the certificate as not yet valid or expired.
• Remedy: Ensure Network Time Protocol (NTP) is synchronized across all forwarders and indexers. Validate that certificates contain wildcard SAN entries covering all indexer FQDNs (e.g., DNS:*.corp.internal).