10.2 Indexer Acknowledgment, Data Cloning & Forwarder Routing

Indexer Acknowledgment, Data Cloning & Forwarder Routing

Quick Summary: Enterprise forwarder topologies require more than simple stream distribution; they must protect data in flight, support multi-environment replication, and selectively direct high-value records. This section explores Indexer Acknowledgment (useACK = true) to protect against loss of in-flight data, parallel Data Cloning for synchronized environments, and the critical distinction between input-level and event-level selective data routing.

---

The Resilience Challenge in High-Volume Ingestion

When a forwarder transmits data over a standard TCP connection, it relies on the operating system's network stack for transport confirmation. A standard TCP ACK confirms only that the packets were received by the operating system kernel buffer on the destination host.

The Critical Failure Scenario: If the destination indexer experiences a kernel panic, sudden power outage, or splunkd crash after the OS receives the network packet but before the Splunk indexing pipeline writes the raw event to disk buckets, the forwarder believes the data was delivered successfully. The forwarder advances its internal read pointers (updating its fishbucket record) and drops the events from memory. When the indexer reboots, the data residing in volatile RAM is gone forever, resulting in silent data loss.

To solve this vulnerability, Splunk Enterprise provides Indexer Acknowledgment.

---

Indexer Acknowledgment (useACK = true)

Indexer acknowledgment adds an application-level confirmation on top of TCP: the receiver confirms each block after it has handled it.

+-----------------------+                         +-----------------------+
|       FORWARDER       |                         |        INDEXER        |
|  (Memory waitACK      |                         |  (Splunk Pipeline &   |
|        Queue)         |                         |     Hot Buckets)      |
+-----------------------+                         +-----------------------+
           |                                                  |
           | ---- 1. Data Block + ACK Token (ID: 5001) ------> |
           |                                                  | (Passes through input,
           |    [Block retained in forwarder memory buffer]   |  parsing, & index queues)
           |                                                  |
           |                                                  | ---- 2. Writes rawdata
           |                                                  |         to hot bucket
           |                                                  |         journal on disk
           |                                                  v
           | <--- 3. Returns ACK Response (ID: 5001) ---------+ [Disk Commit Confirmed]
           |
           v
4. Forwarder releases block
   from memory buffer;
   advances fishbucket pointer

How Indexer Acknowledgment Works

1. Forwarder keeps a copy: With useACK = true, the forwarder keeps a copy of each block it sends in a wait queue until the receiver acknowledges it.
2. Receiver acknowledges: The receiver sends an acknowledgment when it has fully handled the data, typically when it has written it to disk in indexing.
3. Forwarder releases the copy: On acknowledgment, the forwarder drops the copy from its wait queue.
4. No acknowledgment: If none arrives, for example because the indexer crashed or the connection timed out (readTimeout, default 300 seconds, is used when reading acknowledgments), the forwarder resends the data to another indexer in the group.

useACK can be set in [tcpout] or in a [tcpout:<group>] stanza, but not for an individual [tcpout-server:...].

Crash Recovery & At-Least-Once Delivery

If the indexer crashes, the connection breaks, or the acknowledgment does not arrive before the read times out (readTimeout, default 300 seconds):

• The forwarder closes the broken socket.
• The forwarder takes the unacknowledged blocks from its wait queue.
• The forwarder immediately initiates a connection to an alternate, healthy indexer in its load-balancing target group.
• The unacknowledged blocks are retransmitted in order to the new indexer.

[!IMPORTANT] Indexer Acknowledgment enforces an at-least-once delivery model. If an indexer successfully commits a block to disk but crashes a millisecond before the ACK packet reaches the network wire, the forwarder will retransmit the block to another indexer upon reconnection. This can introduce transient duplicate events in the index. However, in enterprise administration, search-time deduplication (dedup) or transactional grouping easily resolves duplicates, whereas silent data loss is irreversible.

---

Architectural Trade-Offs & Queue Tuning for useACK

Indexer Acknowledgment is recommended for data you cannot afford to lose, but plan for its memory and network overhead.

1. Forwarder Memory Footprint (maxQueueSize)

Every block sent with acknowledgment stays in memory until it is acknowledged, so the forwarder needs more queue space:

• With maxQueueSize = auto (the default), the output queue is 500 KB without useACK and 7 MB with it.
• With useACK = true, the forwarder also creates a wait queue three times the size of maxQueueSize (for example, maxQueueSize = 1024KB gives a 3072 KB wait queue).
• If the queues fill, for example on a slow or high-latency link, the forwarder applies backpressure to its inputs. File monitors simply stop reading until the queues drain, and the data stays in the files.
• Increase maxQueueSize deliberately (for example maxQueueSize = 20MB) on forwarders with high throughput or high-latency links.

2. Network Latency & the Bandwidth-Delay Product (BDP)

When forwarding data across Wide Area Networks (WANs) or inter-continental links with high Round-Trip Time (RTT)—such as 100 ms between London and New York—the forwarder must wait 100 ms for every ACK token.

According to the Bandwidth-Delay Product formula: $\text{Buffer Required} = \text{Throughput} \times \text{Round-Trip Latency}$

If the forwarder's queues are smaller than the data that can be in flight during one round trip, the forwarder pauses while waiting for acknowledgments, capping throughput below line speed. Size maxQueueSize for the link's throughput and latency.

3. No Setting Is Needed on the Indexer

Indexer acknowledgment is enabled on the forwarder with useACK in outputs.conf. The receiving [splunktcp://9997] input does not need a matching setting.

---

Data Cloning: Parallel Stream Distribution

Data Cloning is a forwarder capability that duplicates incoming raw data streams and transmits identical, full-fidelity copies to two or more independent indexer clusters or environments simultaneously.

                                    +-----------------------+
                                    |   FORWARDER MEMORY    |
                                    | (Raw Ingestion Block) |
                                    +-----------------------+
                                                |
                 +------------------------------+------------------------------+
                 |                                                             |
                 v (Stream Copy A)                                             v (Stream Copy B)
+-------------------------------------------------+           +-------------------------------------------------+
|        TARGET GROUP: [tcpout:prod_cluster]      |           |        TARGET GROUP: [tcpout:dev_cluster]       |
|  - server = idx-prod01:9997, idx-prod02:9997   |           |  - server = idx-dev01:9997, idx-dev02:9997     |
|  - useACK = true                                |           |  - useACK = false                               |
+-------------------------------------------------+           +-------------------------------------------------+

Data Cloning vs. Indexer Cluster Replication

Administrators frequently confuse Data Cloning with Indexer Cluster Replication. They serve fundamentally different architectural objectives:

Configuring Data Cloning in outputs.conf

Data cloning is configured by listing multiple target groups separated by commas in the defaultGroup attribute within the global [tcpout] stanza:

[tcpout]
defaultGroup = prod_cluster, test_cluster

# Production Cluster Target Group
[tcpout:prod_cluster]
server = idx-p01.corp.internal:9997, idx-p02.corp.internal:9997
autoLBFrequency = 30
useACK = true
maxQueueSize = 20MB

# Staging/Test Environment Target Group
[tcpout:test_cluster]
server = idx-t01.corp.internal:9997, idx-t02.corp.internal:9997
autoLBFrequency = 30
useACK = false

Operational Considerations of Data Cloning

1. Network Egress Multiplication: Data cloning doubles (or triples, depending on group count) the outbound network bandwidth utilized by the forwarder. On saturated WAN links, cloning can cause network interface contention.
2. Independent Queue Execution: Each target group maintains its own independent egress queue and load-balancing engine. If test_cluster experiences an outage, its queue will block independently; however, if forwarder memory resources are constrained, backpressure from the failed group can eventually impact overall forwarder ingestion unless queues are properly isolated.
3. License Metering: Because each destination cluster operates an independent indexing tier, each cluster indexes the full volume of cloned data against its own local license pool.

---

Selective Data Routing: Input-Level vs. Event-Level

In complex enterprise architectures, forwarding all data to all indexers is neither efficient nor cost-effective. Organizations must selectively route specific data streams—such as sending payment logs to a high-security PCI-compliant cluster, directing firewall traffic to an operational cluster, or filtering debug logs to nullQueue.

Splunk provides two distinct mechanisms for selective routing:

1. Input-Level Routing (via _TCP_ROUTING in inputs.conf).
2. Event-Level Routing (via props.conf and transforms.conf).

Method 1: Input-Level Routing via inputs.conf

Input-level routing assigns an entire data input stanza (such as a directory monitor, script, or UDP listener) to one or more specific outputs.conf target groups using the _TCP_ROUTING attribute.

Supported On: Both Universal Forwarders and Heavy Forwarders.

# inputs.conf on Universal Forwarder

# Route Linux audit and authentication logs exclusively to security indexers
[monitor:///var/log/audit/audit.log]
sourcetype = linux:audit
_TCP_ROUTING = security_indexers

# Route standard web server logs exclusively to general indexers
[monitor:///var/log/nginx/access.log]
sourcetype = nginx:access
_TCP_ROUTING = general_indexers

# Selectively clone sensitive database logs to both security and audit clusters
[monitor:///var/log/mysql/mysql-audit.log]
sourcetype = mysql:audit
_TCP_ROUTING = security_indexers, audit_archive

• When _TCP_ROUTING is specified on an input, Splunk ignores the defaultGroup setting in outputs.conf for that specific input and directs the data stream solely to the declared groups.
• Multiple target groups can be comma-separated to clone only that specific input without globally cloning all forwarder data.

Method 2: Event-Level Selective Routing via props.conf & transforms.conf

Event-level routing inspects the raw text of individual events using regular expressions, dynamically directing each record to a target group or discarding it entirely.

[!CAUTION] The Architectural Pipeline Boundary: Event-level routing requires the typing pipeline, where TRANSFORMS run. Universal forwarders do not run it for normal data; they stream unparsed chunks. Therefore, event-level selective routing via props.conf and transforms.conf cannot be executed on a Universal Forwarder. It must run on a Heavy Forwarder or an Indexer.

Configuration on a Heavy Forwarder:

Step 1: Define the transform trigger in props.conf:

# props.conf on Heavy Forwarder
[cisco:asa]
TRANSFORMS-routing = route_firewall_critical, route_firewall_info
TRANSFORMS-filter  = drop_debug_events

Step 2: Configure regex rules in transforms.conf:

# transforms.conf on Heavy Forwarder

# Route critical severity events (%ASA-1 through %ASA-4) to PCI cluster
[route_firewall_critical]
REGEX = %ASA-[1-4]-
DEST_KEY = _TCP_ROUTING
FORMAT = pci_cluster

# Route informational severity events (%ASA-5 through %ASA-6) to general cluster
[route_firewall_info]
REGEX = %ASA-[5-6]-
DEST_KEY = _TCP_ROUTING
FORMAT = general_cluster

# Drop low-value debug noise (%ASA-7-) entirely to nullQueue before forwarding
[drop_debug_events]
REGEX = %ASA-7-
DEST_KEY = queue
FORMAT = nullQueue

• DEST_KEY = _TCP_ROUTING: Reassigns the target group metadata for matching events to the group specified in FORMAT (which must match a [tcpout:<group>] stanza in outputs.conf).
• DEST_KEY = queue and FORMAT = nullQueue: Directs matching events to the bit-bucket (nullQueue), dropping them from memory immediately and preventing them from consuming network bandwidth or indexer storage.

---

Routing Methods Comparison & Decision Matrix

---

Administrative Pitfalls & Troubleshooting

Pitfall 1: Attempting Regex Routing on Universal Forwarders

• Symptom: An administrator deploys an app containing props.conf and transforms.conf with DEST_KEY = _TCP_ROUTING to a fleet of Universal Forwarders, but all events continue flowing to the defaultGroup.
• Root Cause: Universal Forwarders lack the parsing pipeline. They ignore DEST_KEY transformations in transforms.conf completely.
• Remedy: Either migrate the ingestion point to a Heavy Forwarder tier or partition data at the file/directory level using _TCP_ROUTING in inputs.conf.

Pitfall 2: High WAN Latency with Default Queue Limits

• Symptom: Ingestion stalls across inter-site forwarders when useACK = true is enabled, even though bandwidth utilization is well below link capacity.
• Root Cause: The maxQueueSize in outputs.conf is too small to handle the bandwidth-delay product of the network link, forcing the forwarder to wait for ACK tokens before enqueuing new data.
• Remedy: Increase maxQueueSize in outputs.conf on the forwarder to cover the link's bandwidth-delay product (remember the wait queue is three times this value).

Pitfall 3: Indexer Bottlenecks Propagating Upstream

• Symptom: Universal Forwarders across the enterprise stop reading logs; monitoring scripts report file tailing delays.
• Root Cause: Downstream indexers are slow to write to disk. With useACK = true the forwarders wait for acknowledgments, so their queues fill and backpressure pauses their file inputs.
• Remedy: Monitor indexer disk write latencies and pipeline queues using the Monitoring Console. Resolve indexer storage contention to restore forwarder throughput.