17.3 Event Routing, Dynamic Index Assignment & Filtering to nullQueue

Event Routing, Dynamic Index Assignment & Filtering to nullQueue

Quick Summary: Index-time transforms can drop unwanted events, send events to other indexes, and correct metadata. DEST_KEY = queue with FORMAT = nullQueue discards matching events before they are indexed, so they do not count against the license. DEST_KEY = _MetaData:Index changes the index, and MetaData:Host, MetaData:Source, and MetaData:Sourcetype rewrite metadata, provided their values carry the host::, source::, or sourcetype:: prefix.

---

Filtering Noise to nullQueue: Mechanics & License Impact

Not all telemetry generated by enterprise infrastructure provides operational or analytical value. Verbose debug logs, load-balancer health-check pings, and heartbeat packets saturate network queues, consume storage, and inflate licensing costs. Splunk Enterprise provides a built-in mechanism to discard unwanted data at index time: routing to the nullQueue.

Typing Queue Routing Logic

                  +-----------------------------------+
                  |           typingQueue             |
                  |  Evaluates transforms.conf Regex  |
                  +-----------------------------------+
                                    |
                   +----------------+----------------+
                   |                                 |
  (Matches Filter Regex)                  (Does Not Match / Default)
                   |                                 |
                   v                                 v
      +------------------------+        +------------------------+
      |    DEST_KEY = queue    |        |       indexQueue       |
      |   FORMAT = nullQueue   |        |  (indexed & metered)   |
      +------------------------+        +------------------------+
                   |                                 |
                   v                                 v
      +------------------------+        +------------------------+
      |       nullQueue        |        |     Bucket Storage     |
      | (Discarded from Memory)|        |    (Written to Disk)   |
      |  * ZERO LICENSE COST * |        |                        |
      +------------------------+        +------------------------+

The Operational Pipeline Mechanism

nullQueue is an internal pipeline sink within the splunkd process. When an event's destination queue is set to nullQueue:

1. The event enters typingQueue on a Heavy Forwarder or Indexer.
2. Splunk matches the event against the REGEX in transforms.conf.
3. DEST_KEY = queue with FORMAT = nullQueue sets the event's queue key to nullQueue. Later transforms in the same chain still run and can change the key again.
4. After the typing pipeline, an event whose queue is nullQueue is discarded instead of going to indexQueue. It is never written to disk.

The Critical Licensing Advantage

License rule: license usage measures data as it is indexed. Events sent to the nullQueue in the typing pipeline never reach the index pipeline, so they are not counted against the daily license volume. They still cost forwarding bandwidth and parsing CPU, though, because they were read, sent, and parsed first.

This lets you collect a noisy stream, such as debug-level logs or verbose firewall traffic, keep the events that matter, and discard the rest without using license on them.

Concrete nullQueue Configuration: Filtering Kubernetes Health Checks

Suppose an ingress controller logs thousands of HTTP 200 health checks per minute from kube-probe:

# $SPLUNK_HOME/etc/apps/<app_name>/local/props.conf
[kube:ingress:access]
TRANSFORMS-drop_health = discard_kube_probes

# $SPLUNK_HOME/etc/apps/<app_name>/local/transforms.conf
[discard_kube_probes]
REGEX = (?i)"GET\s+/healthz\s+HTTP/\d\.\d"\s+200\s+.*"kube-probe"
DEST_KEY = queue
FORMAT = nullQueue

Default-Drop vs. Selective-Keep Patterns

In security or compliance use cases, administrators often want to drop everything except a whitelist of critical events. This requires chaining two transforms in props.conf:

# props.conf
[windows:security:filtered]
TRANSFORMS-01_filter = set_nullqueue_default, keep_critical_event_ids

# transforms.conf
# Step 1: Set everything to nullQueue by default
[set_nullqueue_default]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

# Step 2: Rescue specific EventIDs and direct them to indexQueue
[keep_critical_event_ids]
REGEX = (?ms)EventCode=(?:4624|4625|4720|4738)\b
DEST_KEY = queue
FORMAT = indexQueue

• set_nullqueue_default matches every single event (REGEX = .) and targets nullQueue.
• keep_critical_event_ids runs next, because it comes later in the same comma-separated list. If the event contains EventCode 4624 (successful logon), 4625 (failed logon), 4720 (user account created), or 4738 (user account changed), it sets the queue back to indexQueue.
• The order is essential. Reversing the list would send every event to the nullQueue, because the catch-all rule would run last and overwrite the keep decision.

---

Dynamic Routing to Alternate Indexes

By default, events are written to the index specified in inputs.conf. However, enterprise organizations frequently require routing events to different indexes based on the event's content—for example, separating PCI-scoped transactions from standard operational data, or routing logs by client tenant.

Static Index Redirection

To redirect events matching a security signature to a dedicated index:

# props.conf
[custom:app:log]
TRANSFORMS-pci_route = route_pci_events

# transforms.conf
[route_pci_events]
REGEX = (?i)(?:payment_gateway|pan_authorization|credit_charge)
DEST_KEY = _MetaData:Index
FORMAT = pci_secure

• DEST_KEY = _MetaData:Index: Note the leading underscore (_MetaData:Index). This tells the pipeline engine to update the target index metadata attribute.
• FORMAT = pci_secure: Overwrites the destination index with the literal string pci_secure.

Dynamic Index Routing Using Regex Capture Groups

Rather than creating dozens of static stanzas for every application or customer, administrators can capture tokens from the event payload and dynamically generate the destination index name:

# props.conf
[cloud:multitenant:audit]
TRANSFORMS-dynamic_tenant = route_by_tenant_id

# transforms.conf
[route_by_tenant_id]
REGEX = ^[^{]*\{"tenant":"([a-zA-Z0-9_-]+)"
DEST_KEY = _MetaData:Index
FORMAT = tenant_$1

• If an incoming event contains {"tenant":"finance", ...}, the capture group $1 extracts finance.
• FORMAT = tenant_$1 resolves to tenant_finance.
• The event is dynamically routed to the tenant_finance index.

Prerequisite: every index that FORMAT can produce must exist on all indexers (defined in indexes.conf, pushed through the cluster manager in a cluster). Per indexes.conf.spec, an event whose index does not exist, whether from an invalid index in inputs.conf or an invalid _MetaData:Index from FORMAT, is dropped entirely, unless lastChanceIndex names an existing, enabled index to receive it. With lastChanceIndex = default, the defaultDatabase index is used.

---

Overriding Core Metadata: Host, Sourcetype, and Source

In complex network topologies, default metadata assigned at ingestion is frequently inaccurate. A classic example is a centralized syslog aggregator (e.g., syslog-ng or a Heavy Forwarder) receiving events from hundreds of network switches. By default, Splunk sets host to the IP or hostname of the syslog relay, masking the identity of the true originating device.

Destination Keys for Metadata Overrides

Syntax: _MetaData:Index has a leading underscore, while MetaData:Host, MetaData:Sourcetype, and MetaData:Source do not. transforms.conf.spec says key names are case-sensitive and must be used exactly as listed, and that the host, source, and source type values must carry their host::, source::, or sourcetype:: prefixes.

Practical Example: Syslog Host Extraction and Sourcetype Reassignment

Consider a network stream arriving on TCP 514 containing Cisco ASA and Palo Alto firewall logs:

# props.conf
[syslog:network:collector]
TRANSFORMS-01_host = extract_syslog_origin_host
TRANSFORMS-02_type = classify_cisco_asa, classify_palo_alto

# transforms.conf
[extract_syslog_origin_host]
REGEX = ^(?:<\d+>)?(?:[A-Z][a-z]{2}\s+\d+\s+[\d:]+)\s+([a-zA-Z0-9\.\-_]+)\s+
DEST_KEY = MetaData:Host
FORMAT = host::$1

[classify_cisco_asa]
REGEX = %ASA-\d-\d+:
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::cisco:asa

[classify_palo_alto]
REGEX = ,TRAFFIC,|,THREAT,|,SYSTEM,
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::pan:traffic

---

Troubleshooting Transforms & Avoiding Race Conditions

Transform configurations operate at index time, meaning errors affect data permanently. Administrators must be skilled at debugging pipeline routing.

1. Validating Configuration Merging with btool

To verify that props.conf and transforms.conf stanzas have been parsed correctly without typos or stanza overrides from other apps:

# Inspect merged transforms definitions and app origin
$SPLUNK_HOME/bin/splunk btool transforms list --debug | grep -A 7 "\[discard_kube_probes\]"

# Inspect merged props mappings
$SPLUNK_HOME/bin/splunk btool props list --debug | grep -A 4 "\[kube:ingress:access\]"

2. Checking splunkd.log and the Results

A regex that fails to compile, or a transform with missing settings, produces warnings or errors in splunkd.log on the instance that parses the data (heavy forwarder or indexer), not on the search head:

index=_internal sourcetype=splunkd (log_level=WARN OR log_level=ERROR) host=<parsing instance>

Then verify the effect with searches. Check that dropped events stop appearing, that routed events land in the new index (index=pci_secure), and that the license usage report reflects the reduction.

3. Pipeline Race Conditions and Execution Ordering

When configuring multiple transforms that alter routing and metadata, execution ordering is critical:

• Rule order: transforms in a chain all run, and a later transform that sets the same key overwrites an earlier one. In a keep pattern, the catch-all nullQueue rule must come before the keep rule.
• Mutual Exclusivity: When multiple rules set DEST_KEY = _MetaData:Index, the last rule executed wins. Ensure regex patterns are mutually exclusive, or use strictly ordered class prefixes (01_, 02_) so that broad general routing precedes specific override rules.
• nullQueue is decided at the end: setting queue = nullQueue does not immediately remove the event. Later transforms still see it and can set indexQueue again. Only the final value when the typing pipeline finishes decides whether the event is dropped.