12.2 Monitor Input Tuning: Whitelists, Blacklists & CRC Settings

Monitor Input Tuning: Whitelists, Blacklists & CRC Settings

Quick Summary: In large-scale enterprise environments, pointing a monitor input at a busy directory tree without precise tuning can saturate indexing pipelines with irrelevant files or cause silent data loss due to hash collisions. Administrators control file selection using Perl-Compatible Regular Expressions (PCRE) with whitelist and blacklist directives, where blacklist rules strictly supersede whitelists. Furthermore, understanding the 256-byte Cyclic Redundancy Check (CRC) mechanism—and using crcSalt or initCrcLength to eliminate identical header collisions—is critical to ensuring data integrity.

---

Filtering Files with Regular Expressions: Whitelists & Blacklists

When a monitor input stanza targets a directory (e.g., [monitor:///var/log]), Splunk attempts to ingest every readable file discovered within that directory structure. To include or exclude specific subsets of files, administrators configure whitelist and blacklist settings in inputs.conf.

Regex Evaluation Rules and Syntax

Both whitelist and blacklist accept Perl-Compatible Regular Expressions (PCRE). A common administrative misconception is that these regular expressions match only the file's base name (e.g., access.log). In reality, Splunk evaluates the regular expression against the entire fully qualified absolute file path.

# Example inputs.conf file filter configuration
[monitor:///var/log/httpd]
disabled = false
index = web_prod
sourcetype = access_combined
# Whitelist only .log files
whitelist = \.log$
# Blacklist rotated, compressed, or temporary files
blacklist = (\.gz$|\.bak$|\.tmp$|/archive/)

Combining Several Patterns

A monitor stanza has one whitelist and one blacklist setting. (Numbered whitelist1–whitelist9 settings exist only for Windows event log inputs.) To combine alternatives, use regex alternation:

[monitor:///opt/data/logs]
whitelist = \.(log|json)$
blacklist = (\.archive\.log$|/temp/|\.gz$)

Do not add comments at the end of a setting line, because the text becomes part of the regex.

The Fundamental Precedence Rule: Blacklist ALWAYS Supersedes Whitelist

When both whitelist and blacklist rules are applied within a monitor stanza, Splunk adheres to a strict, non-negotiable evaluation hierarchy:

$\text{Ingestion Decision} = (\text{Matches Whitelist}) \land \neg (\text{Matches Blacklist})$

If a file matches a whitelist rule but also matches a blacklist rule, the file is unconditionally blacklisted and dropped. Blacklist evaluation takes absolute precedence over whitelist evaluation under all circumstances.

Common Regex Traps and Escaping

1. Unescaped Dots: In regular expressions, a bare dot (.) matches any character. Writing whitelist = .log matches syslog, catalog, and dialog.txt. To match a literal file extension, the dot must be escaped: whitelist = \.log$.
2. Path Separator Escaping: On Windows systems where backslashes are present, backslashes must be escaped with double backslashes in regexes (\\), or forward slashes should be used in the path definition to avoid backslash escaping issues.

---

CRC Checking Mechanics & The Fishbucket

When Splunk discovers a file within a monitored directory, it must determine whether the file is brand new, a continuing stream from an existing file, or a duplicate of a file that has already been indexed. It achieves this using a Cyclic Redundancy Check (CRC) hash.

The 256-Byte Default Inspection Window

By default, when splunkd encounters a file, it reads the first 256 bytes and computes a beginning CRC. It then looks the CRC up in the fishbucket ($SPLUNK_DB/fishbucket/splunk_private_db):

• If the CRC does not exist: the file is new. Splunk reads it from the beginning and records CRCs and seek addresses as it goes.
• If the CRC exists and the data at the saved seekAddress still matches its seekCRC: the file was read before. Splunk seeks to the seekAddress and reads only data appended after it.
• If the CRC exists but the data at the seekAddress does not match: the file was modified in place, or it is a different file with the same opening bytes. Splunk cannot track both streams under one key, which is the collision problem described next.

+-------------------------------------------------------------------------+
|                            Monitored Log File                           |
| [ Byte 0 ........................................ Byte 255 ] [ Byte 256 ]|
| <--------- Initial 256 Bytes Hashed for CRC Check --------->   Payload   |
+-------------------------------------------------------------------------+

---

Identical Header Collisions & Resolution Strategies

While the 256-byte CRC mechanism works seamlessly for standard dynamic logs, it creates a severe vulnerability in enterprise applications that write standardized boilerplate headers: the Identical Header Collision.

Root Cause of the Collision Defect

Many commercial enterprise software packages, databases, and network appliances prepend a static metadata header to every log file they generate. Consider an enterprise middleware system that creates daily log files with the following initial header:

################################################################
# Application: Enterprise Financial Transaction Gateway        #
# Version: 14.2.0-RELEASE (Build 9481)                         #
# Copyright (C) 2026 Global Banking Technologies Inc.          #
# Format: Standard Common Log Format (UTF-8)                   #
################################################################

If this static banner exceeds 256 bytes, every daily log file created by this system—such as app-2026-09-22.log, app-2026-09-23.log, and app-2026-09-24.log—will have an identical first 256 bytes.

The Consequence

1. Splunk reads app-2026-09-22.log and stores its beginning CRC with a seekAddress and seekCRC.
2. The next day, app-2026-09-23.log is created. Its first 256 bytes are identical, so its beginning CRC matches the existing record.
3. Splunk's documentation explains that because the fishbucket is keyed to the beginning CRC, it cannot track progress independently for the two files. Splunk behaves as if it is dealing with one file that was modified.
4. Administrators see missing or duplicated data for the daily files, depending on the timing. The documented fix is further configuration: initCrcLength or crcSalt.

Resolution 1: crcSalt = <SOURCE>

The crcSalt directive forces Splunk to alter the data buffer fed into the CRC calculation function. Setting crcSalt = <SOURCE> directs Splunk to append the file's fully qualified source path to the 256-byte header before generating the hash:

$\text{Buffer} = [\text{First 256 Bytes of File}] + [\text{Full Filepath String}]$

[monitor:///var/log/financial_app/*.log]
index = finance
sourcetype = financial:txngateway
crcSalt = <SOURCE>

Because /var/log/financial_app/app-2026-09-22.log and /var/log/financial_app/app-2026-09-23.log have different filepath strings, their resulting CRC hashes are completely distinct, instantly eliminating the collision.

[!CAUTION] The crcSalt = <SOURCE> Rotation Hazard: Never configure crcSalt = <SOURCE> on log files that undergo move-and-recreate rotation (e.g., app.log rolling to app.log.1). When app.log is renamed to app.log.1, its filepath changes. Because the filepath changed, its CRC signature changes! Splunk fails to recognize app.log.1 as the rotated predecessor of app.log, treats it as a brand-new file, and re-indexes the entire historical file from byte 0, generating massive duplicate event storms.

Resolution 2: initCrcLength = <bytes>

When log files undergo rotation and share identical headers, crcSalt = <SOURCE> cannot be safely used. The correct architectural remedy is initCrcLength.

Instead of reading only 256 bytes, initCrcLength instructs Splunk to expand the CRC inspection window deeper into the file:

[monitor:///var/log/financial_app/*.log]
index = finance
sourcetype = financial:txngateway
# Expand the CRC window to 2048 bytes (past the static banner); allowed range 256-1048576
initCrcLength = 2048

By expanding the inspection window past the static banner into the variable payload area—where dynamic timestamps, thread IDs, or transaction records begin—each file generates a distinct CRC naturally. Because crcSalt is not used, the file can be safely renamed during log rotation without triggering re-indexing.

---

Aging Parameters: ignoreOlderThan

When onboarding a legacy server into Splunk, administrators frequently configure a monitor stanza targeting a directory containing years of historical log files (e.g., /var/log/). Ingesting gigabytes of stale historical logs can exhaust daily license volume and overwhelm indexing queues.

The ignoreOlderThan directive instructs Splunk to skip files whose last modification timestamp (mtime) is older than a specified duration threshold:

[monitor:///var/log/.../*.log]
index = linux_os
sourcetype = syslog
# Ignore any file not modified within the last 14 days
ignoreOlderThan = 14d

Supported Syntax and Operational Mechanics

• Time Span Formats: Accepts standard time units: s (seconds), m (minutes), h (hours), d (days). Examples: 48h, 7d, 30d.
• How it is evaluated: the monitor input compares each file's modification time with the current time. If the difference is greater than ignoreOlderThan, the file is put on an ignore list and is not read.
• No automatic reactivation: per inputs.conf.spec, files put on the ignore list are not checked again until Splunk restarts or the file monitoring subsystem is reconfigured, even if they become newer later. Choose a value that files you want to read can never reach, allowing for downtime. The spec suggests something like 14d.

---

Tailing from EOF: followTail = 1

With followTail = 1, monitoring starts at the end of a file, like tail -f. The input does not read any data that exists in the file when it is first encountered, only data that arrives afterward. The spec calls it an advanced setting, says not to leave it enabled in an ongoing fashion, and says not to use it for rolling log files or files whose names or paths vary.

[monitor:///var/log/debug_stream.log]
index = debug
sourcetype = app:debug
followTail = 1

The Critical Production Danger of followTail

followTail = 1 is easy to misuse because it silently skips data.

Initial Boot:  File discovered -> Seek to EOF -> Tail new writes (Expected)
Forwarder Restart:
               splunkd stops -> Application writes 500 KB to file while down
               splunkd starts -> File re-evaluated as newly discovered if fishbucket lost
               seekptr jumps to NEW EOF -> The 500 KB written during downtime is PERMANENTLY LOST!

Why followTail = 1 Must Never Be Left Permanently Active

1. Evaluation condition: followTail applies whenever a file is encountered for the first time, meaning it has no fishbucket record.
2. Failure on Fishbucket Rebuilds: If an administrator cleans the fishbucket (splunk clean eventdata -index _thefishbucket) or moves the forwarder to a new storage volume, all files become "newly discovered." With followTail = 1 permanently enabled, Splunk seeks to EOF on every log file on the host, permanently skipping all existing log history.
3. Failure on Forwarder Downtime: If a monitored file rotates while the forwarder is stopped, the newly rotated active file appears as a fresh file upon forwarder restart. followTail = 1 forces Splunk to seek to EOF, permanently dropping all log entries written between the rotation and the restart.

[!IMPORTANT] Operational Protocol for followTail: If followTail = 1 is used to bootstrap a massive log file without indexing historical backlog, follow this three-step procedure:

1. Add followTail = 1 to the monitor stanza and start Splunk.
2. Wait long enough for the input to identify the files (new events from them start arriving).
3. Immediately remove followTail = 1 (or set followTail = 0) in inputs.conf and restart Splunk (or remove it through Splunk Web or the CLI, which reconfigures the input). Future startups will then safely rely on the established fishbucket offset.