4.5 The Fishbucket: File Tracking, CRC Collisions & Re-reading Files

The Fishbucket: How Splunk Tracks Monitored Files

The Fishbucket (_thefishbucket)

When Splunk Enterprise or a forwarder monitors files with [monitor://...] inputs, it has to remember how far it has read in each file, so that a restart does not re-index everything or skip data. That memory is the fishbucket.

Where the Fishbucket Lives

• On disk: $SPLUNK_DB/fishbucket/splunk_private_db (that is, $SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db with default paths).
• As an index: it appears as the internal index _thefishbucket.
• In Splunk Cloud Platform you have no file-system access to it. There you check file-read status through REST (/services/admin/inputstatus/TailingProcessor:FileStatus).

What a Fishbucket Record Holds

Splunk's documentation describes the record in terms of CRCs and a seek address, not file names or inodes:

1. Beginning CRC: a CRC computed over the first 256 bytes of the file (initCrcLength). This is the key the fishbucket is looked up by.
2. seekAddress: how many bytes into the file Splunk has already read.
3. seekCRC: a fingerprint of the data at the seek address, used to confirm the file is still the same content.

Monitored file: /var/log/app.log
┌──────────────────────────────┬──────────────────────────────┐
│ first 256 bytes → begin CRC  │ ... data read so far ...     │ ← seekAddress (+ seekCRC)
└──────────────────────────────┴──────────────────────────────┘
                     stored in the fishbucket, keyed by the begin CRC

How Splunk Uses the Record

When the monitor processor finds a file, it hashes the first 256 bytes and looks up that beginning CRC:

1. No match → new file: Splunk reads from the start and records new CRCs and seek addresses as it goes.
2. Match, and the data at the seek address still matches the seekCRC, and the file is larger: the file was read before and has grown. Splunk seeks to the seekAddress and reads only the new data.
3. Match, but the data at the seek address does not match: either the file was modified in place, or it is a different file that starts with the same content. Because the fishbucket is keyed by the beginning CRC, Splunk cannot track the two streams independently without further configuration.

This is also how Splunk handles log rotation. When app.log is renamed to app.log.1, the renamed file has the same beginning CRC and an already-read seek address, so Splunk does not read it again. The newly created app.log has different opening content, so it is treated as a new file. (Compressed .gz and .tar archives are the documented exception: renamed archives are processed.)

The CRC Collision Problem & crcSalt

Because the fishbucket is keyed by a CRC of only the first 256 bytes, files that begin with identical content collide. This is a CRC collision.

Why CRC Collisions Occur

Many enterprise applications and network devices generate log files that begin with identical, static header comments or boilerplate text. For example:

# Logfile created by Enterprise Service Broker v4.2
# Host: production-node
# Format: W3C Extended Log File Format
# Date: 2026-09-23
# Fields: date time c-ip cs-username cs-method cs-uri-stem sc-status ...

If multiple log files in a monitored directory begin with this exact same 256-byte header:

1. Splunk reads File A (web_node1.log), calculates its 256-byte CRC, and indexes it completely.
2. Splunk examines File B (web_node2.log). It calculates the CRC of the first 256 bytes.
3. The CRC of File B is identical to File A!
4. Splunk finds the record for File A and concludes it has already read this content, so File B is skipped or only partly read.
5. Result: Critical log files are silently skipped and never ingested into Splunk.

The Solution: crcSalt = <SOURCE>

To prevent CRC collisions, administrators configure the crcSalt attribute inside the monitor stanza in inputs.conf:

[monitor:///var/log/apps/*.log]
index = web_app
sourcetype = app_access
crcSalt = <SOURCE>

When crcSalt = <SOURCE> is defined, Splunk appends the full filesystem path of the source file to the 256-byte buffer before calculating the CRC.

• Because /var/log/apps/web_node1.log and /var/log/apps/web_node2.log have different file paths, their resulting CRCs are completely different, even if their initial 256 bytes are identical.
• This guarantees unique identification for every file.

[!TIP] Alternatively, set initCrcLength (256 to 1048576 bytes) in inputs.conf so the beginning CRC covers more than the shared header. Unlike crcSalt = <SOURCE>, this keeps rotation detection working, because the CRC still depends only on file content.

[!CAUTION] Splunk warns not to use crcSalt = <SOURCE> on files that the operating system rotates or renames into another monitored path. The renamed file gets a different CRC, looks new, and is indexed again.

Inspecting & Resetting the Fishbucket

Sometimes you need to know whether Splunk has read a file, or force a single file to be read again.

Checking File Status

• Use the tailing processor's file status endpoint on the instance that monitors the file:

curl -k -u admin:<password> https://localhost:8089/services/admin/inputstatus/TailingProcessor:FileStatus

• Or run splunk list inputstatus from the CLI to see the files the tailing processor knows about, how far each has been read, and why any file is being ignored.

Re-reading One File with btprobe

btprobe works directly on the fishbucket database. Run it through splunk cmd so the Splunk environment is set, and reset the record for a single file:

$SPLUNK_HOME/bin/splunk cmd btprobe \
   -d $SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db \
   --file /var/log/secure --reset

After the reset, the next time the monitor processor evaluates /var/log/secure it finds no record and reads the file from the beginning. That file's data is indexed again, and the duplicate volume counts against the license. Do this with Splunk stopped, so the running tailing processor does not rewrite the record while you work.

The Complete Fishbucket Reset (The Nuclear Option)

When a development environment or test forwarder must be completely wiped to re-index all inputs from scratch, administrators can perform a full fishbucket reset using the splunk clean eventdata command:

# Step 1: Stop Splunk
$SPLUNK_HOME/bin/splunk stop

# Step 2: Clean the entire fishbucket index
$SPLUNK_HOME/bin/splunk clean eventdata -index _thefishbucket

# Step 3: Start Splunk
$SPLUNK_HOME/bin/splunk start

[!CAUTION] Running splunk clean eventdata -index _thefishbucket erases the fishbucket. Upon starting, Splunk forgets every file it has ever read. It will scan every file matching monitor stanzas in inputs.conf and re-index all of them from byte 0. In a production environment, this causes massive event duplication, license quota violations, and excessive storage consumption.