4.3 Checking Index Data Integrity

Data Integrity Control

Blueprint objective 4.3 asks you to check index data integrity. Splunk's tool for this is data integrity control: hashes recorded at index time and verified on demand.

Data Integrity Control Architecture

Regulated environments must be able to prove that log data has not been altered since it was indexed. Splunk Enterprise provides data integrity control, which records hashes of the raw data as it is indexed so that the data can be verified later.

Turning It On

Data integrity control is enabled per index in indexes.conf on the indexers. The default is false:

[security_audit]
homePath   = $SPLUNK_DB/security_audit/db
coldPath   = $SPLUNK_DB/security_audit/colddb
thawedPath = $SPLUNK_DB/security_audit/thaweddb
enableDataIntegrityControl = true

It applies only to local indexes. SmartStore (remote storage) indexes do not support it, and Splunk Cloud Platform does not use it.

How It Works

1. Slices: As raw data is indexed, the journal is written in slices. By default a slice, and therefore a hash, covers 128 KB of raw data (rawChunkSizeBytes = 131072).
2. Level-1 hashes: Splunk computes a SHA-256 hash of every slice of newly indexed raw data and writes it to an l1Hashes file.
3. Level-2 hash: When the bucket rolls from hot to warm, Splunk computes a hash of the contents of the l1Hashes file and stores it in l2Hash.
4. Location: Both hash files are stored in the bucket's rawdata directory.

journal (rawdata) ──► slice 1 ─► SHA-256 ┐
                     slice 2 ─► SHA-256 ├─► l1Hashes file ──(hot→warm roll)──► SHA-256 ─► l2Hash
                     slice n ─► SHA-256 ┘

Only the rawdata is hashed. The tsidx and metadata files are not covered, because they can be rebuilt from rawdata at any time.

Verifying Data Integrity via the CLI

Checks are run on demand with splunk check-integrity, either for one bucket or for a whole index:

# Verify one bucket
$SPLUNK_HOME/bin/splunk check-integrity -bucketPath $SPLUNK_DB/security_audit/db/db_1695427200_1695340800_12 -verbose

# Verify every bucket in an index
$SPLUNK_HOME/bin/splunk check-integrity -index security_audit -verbose

The command recomputes hashes from the journal and compares them with the stored hash files. A mismatch means the rawdata changed after it was indexed, whether through tampering or corruption.

Regenerating Lost Hash Files

If the hash files for a bucket are lost, Splunk can recreate them from the hashes stored in the journal:

$SPLUNK_HOME/bin/splunk generate-hash-files -index security_audit
$SPLUNK_HOME/bin/splunk generate-hash-files -bucketPath <bucket_path>

For stronger protection, Splunk suggests you can also copy hash files to storage outside the indexer, in separate directories to avoid name conflicts. An attacker who controls the indexer then cannot quietly change both the data and its hashes.

An Integrity Check Workflow

A typical audit request ("prove these security events have not been altered") works like this:

1. Confirm coverage. Check with splunk btool indexes list <index> --debug | grep enableDataIntegrityControl that the index had the setting enabled for the whole audit period. Buckets created before it was enabled carry no hashes.
2. Run the check on each indexer that holds the index: splunk check-integrity -index <index>. On an indexer cluster, run it on the peers, which hold the bucket copies.
3. Review the result for each bucket. A bucket whose recomputed hashes match l1Hashes/l2Hash passes. A mismatch identifies the bucket whose rawdata no longer matches what was indexed, and -verbose prints more detail.
4. Keep evidence. Save the command output with the audit record. If you copied hash files to separate storage, compare them as well, so that a change to both data and hashes on the indexer would still be detected.

What Integrity Control Does Not Cover

• Missing buckets: deleting a whole bucket leaves nothing to check. Track bucket inventories, and control who can reach the index directories.
• Frozen or archived data: once a bucket is frozen, its protection depends on how and where you archive it.
• Data before indexing: hashes describe the rawdata as it was written, after any masking with SEDCMD or transforms. They say nothing about the original source file.
• Index files: .tsidx files and metadata are not hashed. They can be rebuilt from rawdata.

Operational Notes

Integrity control also records hashes only for data indexed after you enable it, so turn it on before the audit period you need to defend.