4.1 Index Structure: Directories, Buckets & Bucket Files

Index Structure: Directories, Buckets, and Files

In Splunk Enterprise, an index is not a monolithic relational database table or flat log repository. Instead, an index is a logical collection of discrete filesystem directories called buckets. Each bucket encapsulates a self-contained slice of time, containing both the compressed raw event text and the inverted search indexes that allow Splunk to query billions of events in seconds.

Understanding bucket architecture and lifecycle progression is essential for designing resilient storage tiers, managing disk capacity, meeting compliance retention mandates, and maintaining peak search performance.

Anatomy of an Index: Bucket Tiers & Storage Paths

When splunkd indexes incoming event streams, it organizes buckets into five sequential lifecycle tiers based on the age and write state of the data:

1. Hot Tier: Actively being written to by the indexing pipeline. Read/write.
2. Warm Tier: Rolled from hot; no longer receives new data. Read-only.
3. Cold Tier: Moved from warm to lower-cost, high-capacity storage. Read-only.
4. Frozen Tier: Aged out from cold storage; removed from search indexes. Either archived or permanently deleted.
5. Thawed Tier: Reconstructed from archived frozen data back into an active, searchable state.

$SPLUNK_DB/<index_name>/
├── db/                     <-- Specified by homePath (Hot & Warm Buckets)
│   ├── hot_v1_0/           <-- Active Hot bucket (read/write)
│   ├── hot_v1_1/           <-- Active Hot bucket (read/write)
│   ├── db_1695427200_1695340800_1/  <-- Warm bucket (read-only)
│   └── db_1695340799_1695254400_0/  <-- Warm bucket (read-only)
├── colddb/                 <-- Specified by coldPath (Cold Buckets)
│   ├── db_1695254399_1695168000_2/  <-- Cold bucket (read-only)
│   └── db_1695167999_1695081600_3/  <-- Cold bucket (read-only)
└── thaweddb/               <-- Specified by thawedPath (Restored Buckets)
    └── db_1694000000_1693000000_99/ <-- Thawed bucket (read-only, manual cleanup)

Bucket Tier Characteristics

[!IMPORTANT] Both hot and warm buckets reside in the exact same directory specified by the homePath parameter (by default, $SPLUNK_DB/<index_name>/db). Cold buckets reside in a separate directory path defined by coldPath (by default, $SPLUNK_DB/<index_name>/colddb), allowing administrators to place older, less frequently accessed data on less expensive, high-capacity storage tiers.

Deep Dive into Bucket Files

A bucket directory is a self-contained database unit. A warm or cold bucket on a standalone indexer contains files like these (exact names vary by version, and Splunk warns that bucket internals can change):

db_1695427200_1695340800_1/
├── rawdata/
│   ├── journal.zst          <-- Compressed raw events (zstd by default; gzip/lz4 are alternatives)
│   └── l1Hashes / l2Hash    <-- Present only when data integrity control is enabled
├── *.tsidx                  <-- Time-series index files (lexicon + postings)
├── bloomfilter              <-- Lets searches skip buckets that cannot contain a term
├── Hosts.data               <-- Hosts seen in the bucket, with counts and time ranges
├── Sources.data             <-- Sources seen in the bucket
├── SourceTypes.data         <-- Source types seen in the bucket
└── bucket_info.csv          <-- Bucket-level metadata

1. Rawdata Journal (rawdata/journal.zst)

The rawdata directory holds the compressed journal of every event indexed into the bucket, together with each event's index-time metadata (_time, host, source, sourcetype).

• journalCompression in indexes.conf defaults to zstd. It affects only new buckets, and Splunk can search buckets compressed with different algorithms side by side.
• The journal is the "source of truth". Everything else in the bucket can be regenerated from it with splunk rebuild, which is why archiving keeps only rawdata/.

2. Time-Series Index Files (.tsidx)

The .tsidx files are Splunk's inverted index. They contain the lexicon (every indexed term) and postings that point from each term to the events that contain it. A search first uses the tsidx files to find candidate events, so it avoids decompressing the whole journal.

• A hot bucket that is actively receiving data creates many small tsidx files.
• The splunk-optimize process merges those small files into fewer, larger ones as the bucket grows and when it rolls.

3. Metadata Files (*.data)

Hosts.data, Sources.data, and SourceTypes.data record which hosts, sources, and source types appear in the bucket, with counts and first/last times. The metadata search command reads them, which is why | metadata type=hosts index=web answers quickly without scanning events.

4. Bloom Filters and Other Files

• bloomfilter: A compact structure that lets a search rule a bucket out quickly. If the bloom filter says a term is not present, the bucket is skipped.
• bucket_info.csv: Bucket-level metadata used by the indexer.
• Integrity hash files: When enableDataIntegrityControl = true, Splunk writes l1Hashes and l2Hash files in the bucket's rawdata directory (see the data integrity section).

How Much Disk Does a Bucket Use?

Splunk's long-standing sizing rule of thumb is that compressed rawdata is about 15% of the original raw volume and index files are about 35%, so a searchable bucket copy is roughly 50% of the raw data it holds. The index files are the larger share, which is why archiving only rawdata/ saves so much space.