4.2 Bucket Types & the Hot-Warm-Cold-Frozen-Thawed Lifecycle

Bucket Types and the Bucket Lifecycle

An index's buckets move through five states, hot, warm, cold, frozen, and thawed, under rules set in indexes.conf. Knowing which setting moves a bucket from one state to the next is one of the most heavily tested index topics.

Bucket Lifecycle Progression

Data moves deterministically through lifecycle tiers based on administrative rules, disk thresholds, and temporal age.

[Ingestion Pipeline]
         │
         ▼
   ┌───────────┐
   │    HOT    │  ◄── Actively written (Read/Write) in homePath
   └─────┬─────┘
         │  (Trigger: maxDataSize, maxHotSpanSecs, restart, etc.)
         ▼
   ┌───────────┐
   │   WARM    │  ◄── Closed, merged .tsidx (Read-Only) in homePath
   └─────┬─────┘
         │  (Trigger: maxWarmDBCount exceeded)
         ▼
   ┌───────────┐
   │   COLD    │  ◄── Migrated to coldPath (Read-Only on cheaper storage)
   └─────┬─────┘
         │  (Trigger: frozenTimePeriodInSecs OR maxTotalDataSizeMB hit)
         ▼
   ┌───────────┐      Archive configured? (coldToFrozenDir / coldToFrozenScript)
   │  FROZEN   ├───────────────────────────────┬───────────────────────────────┐
   └───────────┘                               │                               │
                                         [YES] ▼                         [NO]  ▼
                                   ┌───────────────┐             ┌───────────────────┐
                                   │ Archived to   │             │ Permanently       │
                                   │ External Store│             │ Deleted from Disk │
                                   └───────┬───────┘             └───────────────────┘
                                           │ (Manual Admin Restore)
                                           ▼
                                   ┌───────────────┐
                                   │    THAWED     │  ◄── Restored to thawedPath; rebuilt via
                                   └───────────────┘      'splunk rebuild'; exempt from aging

1. Hot Tier Mechanics

When events emerge from the indexing pipeline, the index processor writes them into an active hot bucket located in homePath. Hot buckets are named using internal identifiers: hot_v1_<sub_id> (such as hot_v1_0).

• Hot buckets are the only buckets that permit write operations.
• Hot buckets are fully searchable. Splunk handles concurrency so that search head queries can read data from hot buckets simultaneously while new events are being appended.

2. Warm Tier Mechanics

When a hot bucket reaches an operational limit, Splunk closes the bucket and rolls it to warm. During this roll:

1. Splunk flushes all in-memory buffers to disk.
2. The bucket is renamed from hot_v1_<sub_id> to standard warm naming syntax: db_<latest_time>_<earliest_time>_<localid>
3. Small tsidx files continue to be merged by the optimize process.
4. No further events are written to the bucket.
5. The warm bucket remains in homePath.

[!NOTE] Notice the timestamp sequence in the bucket folder name: db_<latest_time>_<earliest_time>_<localid>. The latest epoch timestamp appears first, followed by the earliest epoch timestamp, followed by a monotonically increasing local bucket ID (e.g., db_1695427200_1695340800_12). In an indexer cluster, the originating copy appends the source peer's GUID (db_<newest>_<oldest>_<localid>_<guid>), and replicated copies use the rb_ prefix (rb_<newest>_<oldest>_<localid>_<guid>).

3. Cold Tier Mechanics

As new warm buckets roll from hot, the number of warm buckets in homePath grows. When the count exceeds maxWarmDBCount (default 300), or homePath exceeds homePath.maxDataSizeMB if you set one, Splunk rolls the oldest warm bucket to coldPath ($SPLUNK_DB/<index_name>/colddb).

• Cold buckets maintain the exact same internal directory structure and filename as warm buckets.
• Cold buckets remain fully searchable.
• By separating coldPath from homePath, organizations can retain historical compliance logs on dense, lower-cost storage without degrading performance on the high-speed NVMe storage hosting hot and warm data.

4. Frozen Tier Mechanics

Buckets freeze when every event in the bucket is older than frozenTimePeriodInSecs, or when the index exceeds its size limit (maxTotalDataSizeMB, which freezes the oldest data first). A frozen bucket is no longer searchable.

Splunk handles frozen buckets in one of two ways:

• Archived: If coldToFrozenDir is set, Splunk copies the bucket there, keeping only its rawdata. If coldToFrozenScript is set, Splunk runs your script to archive it.
• Deleted: If neither archiving attribute is defined, Splunk permanently erases the bucket files from disk.

5. Thawed Tier Mechanics

Thawed data represents historical events that have been restored from frozen archives to satisfy legal audits, forensic inquiries, or compliance reviews.

• Restored buckets are placed in thawedPath ($SPLUNK_DB/<index_name>/thaweddb).
• Crucial Rule: Thawed buckets are completely exempt from automated aging rules. They will never roll to frozen or be automatically purged by maxTotalDataSizeMB or frozenTimePeriodInSecs. The administrator must manually delete thawed buckets once search investigations conclude.

Hot Bucket Roll Triggers

A hot bucket does not stay open forever. These conditions roll it to warm:

# Hot roll settings in indexes.conf (defaults shown)
maxDataSize    = auto      # 750 MB (auto_high_volume = 10 GB on 64-bit)
maxHotSpanSecs = 7776000   # 90 days of event time in one hot bucket
maxHotBuckets  = auto      # 3 hot buckets per ingestion pipeline
maxHotIdleSecs = 0         # 0 = no idle timeout

1. maxDataSize Reached

When the size of a hot bucket's rawdata journal and tsidx files exceeds maxDataSize, the bucket rolls to warm. Default settings include:

• auto: Sets the maximum size to 750 MB.
• auto_high_volume: Sets the maximum size to 10,000 MB (10 GB) on 64-bit operating systems (750 MB on 32-bit systems). Recommended for indexes receiving high daily data volume to prevent bucket proliferation.
• Integer value: Can be explicitly set in megabytes (e.g., maxDataSize = 2048).

2. maxHotSpanSecs Exceeded

Specifies the maximum time range (in seconds) between the earliest and latest event allowed inside an individual hot bucket. The default value is 7776000 seconds (90 days).

• If a hot bucket already contains an event from January 1, and an incoming event arrives timestamped April 5, the time span between the events exceeds 90 days.
• Splunk immediately rolls the hot bucket to warm and creates a new hot bucket for the incoming event, preventing buckets from spanning excessive temporal ranges.

3. maxHotBuckets Exceeded

Defines the maximum number of hot buckets per index. The default auto means 3, applied per ingestion pipeline, so an indexer with N pipeline sets can have up to N × 3 hot buckets for one index.

• If an incoming event carries a timestamp that does not fit into any open hot bucket, Splunk must open a new hot bucket.
• If opening a new hot bucket would exceed maxHotBuckets, Splunk rolls the hot bucket that contains the least recent data to warm.
• Separately, maxHotIdleSecs (default 0, disabled) rolls a hot bucket that has received no data for that many seconds.

4. Indexer Daemon Restart or Shutdown

Whenever the splunkd service is stopped or restarted (e.g., splunk restart, system reboot, or software upgrade), Splunk cleanly flushes all in-memory buffers to disk and immediately rolls all currently open hot buckets to warm.

• Upon restarting, Splunk initializes brand new hot buckets (hot_v1_0) for all active indexes.
• Frequent indexer restarts create large numbers of tiny warm buckets, which degrades search performance and wastes inode capacity.

5. Time Clock Skew & Out-of-Order Timestamps

If an endpoint or forwarder sends events with badly skewed timestamps (e.g., a batch of historical logs timestamped two years in the past, or an improperly configured device reporting events in the future), those events may not fit the time span of any open hot bucket (maxHotSpanSecs). Splunk opens another hot bucket, which can push an older one out under maxHotBuckets. Badly skewed timestamps therefore cause many small buckets. Values below 3600 for maxHotSpanSecs are reset to 3600.

Restoring Frozen Data & The Thawing Process

When legal or compliance work needs data that has already aged out to a frozen archive, the administrator restores it by thawing the archived bucket.

What an Archived Bucket Contains

When coldToFrozenDir is set, Splunk archives a frozen bucket by keeping only the rawdata directory, because rawdata contains everything needed to rebuild the bucket. (A custom coldToFrozenScript should do the same unless the index stubs out rawdata, which is a metrics-index special case.) Dropping the index files keeps archives small, because they are the larger share of a bucket.

Step-by-Step Thawing Procedure

Step 1: Copy the Archived Bucket into thaweddb

Copy the whole archived bucket directory into the index's thawedPath (by default $SPLUNK_DB/<index>/thaweddb). Make sure its bucket ID does not collide with a bucket already in the index:

cp -r /mnt/cold_archive/security/db_1680000000_1670000000_50 \
      $SPLUNK_HOME/var/lib/splunk/security/thaweddb/

Step 2: Rebuild the Index Files with splunk rebuild

An archived bucket has only rawdata, so it is not searchable yet. splunk rebuild reads the journal and regenerates the tsidx and metadata files:

$SPLUNK_HOME/bin/splunk rebuild \
   $SPLUNK_HOME/var/lib/splunk/security/thaweddb/db_1680000000_1670000000_50

If the command reports an error, rerun it with the index name as an extra argument (splunk rebuild <bucket_dir> <index_name>). Rebuilding decompresses and re-indexes the journal, so it uses significant CPU and disk I/O.

Step 3: Restart the Indexer

Splunk's documented procedure ends with splunk restart, after which the thawed bucket is searchable.

Step 4: Clean Up When Finished

Thawed buckets are outside the aging scheme. They never roll to frozen and do not count toward maxTotalDataSizeMB, so they stay until an administrator deletes them from thaweddb. Restored data also does not count against your license.