4.4 indexes.conf Options: Paths, Bucket Sizing & Volumes

indexes.conf Options: Paths, Bucket Sizing, and Volumes

All physical storage paths, bucket sizing thresholds, bucket roll triggers, and aging retention policies in Splunk Enterprise are governed by indexes.conf. Located on indexers and cluster peer nodes, indexes.conf controls how storage resources are allocated and ensures that disk partitions do not fill up unexpectedly.

Mastering the attributes inside indexes.conf is essential for balancing ingest throughput, search performance, hardware expenditure, and regulatory retention requirements.

Comprehensive Breakdown of indexes.conf Attributes

Every user-defined index is configured within its own stanza in indexes.conf (e.g., [web_traffic], [firewall]). In addition, global defaults can be established in the [default] stanza.

[web_traffic]
homePath = /opt/splunk/var/lib/splunk/web_traffic/db
coldPath = /mnt/cold_storage/web_traffic/colddb
thawedPath = /opt/splunk/var/lib/splunk/web_traffic/thaweddb
maxDataSize = auto_high_volume
maxWarmDBCount = 400
maxHotSpanSecs = 7776000
maxTotalDataSizeMB = 1000000
frozenTimePeriodInSecs = 15552000

Core Attribute Reference Table

Bucket Sizing: auto vs. auto_high_volume

The maxDataSize attribute directly dictates how large an individual hot bucket can grow before Splunk seals it and rolls it into warm storage. Configuring this setting appropriately is critical for indexing health:

maxDataSize = auto

• Sets the maximum bucket size to 750 MB.
• The default, and the right choice for most indexes.
• indexes.conf.spec recommends auto_high_volume instead for "high volume" indexes, which it describes as receiving more than about 10 GB of data per day.

maxDataSize = auto_high_volume

• Sets the maximum bucket size to 10 GB on 64-bit systems (1 GB on 32-bit systems).
• Recommended for high-volume enterprise ingestion tiers (e.g., firewall logs, netflow, Windows event security channels, endpoint telemetry).

Bucket Sizing Impact:
An indexer writing ~500 GB of bucket data per day into one index:
  maxDataSize = auto (750 MB)            --> roughly 650+ buckets per day
  maxDataSize = auto_high_volume (10 GB) --> roughly 50 buckets per day
Fewer, larger buckets mean fewer files to open and track per search.

[!TIP] If an indexer receives large volumes of data and generates thousands of small warm buckets under maxDataSize = auto, searches suffer from overhead because the indexers must open and check far more buckets and tsidx files for every time range. Switching to auto_high_volume dramatically reduces total bucket count and streamlines search execution.

Volume Definitions & Storage Pooling

Configuring static maxTotalDataSizeMB values independently on dozens of indexes can lead to inefficient storage utilization. If Index A is allocated 500 GB and uses 480 GB, while Index B is allocated 500 GB but only uses 50 GB, Index A will begin freezing data prematurely even though the underlying filesystem has hundreds of gigabytes of free disk space.

To solve this, Splunk introduces Storage Volumes via [volume:<volume_name>] stanzas.

How Volumes Work

A volume definition sets a global size cap on an entire physical mount point or disk array. Multiple indexes then reference this shared volume for their homePath and coldPath configurations.

# Volume definitions in indexes.conf

[volume:fast_storage]
path = /mnt/nvme_pool
maxVolumeDataSizeMB = 2000000       # 2 TB global ceiling for Hot/Warm

[volume:cold_storage]
path = /mnt/hdd_array
maxVolumeDataSizeMB = 10000000      # 10 TB global ceiling for Cold

# Index stanzas referencing shared volumes

[network]
homePath = volume:fast_storage/network/db
coldPath = volume:cold_storage/network/colddb
thawedPath = /opt/splunk/var/lib/splunk/network/thaweddb
maxTotalDataSizeMB = 5000000

[security]
homePath = volume:fast_storage/security/db
coldPath = volume:cold_storage/security/colddb
thawedPath = /opt/splunk/var/lib/splunk/security/thaweddb
maxTotalDataSizeMB = 5000000

[app_logs]
homePath = volume:fast_storage/app_logs/db
coldPath = volume:cold_storage/app_logs/colddb
thawedPath = /opt/splunk/var/lib/splunk/app_logs/thaweddb
maxTotalDataSizeMB = 5000000

Dynamic Volume Rolling Mechanics

When multiple indexes share a volume, Splunk monitors the aggregate capacity used across all member directories:

1. If volume:fast_storage exceeds maxVolumeDataSizeMB, Splunk looks at the buckets of every index that uses that volume (network, security, app_logs).
2. It removes the buckets with the oldest "latest time" (newest event) across the whole volume, moving them to cold, until the volume is under its limit. This trim can even move a bucket straight from hot to cold if it has the oldest latest time.
3. If volume:cold_storage exceeds its limit, the same rule freezes the cold buckets with the oldest latest time across all indexes on that volume.

[!IMPORTANT] When a volume cap is exceeded, which bucket moves is decided by data age across the entire volume, not by which index used the most space. Even if network wrote 80% of the volume's data, an old bucket from security will roll first if its events are chronologically older.