4.6 Applying a Data Retention Policy

Applying a Data Retention Policy

Blueprint objective 4.6 asks you to apply a data retention policy. In Splunk that means deciding how long each index keeps searchable data, how large it may grow, and what happens to buckets when they freeze. It is all configured in indexes.conf on the indexers.

The Retention Toolkit

A frozen bucket is deleted by default. Retention policy is really two decisions: when data freezes, and whether frozen data is archived.

Retention Policy Mechanics: Dual-Trigger Aging

One of the most critical principles tested in Splunk administration is the relationship between size-based aging and time-based aging.

Data aging in Splunk follows a dual-trigger, race-condition model:

              ┌─────────────────────────────────────────────────┐
              │            Active Index Storage                 │
              │         (Hot + Warm + Cold Buckets)             │
              └───────────────────────┬─────────────────────────┘
                                      │
           ┌──────────────────────────┴──────────────────────────┐
           ▼                                                     ▼
┌──────────────────────────────────────┐     ┌──────────────────────────────────────┐
│         Size-Based Trigger           │     │          Time-Based Trigger          │
│         maxTotalDataSizeMB           │     │        frozenTimePeriodInSecs        │
│ (Total index size exceeds threshold) │     │ (Latest event older than max age)    │
└──────────────────┬───────────────────┘     └───────────────────┬──────────────────┘
                   │                                             │
                   └──────────────────────┬──────────────────────┘
                                          │
                         WHICHEVER THRESHOLD IS REACHED FIRST
                                          │
                                          ▼
                             ┌─────────────────────────┐
                             │  Bucket Rolls to Frozen │
                             └─────────────────────────┘

The Dual-Trigger Rule

Data rolls to frozen whenever either of the following conditions occurs first:

1. Size Limit Exceeded: The cumulative size of all hot, warm, and cold buckets exceeds maxTotalDataSizeMB.
2. Time Limit Exceeded: The latest event timestamp inside a bucket is older than now - frozenTimePeriodInSecs.

Real-World Retention Dilemma: The Premature Freeze

Consider an enterprise compliance policy that legally mandates keeping firewall logs for 365 days (1 year).

An administrator configures indexes.conf as follows:

[firewall_logs]
homePath = /opt/splunk/var/lib/splunk/firewall/db
coldPath = /mnt/storage/firewall/colddb
thawedPath = /opt/splunk/var/lib/splunk/firewall/thaweddb
frozenTimePeriodInSecs = 31536000     # 365 days (1 year)
maxTotalDataSizeMB = 500000           # 500 GB

What happens in production?

• The organization ingests 50 GB of firewall logs per day.
• At roughly 50% of raw volume on disk (rawdata plus index files), 50 GB of daily logs needs ~25 GB of disk space per day.
• In only 20 days, total index data reaches 500 GB (20 days * 25 GB/day = 500 GB).
• Because maxTotalDataSizeMB has been reached, Splunk immediately freezes the oldest bucket.
• Result: Historical data is frozen and removed after only 20 days, violating the 365-day compliance mandate!

[!CAUTION] Setting frozenTimePeriodInSecs does not guarantee data will be retained for that duration. If maxTotalDataSizeMB (or a volume quota) is set too low, data will freeze prematurely to protect disk capacity. Administrators must calculate storage capacity accurately to ensure size limits exceed temporal requirements.

Archiving Instead of Deleting

• coldToFrozenDir = <path>: Splunk automatically copies frozen buckets to this directory, keeping only the rawdata. It cannot use a volume reference, and changing it requires a restart. If both coldToFrozenDir and coldToFrozenScript are set, coldToFrozenDir takes precedence.
• coldToFrozenScript = <script>: Runs a script you supply, which must live in $SPLUNK_HOME/bin or a subdirectory, for custom archiving such as copying to object storage.
• Archived data can later be thawed (Chapter 4's bucket lifecycle section): copy the bucket into thaweddb, run splunk rebuild, and restart.

Worked Retention Design

A security team must keep firewall data searchable for one year and archive it for seven:

[firewall]
homePath   = volume:hot_warm/firewall/db
coldPath   = volume:cold/firewall/colddb
thawedPath = $SPLUNK_DB/firewall/thaweddb
frozenTimePeriodInSecs = 31536000      # 365 days searchable
maxTotalDataSizeMB     = 20000000      # sized so the time limit, not size, drives freezing
coldToFrozenDir        = /mnt/archive/firewall   # keep rawdata for the 7-year archive

Check it after deployment with splunk btool indexes list firewall --debug. The Monitoring Console's index detail dashboards show whether buckets are freezing by age or by size.

Storage Sizing Calculations & Capacity Planning

Designing a Splunk indexing architecture requires calculating disk capacity requirements based on daily ingestion volume, indexing overhead, retention windows, and cluster replication.

Storage Calculation Formula

Splunk's rule of thumb is that compressed rawdata is about 15% of raw volume and the index files are about 35%. In an indexer cluster, every one of the RF copies stores rawdata, but only the SF (searchable) copies also store index files:

$\text{Cluster disk per day} = \text{Raw GB/day} \times (0.15 \times RF + 0.35 \times SF)$

1. Daily Ingest: Raw, uncompressed volume per day (the same number your license measures).
2. Rawdata share (~15%): Kept by every replicated copy.
3. Index-file share (~35%): Kept only by searchable copies. A non-clustered indexer is effectively RF = SF = 1, which gives the familiar "about 50% of raw" rule.
4. Retention: Days the data must stay searchable (hot + warm + cold).

Concrete Storage Planning Scenario

An enterprise architecture team is designing an indexing cluster with the following requirements:

• Daily Ingestion: 1,000 GB/day (1 TB/day)

• Retention Requirement: 90 days of searchable active data

• Tiering Strategy: 14 days in Hot/Warm (NVMe) + 76 days in Cold (HDD)

• Cluster Replication Factor: RF = 3

• Search Factor: SF = 2

Step 1: Daily cluster-wide disk
  1,000 GB x (0.15 x 3 + 0.35 x 2) = 1,000 x (0.45 + 0.70) = 1,150 GB/day

Step 2: Hot/Warm tier (14 days)
  14 x 1,150 GB = 16,100 GB (~16.1 TB) across the cluster

Step 3: Cold tier (76 days)
  76 x 1,150 GB = 87,400 GB (~87.4 TB) across the cluster

Step 4: Total
  16.1 TB + 87.4 TB = ~103.5 TB, before headroom

Storage Planning Best Practices

• Filesystem Headroom: Always allocate an additional 15–20% of disk space beyond calculated values to accommodate index optimization temporary files, operating system logs, and search dispatch artifacts.
• IOPS Allocation: Indexer storage for hot and warm buckets needs fast random I/O. Splunk's reference hardware guidance for indexers calls for storage able to sustain roughly 800 IOPS or more. Cold storage mostly serves historical searches and can be slower.