2.2 License Violations, Rolling Window Warnings & Troubleshooting

License Violations, Rolling Window Warnings & Troubleshooting

In Splunk Enterprise, managing license compliance requires knowing exactly what data consumes daily quota, how overage warnings are calculated, the operational consequences of reaching violation thresholds, and how to rapidly isolate and remediate unexpected ingestion spikes.

---

Daily License Calculation: What Counts vs. What Is Exempt

Splunk Enterprise calculates license consumption based on the volume of data entering the indexing pipeline during each 24-hour cycle. Understanding the boundary between billable event data and exempt overhead is fundamental to capacity management.

The Calculation Standard: Raw Uncompressed Volume

Splunk measures daily license usage by calculating the uncompressed raw byte volume of incoming data at the moment it is indexed.

• Point of Ingestion: Metering happens when data is placed into the indexing pipeline on the instance that indexes it, so events filtered and dropped before indexing never count.
• Raw Event Sizing: Splunk calculates the byte size of the raw incoming character stream (the _raw field), including any delimiter or line-breaking characters added during the event-breaking phase.
• Index vs. Disk Footprint: License measurement is completely independent of index storage size on disk. Whether indexing overhead increases disk usage by 200% (due to index keys, bloom filters, and dictionary structures) or compression shrinks the raw text to 50% on disk, Splunk charges strictly for the raw, uncompressed event payload.
• Re-indexing Consideration: If an administrator re-indexes an existing data set—such as by resetting the fishbucket on a forwarder or re-reading an archived log file—all re-ingested bytes are counted against today's license quota as new data.

Data Accounting Breakdown

---

The Daily Reset & Rolling Warning Windows

The Midnight Reset

• Daily license volume is measured from midnight to midnight using the system clock on the license manager.
• An indexer in Tokyo or London reports to that manager every minute, but the license day it counts against is the manager's day. Keep every Splunk host on synchronized time (NTP).
• If Splunk Web shows a license warning message before midnight, you are already over quota for the day. The warning is recorded when the quota resets at midnight, which gives you time to add license volume before the warning counts.

What Generates a Warning

Splunk's documentation lists these warning conditions:

1. A license pool reaches its daily license volume limit.
2. A license stack reaches its daily license volume limit.
3. A license peer cannot communicate with the license manager.

---

Violation Thresholds & Operational Enforcement

Violation Thresholds by License Type

Exam trap: Older courseware and many practice sites still quote "5 warnings in 30 days" for every Enterprise license. Splunk's current administration manual gives 45 warnings in 60 days for Enterprise stacks under 100 GB/day and no search blocking for stacks of 100 GB/day or more. The 5-in-30 rule now applies to Trial, Dev/Test, and Developer licenses.

Operational Impact of a License Violation

THE GOLDEN RULE OF SPLUNK LICENSE VIOLATIONS:
Search is blocked; indexing and data ingestion continue.

1. Search Lockout: For affected stacks and pools, search is blocked, including scheduled reports and alerts. With a license manager, a message appears in the Search app and in the Global Messages bar telling users the license is in violation.
2. Indexing Never Stops: Indexers keep receiving and indexing data, so a violation causes no data loss. You simply cannot search that data until the violation clears.
3. Internal Searches Remain Permitted: Searching the internal indexes is not blocked, so you can use the Monitoring Console or searches against _internal to find the cause.

Lost Contact With the License Manager

A license peer that cannot reach the license manager for 72 hours or more is placed in violation and search is blocked on that peer until it reconnects. Indexing continues. Search _internal for LMTracker errors such as "failed to send rows" or "unable to connect".

---

Investigating and Troubleshooting Ingestion Spikes

When an unexpected data surge threatens a license violation, administrators must quickly identify the offending inputs, sourcetypes, or hosts.

The Key Log: license_usage.log

All licensing telemetry is recorded in $SPLUNK_HOME/var/log/splunk/license_usage.log, indexed automatically into _internal.

Key fields in license_usage.log:

• b: Number of raw bytes metered in the reporting event.
• s: Source identifier.
• st: Sourcetype of the indexed events.
• h: Originating host.
• idx: Target index.
• pool: Assigned license pool.
• i: GUID of the license peer that indexed the data.
• type: Usage for the per-minute usage records and RolloverSummary for the daily total written at midnight.

Diagnostic SPL Queries

1. Daily License Volume by Pool (Last 30 Days)

Use the midnight rollover summaries to see each pool's daily total:

index=_internal source=*license_usage.log type=RolloverSummary
| eval total_GB = round(b/1024/1024/1024, 2)
| timechart span=1d sum(total_GB) by pool

2. Top License-Consuming Sourcetypes Today

Identify which data types are responsible for current consumption:

index=_internal source=*license_usage.log type=Usage
| stats sum(b) as total_bytes by st
| eval total_GB = round(total_bytes/1024/1024/1024, 2)
| sort - total_GB
| head 10

3. Pinpointing High-Volume Hosts and Sources for a Specific Sourcetype

When an anomalous sourcetype (e.g., syslog or cisco:asa) spikes, isolate the individual host and file path:

index=_internal source=*license_usage.log type=Usage st="cisco:asa"
| stats sum(b) as total_bytes by h, s
| eval total_MB = round(total_bytes/1024/1024, 2)
| sort - total_MB
| head 20

Common Causes of License Overages

1. Application Debug Logging: An application team accidentally enables DEBUG or TRACE level logging in production, generating millions of verbose log lines per hour.
2. Syslog and Network Echo Loops: A misconfigured syslog forwarder, load balancer, or network switch creates a packet reflection loop, repeatedly resending identical messages.
3. Recursive Forwarder Directory Monitors: A wildcard monitor in inputs.conf (e.g., [monitor:///var/log/...]) traverses into an active backup or archive directory, indexing compressed .tar.gz or .zip files as binary text.
4. crcSalt = <SOURCE> on Rotating Logs: Adding crcSalt = <SOURCE> to a monitor input whose files are renamed by log rotation makes every renamed file look new, so Splunk indexes the rotated copies again. Splunk's documentation warns against using crcSalt = <SOURCE> with rotated logs.

Remediation Strategies

• Reset License: For an Enterprise stack under 100 GB/day that has lost search, request a reset license from Splunk Sales. Dev/Test customers use the Dev/Test request form, and Developer license holders email Splunk developer support. Trial and Free licenses have no reset license.
• License Pool Adjustments: If one pool is collecting warnings because of organizational growth, adjust pool allocations under Settings > Licensing to shift unused capacity from development pools to production pools.
• Pre-Indexing Ingestion Filters: Configure routing and filtering rules on Heavy Forwarders or Indexers using props.conf and transforms.conf to discard repetitive or useless events (DEST_KEY = queue, FORMAT = nullQueue) before they enter the indexing pipeline and consume quota.
• Stack Expansion: If genuine business data growth consistently exceeds capacity, install additional stacking Enterprise license keys to permanently expand the Enterprise stack.