16.2 Timestamp Extraction, Time Zones & Clock Skew Handling

Timestamp Extraction, Time Zones & Clock Skew Handling

Quick Summary: Every event's _time is stored as a UTC epoch value. Splunk finds the timestamp during the merging pipeline, using datetime.xml patterns unless you tell it exactly where and how to look with TIME_PREFIX, TIME_FORMAT, and MAX_TIMESTAMP_LOOKAHEAD. Know the time zone order from props.conf.spec (a zone in the event text, then TZ, then the forwarder's zone, then the indexer's zone), what DATETIME_CONFIG = NONE and CURRENT do, the fallback when no timestamp is found, and how MAX_DAYS_AGO and MAX_DAYS_HENCE handle implausible dates.

Timestamp Extraction in props.conf

Without specific settings, Splunk looks for a timestamp using the patterns in $SPLUNK_HOME/etc/datetime.xml (the default DATETIME_CONFIG). That works for many common formats, but it has costs:

• Performance: checking many patterns in every event is slower than parsing one known format.
• Wrong matches: numbers such as version strings, IDs, or IP-like values can be mistaken for dates, and the wrong date may be chosen when an event contains several.

Three settings make extraction precise and fast.

1. TIME_PREFIX = <regex>

• Splunk looks for a timestamp only in the text after the end of the first match of this regex.
• If TIME_PREFIX is not found in the event, timestamp extraction does not occur for that event, and the fallback rules below apply.
• Used together with TIME_FORMAT, the prefix must match up to and including the character just before the date.

# Timestamp right after an opening bracket
TIME_PREFIX = ^\[

# Timestamp inside a key-value pair
TIME_PREFIX = timestamp="

2. TIME_FORMAT = <strptime-style format>

• Gives the exact strptime layout of the timestamp, read starting right after TIME_PREFIX.
• For good results, the format should describe both the date and the time of day.

3. MAX_TIMESTAMP_LOOKAHEAD = <integer>

• How many characters into the event, counted from the TIME_PREFIX location, Splunk looks for the timestamp. If TIME_PREFIX lands 11 characters in and the lookahead is 10, extraction is limited to characters 11 through 20.
• Default: 128.
• 0 or -1 removes the limit, which the spec warns can hurt performance for long events.
• Set it to about the length of the timestamp, for example 30 for an ISO 8601 string with milliseconds and offset.

Example: A Tuned Source Type

[app:json:audit]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)\{"timestamp":
TIME_PREFIX = \{"timestamp":"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%z
MAX_TIMESTAMP_LOOKAHEAD = 30
TRUNCATE = 100000

Time Zone Resolution

Splunk stores _time in UTC and shows it in each user's time zone preference. To convert a timestamp to UTC, it decides the event's zone using the order defined in props.conf.spec for TZ:

1. A time zone in the event's raw text (for example UTC, -08:00, +0000)
        | if none
        v
2. The TZ setting in props.conf (for example TZ = America/Chicago)
        | if not set
        v
3. The time zone sent by the forwarder (6.0+ forwarding protocol)
        | if not forwarded
        v
4. The time zone of the system running splunkd (the parsing instance)

What This Means in Practice

1. TZ does not override an explicit zone in the event. If the event says +0000, that offset is used even when the stanza sets TZ = America/Chicago. TZ fixes events that carry no zone information.
2. Forwarded data defaults to the forwarder's zone. A universal forwarder in London sends its time zone with the data, so un-zoned timestamps from its files are interpreted as London time, even if the indexer is in New York.
3. Direct network inputs have no forwarder zone. A London router sending BSD syslog (for example Oct 14 04:12:01, with no year or zone) straight to a New York indexer is interpreted in the indexer's zone, and its events appear five hours off. Fix it with TZ in a [host::<router>] or source type stanza.
4. Ambiguous abbreviations: TZ_ALIAS controls how zone strings found in event text are interpreted. For example, EST maps to US Eastern by default; TZ_ALIAS = EST=GMT+10:00 changes that. It has no effect on the TZ setting.

DATETIME_CONFIG and Fallback Behavior

DATETIME_CONFIG

Both NONE and CURRENT disable timestamp identification in the text. BREAK_ONLY_BEFORE_DATE then cannot find event boundaries, so the spec pairs them with SHOULD_LINEMERGE = false or explicit BREAK_ONLY_* and MUST_BREAK_* rules. The spec notes that turning timestamp extraction off can speed up indexing.

When No Timestamp Is Found

If extraction runs but finds nothing usable, Splunk falls back in this order:

1. The timestamp of the previous event from the same source.
2. If no events in the source have a date, a date found in the source or file name (the events need a time of day).
3. For files, the file's modification time.
4. As a last resort, the current system time when the event is indexed.

Guardrails: MAX_DAYS_AGO and MAX_DAYS_HENCE

Both limits are measured from the current date as provided by the input layer, such as the forwarder's current time or a file's modification time.

MAX_DAYS_AGO

• Default: 2000 days (about 5.48 years). The highest legal value is 10951 (30 years).
• An extracted date older than this is not trusted. The event is still indexed, with the timestamp of the last acceptable event, or the current time if there is none.
• Backfilling old archives: when loading data older than 2000 days, increase MAX_DAYS_AGO for that source type first. Otherwise, those events are stamped with other events' times.

MAX_DAYS_HENCE

• Default: 2 days, which allows dates up to one day in the future. The highest legal value is 10950.
• A date further in the future gets the timestamp of the last acceptable event, or the current time if none exists.
• If servers have the wrong date, or are in a time zone a day ahead, the spec suggests increasing it to at least 3. A smaller window makes false positives less likely.
• Wildly wrong future dates are worth preventing, because they also stretch the time range of the bucket that holds them. That affects which searches must open the bucket and when time-based retention can freeze it.

Related: MAX_DIFF_SECS_AGO / MAX_DIFF_SECS_HENCE

These settings (defaults 3600 seconds and 604800 seconds) do not filter events. They make Splunk warn about timestamps that jump backward or forward compared with the previous event, and use a different format from most events in the source.