15.2 Character Set Encoding & Handling Binary Data

Character Set Encoding & Handling Binary Data

Quick Summary: Splunk stores and searches text as UTF-8. When a source writes another encoding, such as UTF-16LE from Windows tools or ISO-8859-1 from legacy systems, set CHARSET in props.conf so that Splunk converts it correctly. Splunk also skips files it classifies as binary unless NO_BINARY_CHECK = true. The key exam point: both settings apply at input time, where the data is first read. For monitored files, that is the forwarder with the input, including a universal forwarder.

CHARSET in props.conf

[<sourcetype>]            # or [source::<spec>]; not [host::<spec>]
CHARSET = <encoding>

Per props.conf.spec:

• Splunk assumes input for the stanza is in the named encoding and converts it to UTF-8.
• It can be set only in [<sourcetype>] or [source::<spec>] stanzas, not [host::<spec>].
• Valid names are those listed by iconv -l on most *nix systems.
• An invalid encoding name logs a warning during initial configuration, and further input from that stanza is discarded.
• Characters that are invalid in a valid encoding are escaped as hex, for example \xF3.
• CHARSET = AUTO makes Splunk try to detect the encoding automatically and convert it to UTF-8.
• Defaults: AUTO on Windows machines and UTF-8 everywhere else.
• Where it applies: at input time, when data is first read, such as on a forwarder that has configured inputs acquiring the data.

What Goes Wrong Without It

When non-UTF-8 bytes are read as UTF-8:

1. Accented and non-Latin characters appear as escaped hex or replacement characters.
2. Timestamps whose month or day names contain non-ASCII characters may not be recognized, so events get a fallback time.
3. Field extractions and searches for those values stop matching.
4. With UTF-16 in particular, every other byte of ordinary text is a null byte. The data looks like binary, and line-breaking patterns may not match as expected.

Common Encodings

A byte order mark (BOM) at the start of a file (EF BB BF for UTF-8, FF FE for UTF-16LE, FE FF for UTF-16BE) helps detection. However, network streams and many rotated files do not start with one. For known sources, name the encoding explicitly instead of relying on detection.

Where to Put CHARSET: Input Time

This differs from most parsing settings, such as LINE_BREAKER, TIME_FORMAT, and TRANSFORMS. Those run on the first full Splunk instance (a heavy forwarder or an indexer), not on a universal forwarder. CHARSET and NO_BINARY_CHECK belong to the small set of input-time props.conf settings. Many add-ons therefore install the same props.conf on forwarders and indexers.

Binary File Detection

Monitored directories often contain files that are not text, such as executables, core dumps, database files, and compressed files that are not handled as archives. Indexing them would waste license and disk space on meaningless terms. Splunk therefore examines each new file and ignores files it classifies as binary. props.conf.spec describes the default as NO_BINARY_CHECK = false (binary files are ignored).

A rejected file leaves a warning from the file classifier in splunkd.log, typically stating that the file is invalid with the reason binary. To find such messages:

index=_internal sourcetype=splunkd component=FileClassifierManager

The UTF-16 False Positive

In UTF-16LE, the text ABC is stored as 41 00 42 00 43 00. A file full of null bytes looks binary, so a UTF-16 log without a correct encoding declaration can be skipped entirely. The fix is to declare the encoding, and, if the file is still classified as binary, to allow it:

# props.conf on the universal forwarder that monitors the transcripts
[powershell:transcript]
CHARSET = UTF-16LE
NO_BINARY_CHECK = true

For UTF-16 data, the spec also advises detect_trailing_nulls = false, so that null bytes that are part of the text are not trimmed.

NO_BINARY_CHECK

• NO_BINARY_CHECK = true makes Splunk process binary files for that stanza.
• It can be set only in [<sourcetype>] or [source::<source>] stanzas, not [host::<host>].
• Default: false.
• Applies at input time, on the forwarder or instance that reads the data.

Splunk's own props.conf uses it for data that only looks binary. One example is [source::....Z(.\d+)?], which decompresses .Z files with unarchive_cmd and sets NO_BINARY_CHECK = true.

Legitimate Uses

1. UTF-16 or other multi-byte encodings that are otherwise classified as binary.
2. Text records with a small binary header or null padding.

Why Not Set It Broadly

Setting NO_BINARY_CHECK = true in [default] or on a broad [source::...] pattern removes the protection everywhere:

Best practice: set NO_BINARY_CHECK = true only in a specific [<sourcetype>] or narrow [source::...] stanza, together with the correct CHARSET, and deploy it where the input runs.