15.1 Input-Phase Processing & Default Metadata Assignment

Input-Phase Processing & Default Metadata Assignment

Quick Summary: Every event carries four default metadata fields: host, source, sourcetype, and index. They are assigned by the input that reads the data, on the forwarder or on the first full Splunk instance, before parsing starts. You control them in inputs.conf with static values, path-based host extraction (host_segment, host_regex), and connection_host for network inputs. Setting them explicitly, especially sourcetype, is what makes the later parsing rules in props.conf apply correctly.

Where Metadata Is Assigned in the Pipeline

Splunk's data pipeline starts with the input segment, followed by the parsing, merging, typing, and index pipelines (covered in detail in the staging chapter):

1. Input: the input reads the data (file, network port, script, Windows API), breaks it into blocks, and attaches host, source, sourcetype, and index from inputs.conf. Character encoding (CHARSET) is also applied here.
2. Parsing pipeline: line breaking (LINE_BREAKER) and header processing.
3. Merging pipeline: timestamp extraction and, when SHOULD_LINEMERGE = true, line merging.
4. Typing pipeline: regex-based TRANSFORMS and SEDCMD, such as masking, routing, and metadata rewrites.
5. Index pipeline: writing raw data and index files to buckets, or forwarding.

Why Input-Time Metadata Matters

• props.conf matching: parsing settings are found by matching the event's metadata against props.conf stanzas. When several stanzas match, [source::<source>] overrides [host::<host>], which overrides [<sourcetype>]. rule:: and delayedrule:: stanzas are only used to classify data that has no source type. If the input assigns the wrong source type, the right parsing rules never run.
• Routing and filtering: transforms that send events to other indexes, output groups, or the nullQueue usually key on host, source, or sourcetype.
• Index placement: the index setting decides where data lands and therefore which roles can search it and which retention settings apply. It defaults to the default index (main).

Host Assignment in inputs.conf

1. Static Host (host = <value>)

Every event from the stanza gets that host value. If a stanza does not set host, the value comes from the [default] stanza of inputs.conf, which Splunk sets to the machine's host name at installation.

[monitor:///opt/oracle/admin/diag/rdbms/prod/alert.log]
host = db-cluster-primary.corp.example
sourcetype = oracle:alert:log
index = databases

2. Host from a Path Segment (host_segment = <N>)

When a central server stores logs in per-host directories (for example, a syslog server writing /var/log/remote_hosts/<host>/...), host_segment uses the Nth /-separated segment of the path as the host.

Path: /var/log/remote_hosts/web-prod-01/nginx/access.log

[monitor:///var/log/remote_hosts/.../*.log]
host_segment = 4
sourcetype = nginx:access
index = web_logs

• If the value is not an integer or is less than 1, the default host setting is used.
• Windows: the drive letter and colon do not count as a segment. With host_segment = 3 and the path D:\logs\servers\host01, the host is host01.

3. Host from a Regular Expression (host_regex = <regex>)

When host names sit at varying depths or inside file names, host_regex applies a regex to the path and uses the first capturing group as the host.

[monitor:///mnt/storage/.../logs/*.log]
host_regex = /mnt/storage/(?:prod|dr)/([^/]+)/

[monitor:///var/log/syslog-*.log]
host_regex = syslog-[a-z]+-([a-zA-Z0-9-]+)\.corp\.log

• The regex is matched against the source value. If source is set explicitly in the stanza, that string is matched instead of the file name.
• If the regex does not match, the default host setting is used.

Which Setting Wins?

Per inputs.conf.spec: if host_regex and host_segment are both set, host_regex is ignored. If the chosen method does not produce a value, the stanza's host, or the [default] host, is used. A source type with a host-override transform (as with syslog data) can still replace host later in the typing pipeline.

4. Network Inputs (connection_host)

For [tcp://] and [udp://] inputs, the host usually comes from the sender:

For [splunktcp] inputs, the forwarder normally sends its own host value. connection_host is used only if the remote instance does not set a host. When data passes through a relay, the sender's address is the relay's, so extract the true host from the event (for example, the syslog header) with a host-override transform.

Source Type Assignment

The source type decides how Splunk breaks events, finds timestamps, and applies field extractions.

Explicit vs. Automatic

1. Explicit (sourcetype = <name>): you name the source type in the input. This is the best practice for every production input.
2. Automatic: if the input sets no source type, Splunk tries to recognize the data. It uses its pretrained source types and the rule:: and delayedrule:: stanzas in props.conf, and as a last resort it generates a new source type based on the source. Very small files can end up as too_small or <sourcename>-too_small.

Why Automatic Source Typing Is Avoided in Production

• Inconsistent names: similar files can be classified differently depending on their content, and learned names such as app-too_small or generated source types do not match what add-ons, dashboards, and data models expect.
• Generic parsing: a source type with no specific settings runs on the [default] values in props.conf, such as SHOULD_LINEMERGE = true and MAX_TIMESTAMP_LOOKAHEAD = 128. These defaults are slower and less accurate than a tuned source type with LINE_BREAKER, SHOULD_LINEMERGE = false, TIME_PREFIX, and TIME_FORMAT.
• Knowledge objects: event types, tags, field aliases, and CIM data models are usually written for specific source type names such as cisco:asa or pan:traffic.

Best practice: set sourcetype explicitly in every input stanza, preferably using the names that the relevant Splunkbase add-on expects.

Source Assignment and Overrides

By default, source is:

• Monitored files: the full path of the file, such as /var/log/apache2/access.log.
• Network inputs: the protocol and port, such as udp:514 or tcp:1514.
• Scripted and modular inputs: the script path or input name.

source = <name> in the stanza overrides it. This is rarely necessary for files, because the path is valuable information, but it can help when paths contain random container IDs. Remember that host_regex matches the overridden value.