17.1 Transformation Mechanics: props.conf & transforms.conf

Transformation Mechanics: props.conf & transforms.conf

Quick Summary: Index-time transformations are split across two files. props.conf chooses which data is affected and invokes transforms with TRANSFORMS-<class>, and transforms.conf defines what each transform does (REGEX, FORMAT, DEST_KEY). They run in the typing pipeline, so they take effect on heavy forwarders and indexers. A universal forwarder ignores them unless force_local_processing is enabled on it.

---

Architectural Division of Labor: props.conf vs. transforms.conf

Splunk Enterprise manages data onboarding and index-time transformations through a two-file configuration model. This architectural separation decouples where and when an action occurs from what and how the data is manipulated.

+-----------------------------------------------------------------------+
| props.conf                                                            |
| Identifies Input Scope: [<spec>] (sourcetype, source, or host)        |
| Invokes Transform Class: TRANSFORMS-<class> = <stanza1>, <stanza2>    |
+-----------------------------------------------------------------------+
                                   |
                                   v (Calls Stanza Definition)
+-----------------------------------------------------------------------+
| transforms.conf                                                       |
| Defines Mutation Logic: [<stanza1>]                                   |
| Directives: REGEX, FORMAT, DEST_KEY, WRITE_META, LOOKAHEAD            |
+-----------------------------------------------------------------------+

1. The Role of props.conf (Scoping & Orchestration)

In props.conf, administrators identify the context of the incoming data stream. The stanza header defines the matching boundary:

• [<sourcetype>]: Applies the rule to all data assigned the specified sourcetype (most common and recommended practice).
• [source::<source>]: Applies the rule to events matching a source pattern (using glob wildcards, e.g., [source::.../access*.log]).
• [host::<host>]: Applies the rule to data arriving from a specific host.

Within that stanza, the TRANSFORMS-<class> directive references one or more transformation definitions located in transforms.conf:

# $SPLUNK_HOME/etc/apps/<app_name>/local/props.conf
[custom:app:json]
TRANSFORMS-route_and_filter = drop_healthcheck, route_critical_to_sec

Here, <class> represents a unique label (route_and_filter) used to differentiate multiple transform directives. It points directly to stanza headers declared in transforms.conf.

2. The Role of transforms.conf (Mutation Specification)

In transforms.conf, each stanza header matches a name referenced by props.conf. This file contains the granular regular expression matching, field formatting, and destination queue or metadata targeting instructions.

---

Processing Tiers: Where Transformations Execute

Index-time TRANSFORMS run in the typing pipeline, in the regex-replacement processor. The pipelines in order are parsing (line breaking), merging (timestamps and line merging), typing (TRANSFORMS, SEDCMD), and index.

Universal forwarder (default)
  input -> (character encoding, input-time settings) -> tcpout to 9997
  * no line breaker, aggregator, or regex-replacement processor
  * TRANSFORMS in its props.conf have no effect
                 |
                 v  (unparsed data)
Heavy forwarder or indexer
  parsingQueue  -> line breaking (LINE_BREAKER, TRUNCATE)
  aggQueue      -> timestamps, line merging
  typingQueue   -> TRANSFORMS / SEDCMD (regex replacement)
                     queue = nullQueue  -> event dropped
                     _MetaData:Index    -> index changed
  indexQueue    -> written to the index (or forwarded, on a heavy forwarder)

The Tier Rule

1. Universal forwarders: by default, a UF does not break lines, merge lines, or run regex replacement, so TRANSFORMS-<class> in its props.conf does nothing, and the data leaves unmodified. The exception is the UF-only props.conf setting force_local_processing = true. For that source type, it makes the forwarder run the line breaker, aggregator, and regex-replacement processors locally, at the cost of extra CPU and memory.
2. Heavy forwarders: a full Splunk instance parses data and applies TRANSFORMS before forwarding. Masking, filtering, host overrides, and routing done there reach the indexers already applied.
3. Indexers: apply TRANSFORMS to data that arrives unparsed, for example from universal forwarders. props.conf.spec notes that the processor ignores TRANSFORMS on an indexer if they were already processed on the heavy forwarder. The newer RULESET-<class> setting (ingest actions) differs: it is processed on both tiers.

---

Transformation Execution Sequence and Chaining

Enterprise architectures frequently require applying several discrete transformations to a single event stream—for example, scrubbing an API key, overriding the host header, and routing high-priority errors to a dedicated index. Splunk provides two distinct mechanisms for chaining multiple transforms:

Chaining Method 1: Comma-Separated List in a Single Directive

Administrators can list multiple transform stanzas separated by commas under a single TRANSFORMS-<class> key in props.conf:

# props.conf
[cisco:asa]
TRANSFORMS-pipeline = strip_syslog_header, override_asa_host, route_security_index

Evaluation Rule: Splunk executes comma-separated transforms strictly in left-to-right order:

1. strip_syslog_header executes first.
2. override_asa_host executes second, operating on the event state produced by the first transform.
3. route_security_index executes third, evaluating against the modified payload and metadata.

Chaining Method 2: Multiple Directives via Class Suffixes

Alternatively, administrators can declare multiple TRANSFORMS keys, each with a unique <class> identifier:

# props.conf
[cisco:asa]
TRANSFORMS-01_strip_header  = strip_syslog_header
TRANSFORMS-02_override_host = override_asa_host
TRANSFORMS-03_route_index   = route_security_index

Evaluation rule: when a stanza has several TRANSFORMS-<class> settings, they are applied in ASCII (lexicographic) order of the <class> names, not in the order they are written in the file:

• 01_strip_header executes before 02_override_host.
• 02_override_host executes before 03_route_index.

Best practice: when order matters, put the transforms in one comma-separated list, which props.conf.spec guarantees is applied in list order. If you use separate classes, give them sortable names (01_, 02_, …). Otherwise TRANSFORMS-clean runs before TRANSFORMS-route simply because c sorts before r.

Sequential Event State Mutation

Transformations are not isolated evaluations; they mutate the event in memory as it traverses the typingQueue. Consider an event entering the pipeline:

Raw Ingest:  "2026-09-23 10:14:02 gw01.corp ASA-6-302013: Built inbound TCP connection..."

1. Step 1 (strip_syslog_header): Modifies _raw by removing the timestamp and relay prefix. The event in memory is now: "ASA-6-302013: Built inbound TCP connection..."
2. Step 2 (override_asa_host): sets DEST_KEY = MetaData:Host with FORMAT = host::$1. The value must carry the host:: prefix. Because Step 1 already removed the host from _raw, this transform must capture it before Step 1 runs, or read it from another key with SOURCE_KEY. Order matters.
3. Step 3 (route_security_index): Evaluates REGEX = ^ASA-6- against the current state of _raw. Because the syslog header was already stripped in Step 1, the regex matches immediately at index 0 (^). If Step 1 had not executed, an unanchored regex would have been required.

---

Advanced transforms.conf Attributes: LOOKAHEAD & WRITE_META

Beyond basic regex matching, fine-tuning transformation behavior requires mastery of operational attributes:

1. The LOOKAHEAD Setting

By default, LOOKAHEAD = 4096, so REGEX searches only the first 4,096 characters of the event.

• If an event is a large stack trace or a 32 KB XML or JSON payload, and the text needed for routing or masking appears at character 8,000, the transform does not match.
• In such scenarios, administrators must explicitly raise LOOKAHEAD in transforms.conf:

[route_large_payloads]
REGEX = "audit_classification":\s*"RESTRICTED"
DEST_KEY = _MetaData:Index
FORMAT = compliance_vault
LOOKAHEAD = 32768

2. WRITE_META and Indexed Fields

Setting WRITE_META = true writes the FORMAT result (for example FORMAT = err_code::$1) into the event's metadata as an indexed field, stored in the index files alongside the other indexed terms. The spec recommends WRITE_META = true rather than DEST_KEY = _meta.

• Search Impact: Allows searches like fieldname::value to be resolved entirely at the indexer storage layer.
• Cost: every indexed field adds to the index files and to indexing work. Splunk's general guidance is to prefer search-time extractions and to create indexed fields only where they bring a clear benefit, such as frequently used fields in very large data sets.

---

Administrative Traps and Troubleshooting