3.2 Configuration Layering, Precedence & Resolution Rules

Configuration Layering, Precedence & Resolution Rules

One of the most powerful architectural features of Splunk Enterprise is its layered configuration model. Multiple applications, add-ons, and system files can define settings for the exact same stanza simultaneously. When the splunkd daemon initializes or executes a task, it dynamically merges these disparate files into a single, cohesive, effective runtime configuration.

To manage this layering reliably, Splunk enforces deterministic precedence hierarchies. The exact sequence of precedence depends entirely on whether a configuration setting is evaluated within the Global context or the App and User context.

---

The Two Configuration Contexts

Splunk configuration files are partitioned into two distinct evaluation contexts based on the operational subsystem requiring the configuration:

Configuration Context Boundary
├── Global Context (System, Indexing, Inputs, Forwarding)
│   └── Evaluated independent of UI sessions, user identities, or active apps.
│       Governs background daemons, network ports, index storage, and ingestion.
│
└── App and User Context (Search Time & Knowledge Objects)
    └── Evaluated within a user session and active application framework.
        Governs dashboards, saved searches, event types, field extractions, and tags.

1. Global Context

Global context applies to background daemon processes, data ingestion pipelines, indexing operations, forwarder egress, and system administration. These activities affect the Splunk instance as an entire operating unit, independent of who is logged into Splunk Web or which application view is active.

Key files operating in the Global context include:

• server.conf (clustering, licensing, daemon identity)
• inputs.conf (file monitors, network listening ports, modular inputs)
• outputs.conf (forwarder destination groups, load balancing)
• indexes.conf (index definitions, bucket paths, retention)
• authentication.conf (LDAP, SAML, user accounts)
• authorize.conf (role definitions and capabilities)
• distsearch.conf (search peer communication)
• web.conf (Splunk Web server configuration)

2. App and User Context

App and User context applies to search activities, user interfaces, and knowledge objects. When a user runs a search or loads a dashboard, Splunk evaluates configurations based on who is running the search and where (in which app) the search is executing.

Key files operating in the App/User context include:

• savedsearches.conf (reports, alerts, scheduled searches)
• eventtypes.conf (event classifications)
• tags.conf (field tags)
• macros.conf (search macros)
• collections.conf (KV Store collections)
• Search-time aspects of props.conf and transforms.conf (search-time field extractions, lookups, calculated fields)

---

Global Context Precedence Hierarchy

When splunkd evaluates configurations in the Global context, it follows a strict four-tier precedence hierarchy. Higher-numbered tiers override lower-numbered tiers:

Global Precedence Order:
System local  -->  App local  -->  App default  -->  System default
[HIGHEST]                                            [LOWEST]

Conflict Resolution Between Competing Apps (Lexical Ordering)

In an enterprise environment, many apps and add-ons live in $SPLUNK_HOME/etc/apps/. What happens when two apps set the same attribute in the same stanza of a global-context file such as inputs.conf or indexes.conf?

First, all app local directories outrank all app default directories. Within the app local tier (or within the app default tier), Splunk breaks ties by the lexicographical (ASCII/UTF-8) order of the app directory names, and the name that sorts first wins:

• Numbers sort before letters and are compared character by character: myapp1 > myapp10 > myapp2 > myapp20.
• Uppercase letters sort before lowercase letters. Z outranks a, so any app named with an uppercase first letter outranks apps named in lowercase.

[!IMPORTANT] In the global context, when two apps set the same attribute at the same tier (for example, both in local/), the app whose directory name comes first in lexicographical order wins. In the app/user context the order is reversed. There, other apps are evaluated in reverse-lexicographical order, and only for exported settings.

For example, suppose both etc/apps/Splunk_TA_nix/local/inputs.conf and etc/apps/custom_inputs/local/inputs.conf set a different index for [monitor:///var/log/messages]. Uppercase S (0x53) sorts before lowercase c (0x63), so Splunk_TA_nix comes first and its value wins in the global context.

Administrators use this to force an outcome deliberately. A site-wide override app named with a leading digit, such as 000_corp_overrides, sorts ahead of alphabetic app names, so its global-context settings win ties against them. A name like zzz_... would lose.

Expanded Order on Indexer Cluster Peers

Indexer cluster peers add a directory type for configuration that the cluster manager distributes. On a peer, the global-context order is: peer-apps local > system local > app local > peer-apps default > app default > system default. The cluster manager pushes common peer configuration from $SPLUNK_HOME/etc/manager-apps/ (formerly master-apps/) into $SPLUNK_HOME/etc/peer-apps/ (formerly slave-apps/).

---

App and User Context Precedence Hierarchy

When a search executes or a knowledge object is referenced, Splunk evaluates configurations through the App and User context. In this context, user customization takes precedence over app defaults, and current app settings take precedence over external apps:

App/User Precedence Order:
User local --> Current App local --> Current App default --> Other Apps --> System local --> System default
[HIGHEST]                                                                                    [LOWEST]

Knowledge Object Export Scoping (metadata)

An object defined inside an app is only visible to "Other Apps" (tiers 4 and 5) if its sharing scope has been configured to global. Object sharing is governed by metadata/default.meta and metadata/local.meta:

• export = none: Accessible only within the defining app.
• export = system: Exported globally across all applications and search contexts.

---

Stanza-Level and Attribute-Level Merging Mechanics

A critical misconception among new administrators is the belief that higher-precedence files overwrite entire configuration files or entire stanzas. This is false.

Splunk uses a fine-grained, non-destructive merging model:

1. Files are not replaced: An attribute defined in system/local does not replace the file in apps/*/local.
2. Stanzas are not discarded: If a stanza exists in both a default file and a local file, Splunk merges the contents of both stanzas into a single composite stanza.
3. Attribute-level resolution: Splunk inspects each key-value pair independently:
   • If an attribute exists in a higher-precedence tier, that attribute's value wins.
   • If an attribute is absent from the higher-precedence tier, the value is inherited from the next highest tier where the key is defined.
   • Any non-conflicting keys across all tiers are preserved in the final merged stanza.

---

Concrete Merging Walkthrough: Data Ingestion Scenario

Consider an administrator monitoring /var/log/secure on a Linux host. Four configuration files contribute settings to the input [monitor:///var/log/secure]:

Tier 4: System Default ($SPLUNK_HOME/etc/system/default/inputs.conf)

The shipped file does not contain this monitor stanza. It does contain a [default] stanza, and settings in [default] apply to every input stanza that does not set them itself:

[default]
index = default

Tier 3: App Default ($SPLUNK_HOME/etc/apps/Splunk_TA_nix/default/inputs.conf)

[monitor:///var/log/secure]
sourcetype = linux_secure
index = os

Tier 2: App Local ($SPLUNK_HOME/etc/apps/Splunk_TA_nix/local/inputs.conf)

[monitor:///var/log/secure]
index = linux_security
ignoreOlderThan = 7d

Tier 1: System Local ($SPLUNK_HOME/etc/system/local/inputs.conf)

[monitor:///var/log/secure]
disabled = 1

Step-by-Step Resolution Analysis

When splunkd evaluates this configuration in the Global context, it merges the stanza attribute-by-attribute across the four tiers:

Final Effective Runtime Stanza Calculated by Splunk:

[monitor:///var/log/secure]
disabled = 1
index = linux_security
sourcetype = linux_secure
ignoreOlderThan = 7d

The result combines settings from three layers (system/local, apps/Splunk_TA_nix/local, and apps/Splunk_TA_nix/default), plus the [default] fallback from system/default. Splunk merges attributes; it does not discard whole files or stanzas.