3.1 Splunk Configuration Directory Structure

Splunk Configuration Directory Structure

Splunk Enterprise manages virtually all aspects of its behavior—from data ingestion and parsing to search execution, user authorization, and distributed clustering—through plain-text configuration files. These files, identifiable by their .conf extension, are distributed throughout a structured filesystem hierarchy rooted in the $SPLUNK_HOME/etc directory.

Understanding the anatomical layout of $SPLUNK_HOME/etc is foundational for every Splunk administrator. Rather than maintaining a single monolithic configuration file, Splunk divides settings across modular directories according to scope, operational context, and management tier.

---

Anatomy of $SPLUNK_HOME/etc

On any operating system hosting Splunk Enterprise or the Universal Forwarder, the environment variable $SPLUNK_HOME points to the root installation directory (typically /opt/splunk or /opt/splunkforwarder on Linux, and C:\Program Files\Splunk on Windows). Within $SPLUNK_HOME, the etc/ subdirectory contains all configuration, security, user, and application metadata.

---

System vs. App vs. User Scopes

The configuration hierarchy under $SPLUNK_HOME/etc is partitioned into three primary operational tiers:

1. System Directory ($SPLUNK_HOME/etc/system/)

The system tier defines instance-wide parameters that apply globally to the underlying splunkd process. Settings located here govern foundational daemon operations such as management ports, server certificates, authentication backends, and base licensing. Subdirectories include:

• system/default/: Houses the baseline configuration files installed with the Splunk binary.
• system/local/: Houses host-specific overrides created by administrators directly or through system-level CLI commands (e.g., splunk set web-port 8443).

2. Applications Directory ($SPLUNK_HOME/etc/apps/)

Splunk Enterprise organizes functional capabilities into applications and Technology Add-ons (TAs). Each subdirectory under etc/apps/ represents a self-contained operational unit with its own internal directory tree:

$SPLUNK_HOME/etc/apps/<app_name>/
├── default/         <-- Vendor-supplied configurations and baseline stanzas
├── local/           <-- Site-specific modifications and UI edits for this app
├── metadata/        <-- Object permissions and export scoping (default.meta, local.meta)
├── bin/             <-- Custom scripted inputs, alert actions, modular inputs
├── lookups/         <-- CSV tables and geospatial KMZ lookup files
└── static/          <-- Web assets (CSS, JavaScript, images) used by dashboards

3. Users Directory ($SPLUNK_HOME/etc/users/)

When users interact with Splunk Web, knowledge objects (such as saved reports, event types, field extractions, and calculated fields) default to private sharing. Splunk stores these private objects in the user's personal configuration hierarchy:

$SPLUNK_HOME/etc/users/<username>/<app_name>/local/

If an object is later shared with other users of the app or globally across all apps, Splunk migrates the stanza out of etc/users/ and promotes it to $SPLUNK_HOME/etc/apps/<app_name>/local/ while recording permission changes in etc/apps/<app_name>/metadata/local.meta.

---

Centralized Distribution Repositories

In enterprise environments, administrators rarely configure individual indexers, search heads, or forwarders directly on each host. Instead, specialized management nodes distribute configurations downstream using designated staging repositories:

Deployment Server ($SPLUNK_HOME/etc/deployment-apps/)

On a Splunk Deployment Server (DS), applications destined for deployment clients (such as Universal Forwarders and Heavy Forwarders) are staged inside etc/deployment-apps/. The Deployment Server references serverclass.conf to map these apps to specific client tiers. When clients poll the Deployment Server on management port 8089, updated apps are downloaded into the client's local $SPLUNK_HOME/etc/apps/ directory.

Indexer Cluster Manager ($SPLUNK_HOME/etc/manager-apps/, formerly master-apps/)

In an Indexer Cluster, peer indexers must maintain uniform indexing and parsing configurations (indexes.conf, props.conf, transforms.conf). Configurations intended for peer indexers are staged on the Cluster Manager in $SPLUNK_HOME/etc/manager-apps/ (master-apps/ in older releases). When an administrator executes splunk apply cluster-bundle, the Cluster Manager validates the bundle and deploys it to each peer's $SPLUNK_HOME/etc/peer-apps/ (formerly slave-apps/) directory.

Search Head Cluster Deployer ($SPLUNK_HOME/etc/shcluster/apps/)

In a Search Head Cluster (SHC), runtime knowledge-object changes that users make on any member (saved searches, dashboards, and similar) are replicated to the other members through the captain. However, packaged applications, technology add-ons, and baseline settings must be deployed using the Deployer. Staged within $SPLUNK_HOME/etc/shcluster/apps/, these configurations are pushed to members via the command splunk apply shcluster-bundle -target <member_uri>.

---

The Sacred Rule: default vs. local

Within both $SPLUNK_HOME/etc/system and each individual directory under $SPLUNK_HOME/etc/apps/, configuration files reside inside either a default/ or local/ subfolder. This duality is one of the most critical design concepts in Splunk administration.

Why default Files Must Never Be Edited

Files inside default/ directories represent the factory baseline shipped by Splunk or an app developer. When Splunk is upgraded to a newer maintenance release or major version, the upgrade installer completely replaces the contents of $SPLUNK_HOME/etc/system/default/ and the default/ directories of core apps (such as search and launcher).

[!CAUTION] If an administrator modifies files directly inside a default/ directory, those changes will be permanently overwritten and lost during the next software or app upgrade without warning.

How local Isolates Customizations

The local/ directory exists precisely to store administrator modifications, customizations, and overrides. Key principles of local/ management include:

• Delta Storage Only: A local configuration file should never be a full copy of the default file. It should contain only the specific stanzas and attributes that differ from defaults.
• Upgrade Resilience: Splunk upgrade installers never touch or overwrite existing files inside local/ directories.
• Splunk Web Write Target: Whenever an administrator or user modifies configurations through Splunk Web (such as adding a data input, adjusting an index size limit, or editing a role), Splunk writes those changes exclusively to the corresponding local/ directory.

---

Stanza and Attribute Syntax Rules

All .conf files follow an ASCII or UTF-8 text structure governed by explicit syntax conventions:

# Comment lines begin with a hash character
[stanza_name]
attribute_key = attribute_value
secondary_key = secondary_value

[secondary_stanza]
enable_feature = true

• Stanzas ([...]): Define the operational scope or object identifier (e.g., [monitor:///var/log/syslog], [tcpout:primary_indexers], or [volume:hot1]). Stanza names are case-sensitive.
• Attributes (key = value): Define discrete operational directives. Attribute names are case-sensitive (sourcetype and SOURCETYPE are different keys), while attribute values may be case-sensitive depending on whether they represent filesystem paths, regexes, or boolean flags (true/false, 1/0).
• Whitespace: Leading and trailing whitespace around the equals sign (=) is ignored by the parser.
• Comments: Start a comment line with #. Do not put comments at the end of a setting line; Splunk treats the text after the value as part of the value.

---

Core Configuration Files Overview

Splunk Enterprise utilizes dozens of configuration files, but ten core files form the bedrock of system administration, data routing, parsing, and security:

Core Administrative Configuration Files
├── server.conf          <-- Server identity, clustering, SSL, licensing
├── inputs.conf          <-- Ingestion definitions (files, TCP/UDP, scripts, HEC)
├── outputs.conf         <-- Forwarder targets, load balancing, queue sizing
├── indexes.conf         <-- Index paths, bucket retention, size constraints
├── props.conf           <-- Parsing, line breaking, timestamping, sourcetypes
├── transforms.conf      <-- Regex routing, masking, index-time transformations
├── authentication.conf  <-- User auth backends (LDAP, SAML, internal)
├── authorize.conf       <-- Roles, capabilities, search filters, index access
├── distsearch.conf      <-- Distributed search peers and bundle replication
└── web.conf             <-- Splunk Web UI server, port 8000, HTTPS SSL

1. server.conf

Defines the core identity of the Splunk instance. Controls the management port (default 8089), server name, licensing configuration (pointing to a License Manager), cluster roles (Cluster Manager, peer indexer, search head cluster member), indexer discovery, and SSL certificate parameters for inter-node communication.

[general]
serverName = idx01.corp.internal
pass4SymmKey = $7$EncryptedPasswordString

[clustering]
mode = peer
manager_uri = https://cm01.corp.internal:8089
pass4SymmKey = $7$ClusterSecurityKeyString

2. inputs.conf

Governs all data onboarding into Splunk. Defines file and directory monitoring ([monitor://...]), network ingestion ports ([tcp://...], [udp://...]), HTTP Event Collector endpoints ([http://...]), and scripted inputs ([script://...]). Sets default metadata values (index, sourcetype, source, host) at the input layer.

[monitor:///var/log/secure]
disabled = 0
index = os_security
sourcetype = linux_secure

3. outputs.conf

Configured primarily on Universal and Heavy Forwarders to govern data egress. Defines receiving indexer target groups, automated TCP load balancing, indexer acknowledgment (useACK = true), SSL/TLS transport encryption, and local queue memory buffers.

[tcpout]
defaultGroup = production_indexers

[tcpout:production_indexers]
server = 10.1.0.11:9997, 10.1.0.12:9997, 10.1.0.13:9997
useACK = true

4. indexes.conf

Configured on indexers to define database repositories for indexed events. Specifies filesystem paths for bucket tiers (homePath for hot/warm, coldPath for cold, thawedPath for restored data), storage size limits (maxTotalDataSizeMB), time-based retention thresholds (frozenTimePeriodInSecs), volume definitions for tiered disk storage, and cold-to-frozen archiving scripts.

[volume:primary]
path = /mnt/fast_nvme
maxVolumeDataSizeMB = 2000000

[web_traffic]
homePath = volume:primary/web_traffic/db
coldPath = /mnt/storage/web_traffic/colddb
thawedPath = $SPLUNK_DB/web_traffic/thaweddb
maxTotalDataSizeMB = 500000
frozenTimePeriodInSecs = 7776000

5. props.conf

Acts as the central configuration file for event processing and knowledge object associations. During the parsing pipeline on indexers or Heavy Forwarders, it controls line breaking (LINE_BREAKER, SHOULD_LINEMERGE), timestamp extraction (TIME_PREFIX, TIME_FORMAT, MAX_TIMESTAMP_LOOKAHEAD), and character encoding (CHARSET). At search time on search heads, it orchestrates field extractions (EXTRACT-*), lookup table associations (LOOKUP-*), and calculated fields.

[custom:json:app]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
TIME_PREFIX = "timestamp":\s*"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%z
MAX_TIMESTAMP_LOOKAHEAD = 35
TRUNCATE = 20000

6. transforms.conf

Works in tandem with props.conf to execute complex regular expression operations. At index time, it handles data routing to alternate indexes, host/sourcetype overrides, masking by rewriting _raw (DEST_KEY = _raw), and filtering events to nullQueue. (SEDCMD masking is configured in props.conf, not here.) At search time, it defines delimiter-based field extractions and lookup table specifications (e.g., CSV filenames, CIDR matching rules).

[filter_debug_events]
REGEX = (?i)log_level=(?:DEBUG|TRACE)
DEST_KEY = queue
FORMAT = nullQueue

7. authentication.conf

Defines user authentication backends and security integrations. Configures native internal user storage, LDAP/Active Directory connectivity (bind DN, server host, port, user/group search filters), SAML 2.0 Identity Provider (IdP) integration for Single Sign-On (SSO), and Multi-Factor Authentication (MFA).

[authentication]
authType = LDAP
authSettings = corp_ad

[corp_ad]
host = ad.corp.internal
port = 636
SSLEnabled = 1
bindDN = CN=SplunkService,OU=Services,DC=corp,DC=internal

8. authorize.conf

Controls Role-Based Access Control (RBAC). Defines roles (e.g., admin, power, user), inherits capabilities between roles, controls search constraints (maximum concurrent searches, search time windows), specifies searchable index access (srchIndexesAllowed, srchIndexesDefault), and enforces search filter restrictions (srchFilter).

[role_security_analyst]
importRoles = user
srchIndexesAllowed = os_security;network_logs;firewall
srchIndexesDefault = os_security
srchFilter = host=pci-* OR host=dmz-*

9. distsearch.conf

Configured on search heads to manage distributed search infrastructure. Defines connections to peer indexers ([distributedSearch], servers = idx01:8089, idx02:8089), search peer status monitoring, connection timeouts, and knowledge bundle replication settings such as [replicationSettings] (replicationPolicy, maxBundleSize) and [replicationDenylist].

[distributedSearch]
servers = https://idx01.corp.internal:8089, https://idx02.corp.internal:8089
statusTimeout = 10
connectionTimeout = 10

10. web.conf

Configures the Splunk Web application server (running on the CherryPy/Python web engine). Controls HTTP/HTTPS listener settings (default TCP port 8000), TLS/SSL certificates (serverCert, privKeyPath), cipher suites, user session timeout durations, and feature toggles for web-based developer tools.

[settings]
httpport = 8443
enableSplunkWebSSL = true
privKeyPath = /opt/splunk/etc/auth/splunkweb/web_key.pem
serverCert = /opt/splunk/etc/auth/splunkweb/web_cert.pem
tools.sessions.timeout = 60