3.3 Using btool to Inspect & Debug Configurations

Using btool to Inspect & Debug Configurations

In an enterprise Splunk deployment with dozens of apps, Technology Add-ons, and system-level overrides, manually determining which configuration file defined a specific setting is nearly impossible. Searching through directories using grep or find fails to account for Splunk's complex precedence rules, lexical sorting, and attribute-level merging.

To solve this challenge, Splunk provides btool—a purpose-built command-line diagnostic utility that executes the exact same parsing and layering logic as the core splunkd engine, revealing the final effective configuration on disk.

---

The Purpose and Mechanics of btool

The primary purpose of btool is to answer two critical administrative questions:

1. What is the effective value of a configuration setting? (btool ... list)
2. Which exact file on disk caused that setting to win? (btool ... list --debug)

btool traverses the directory hierarchy under $SPLUNK_HOME/etc, parses every relevant .conf file, resolves all precedence conflicts, and outputs the resulting composite configuration to standard output (stdout).

Diagnostic Flow of btool:
etc/system/default/ ──┐
etc/apps/*/default/ ──┼──> [ btool Layering Engine ] ──> Merged Effective Output
etc/apps/*/local/   ──┤        (Applies Global or               (with optional file
etc/system/local/   ──┘         App/User Precedence)             provenance via --debug)

---

Core Command Syntax

The btool utility is invoked through the Splunk CLI wrapper located at $SPLUNK_HOME/bin/splunk. The fundamental syntax structure is:

splunk btool <conf_file_prefix> list [stanza_name] [options]

[!IMPORTANT] When specifying the configuration file prefix, omit the .conf file extension. To inspect inputs.conf, specify inputs; to inspect server.conf, specify server.

Essential Command Invocations

1. Listing the Entire Effective Configuration

To display the complete merged configuration for a specific configuration file across all stanzas:

$SPLUNK_HOME/bin/splunk btool inputs list
$SPLUNK_HOME/bin/splunk btool props list
$SPLUNK_HOME/bin/splunk btool indexes list

2. Filtering by a Specific Stanza

Because configuration outputs can span thousands of lines, you can isolate an individual stanza by appending the exact stanza name in brackets:

$SPLUNK_HOME/bin/splunk btool inputs list [monitor:///var/log/messages]
$SPLUNK_HOME/bin/splunk btool indexes list [web_traffic]
$SPLUNK_HOME/bin/splunk btool server list [clustering]

3. Combining with Operating System Utilities

Because btool outputs raw text, it is routinely piped to standard administrative command-line tools:

# Page through effective settings
$SPLUNK_HOME/bin/splunk btool server list | less

# Search for specific attribute keys
$SPLUNK_HOME/bin/splunk btool outputs list | grep -E "(server|autoLB|useACK)"

---

Tracing Attribute Provenance with --debug

The most important diagnostic flag available in btool is --debug. When passed to the list command, --debug prefixes every single line of output with the absolute filesystem path of the configuration file where that specific stanza header or attribute key was defined.

$SPLUNK_HOME/bin/splunk btool inputs list --debug

Deconstructing --debug Output

Examine this sample output obtained by running btool on an indexer:

/opt/splunk/etc/system/local/inputs.conf                   [monitor:///var/log/secure]
/opt/splunk/etc/system/local/inputs.conf                   disabled = 1
/opt/splunk/etc/apps/Splunk_TA_nix/local/inputs.conf       ignoreOlderThan = 7d
/opt/splunk/etc/apps/Splunk_TA_nix/local/inputs.conf       index = linux_security
/opt/splunk/etc/apps/Splunk_TA_nix/default/inputs.conf     sourcetype = linux_secure
/opt/splunk/etc/system/default/inputs.conf                 host = $decideOnStartup

Notice how immediately clear the root cause of any behavior becomes:

• The stanza header is listed with the highest-priority file that contains it (system/local/inputs.conf).
• The attribute disabled = 1 was enforced by system/local/inputs.conf, completely overriding any app settings.
• The attribute index = linux_security originated from apps/Splunk_TA_nix/local/inputs.conf.
• The attribute sourcetype = linux_secure was supplied by apps/Splunk_TA_nix/default/inputs.conf.
• The attributes ignoreOlderThan and index came from apps/Splunk_TA_nix/local/inputs.conf, and host was inherited from the [default] stanza in system/default/inputs.conf.

Without btool --debug, an administrator attempting to troubleshoot why /var/log/secure is not ingesting data might spend hours checking Splunk_TA_nix, completely unaware that an override in system/local/inputs.conf disabled the input.

---

Validating Syntax and Integrity with btool check

In addition to merging configurations, btool includes a built-in syntax verification engine executed via btool check:

$SPLUNK_HOME/bin/splunk btool check

How btool check Works

Inside $SPLUNK_HOME/etc/system/README/, Splunk maintains configuration specification files (*.spec) and example files (*.example) for every supported .conf file (e.g., inputs.conf.spec, props.conf.spec). These specification files define:

• All legitimate stanza regex formats
• All supported attribute keys
• Expected value types (integer, boolean, string)

When btool check runs, it scans the .conf files and compares their keys against these .spec files. A mistyped setting name is reported as an invalid key with its file, line number, and value. btool check does not suggest the setting you meant, so compare against the spec yourself.

Example btool check Output:

Checking: /opt/splunk/etc/apps/search/local/inputs.conf
	Invalid key in stanza [monitor:///var/log/httpd/access.log] in /opt/splunk/etc/apps/search/local/inputs.conf, line 4: indx  (value:  web)
Checking: /opt/splunk/etc/apps/custom_ta/local/props.conf
	Invalid key in stanza [custom_syslog] in /opt/splunk/etc/apps/custom_ta/local/props.conf, line 2: TIME_FORM  (value:  %Y-%m-%d)

Appending --debug to btool check prints the status of every file examined, including valid files, which is useful when diagnosing file permission issues:

$SPLUNK_HOME/bin/splunk btool check --debug

---

Contextual Evaluation: The --app and --user Parameters

By default, btool evaluates configurations strictly within the Global context. While this works perfectly for inputs.conf, outputs.conf, server.conf, and indexes.conf, it produces inaccurate results when debugging search-time knowledge objects (such as savedsearches.conf or search-time props.conf).

To instruct btool to evaluate configurations through the lens of a specific application or user, pass the --app and --user flags:

Evaluating Within an Application Scope

$SPLUNK_HOME/bin/splunk btool props list --app=Splunk_TA_nix
$SPLUNK_HOME/bin/splunk btool savedsearches list --app=search

Evaluating Within a Specific User Scope

$SPLUNK_HOME/bin/splunk btool savedsearches list --app=search --user=analyst_jane --debug

This command evaluates settings starting from $SPLUNK_HOME/etc/users/analyst_jane/search/local/savedsearches.conf down through the App/User precedence chain, allowing administrators to diagnose why a user's private report or dashboard is behaving unexpectedly.

---

Critical Operational Pitfalls of btool

While btool is an indispensable utility, misunderstanding its operational boundaries leads to significant administrative confusion. Administrators must keep four critical caveats in mind:

1. On-Disk State vs. In-Memory Daemon State

btool reads files directly from the filesystem at the moment the command is executed. It does not query the running splunkd process or inspect in-memory memory structures.

• If an administrator modifies an inputs.conf file on disk, running btool immediately displays the new setting.
• However, the running splunkd service continues operating under its old configuration until the service is restarted (splunk restart) or the relevant subsystem is explicitly reloaded (for example splunk reload deploy-server on a deployment server, or Splunk Web's /debug/refresh endpoint for many knowledge objects). When in doubt, restart.
• Pitfall: An administrator sees the correct setting in btool and assumes the live system is using it, when in reality splunkd has not reloaded the file.

2. Offline Operation & Service Independence

Because btool operates strictly as a static file parser, it does not require splunkd to be running. It can be executed while Splunk is completely stopped. This makes it an ideal disaster recovery tool for troubleshooting startup failures caused by corrupt configurations or invalid stanzas.

3. Management Node Boundary

Running btool on a Deployment Server or Cluster Manager evaluates only the configurations active on that management node itself. It cannot evaluate what a downstream forwarder or peer indexer will compute after receiving a deployment bundle.

• To troubleshoot forwarder configuration, btool must be run on the forwarder itself.
• To troubleshoot indexer cluster configuration, btool must be run on an individual peer indexer.

4. Operating System User Permissions

Always run btool as the operating system user that owns the Splunk installation (typically the splunk service account). If executed as root without preserving the environment, btool may generate temporary cache files or lock files owned by root, preventing splunkd from accessing them later.

---

Essential btool Diagnostic Reference Table