7.1 CLI Administration & Basic Data Onboarding Workflows

CLI Administration & Basic Data Onboarding Workflows

Quick Summary: Proficient Splunk administration relies heavily on the Command Line Interface (CLI). From managing the core splunkd daemon lifecycle to onboarding data sources and verifying configuration precedence with btool, the CLI allows administrators to execute precise, scriptable operations. Understanding CLI syntax, token-based authentication mechanisms, and the underlying configuration files altered by CLI commands is essential for enterprise operations.

The Splunk CLI Environment & Daemon Lifecycle Controls

The primary executable for all command-line operations is $SPLUNK_HOME/bin/splunk.

• In Unix/Linux environments, $SPLUNK_HOME defaults to /opt/splunk for Splunk Enterprise or /opt/splunkforwarder for Universal Forwarders.
• In Windows environments, the default path is C:\Program Files\Splunk or C:\Program Files\SplunkUniversalForwarder.
• Administrators typically add $SPLUNK_HOME/bin to the system $PATH or execute commands directly from the bin directory.

Core Daemon Lifecycle Commands

Managing the splunkd background daemon involves several critical administrative commands:

# Start the Splunk daemon
splunk start

# Start Splunk, automatically accepting the license agreement without prompt
splunk start --accept-license --answer-yes

# Stop the Splunk daemon gracefully
splunk stop

# Perform a clean restart (stops daemon, flushes queues, reloads configs, restarts)
splunk restart

# Check whether splunkd is currently running and view its Process ID (PID)
splunk status

# Configure Splunk to launch automatically on host boot under a dedicated service user
splunk enable boot-start -user splunk

Operational Mechanics of Lifecycle Commands

• splunk start: Validates critical configuration file syntax, verifies port availability (checking that 8089, 8000, etc., are not bound by other processes), checks for conflicts such as a stale lock or PID file, and launches splunkd (which in turn starts helper processes such as the KV store).
• splunk stop: Initiates a graceful shutdown sequence. It stops accepting new ingestion connections, signals running search jobs to terminate or finalize, flushes memory queues to disk, closes open bucket file handles, and terminates the daemon cleanly. Abruptly killing the process (kill -9) can corrupt active hot bucket journal files.
• splunk enable boot-start: Generates system startup scripts. In modern Linux distributions running systemd, running splunk enable boot-start -user splunk -systemd-managed 1 creates a validated systemd unit file (splunk.service) configured with appropriate user privileges, resource limits (ulimits), and restart policies.

---

Authentication, Credentials & Session Token Mechanics

Every CLI command that inspects or alters the state of Splunk communicates with the splunkd process over the management port (default TCP 8089) via REST API endpoints. Consequently, CLI commands require administrative authentication.

Providing Credentials

Administrators can supply credentials using several methods:

1. Inline Credentials (-auth flag):

   splunk list forward-server -auth admin:MyPassword123
   

   Caution: Passing credentials directly via -auth exposes plaintext passwords in the shell process table (ps aux) and in user command history (.bash_history).

2. Interactive Prompt: If a command requiring authentication is executed without the -auth flag:

   splunk list forward-server
   

   The CLI prompts interactively:

   Your session is invalid.  Please login.
   Splunk username: admin
   Password:
   

3. Session Token Storage: Upon successful interactive authentication, the CLI generates an authentication session token and caches it on the host filesystem:

   • Linux/macOS: $HOME/.splunk/authToken (or $SPLUNK_HOME/var/run/splunk/authToken_8089)
   • Windows: %USERPROFILE%\.splunk\authToken Subsequent CLI commands automatically read this cached token, permitting administrative operations without prompting until the token reaches its expiration timeout (configured by sessionTimeout in server.conf, defaulting to 1 hour).

4. Dedicated CLI Login / Logout: Administrators can explicitly establish or terminate a CLI session:

   # Log in and cache a session token
   splunk login -auth admin:MyPassword123
   
   # Invalidate and delete the cached session token
   splunk logout
   

---

The Basic Settings Every Input Carries

Whether you create an input in Splunk Web, with the CLI, or by editing inputs.conf, every input sets the same small group of basics. These defaults apply before any parsing happens:

In Splunk Web, Settings > Add Data walks through the same choices. You pick a method (Upload, Monitor, or Forward), select the source, set the source type on the Set Source Type page, and then choose app context, host, and index on the Input Settings page. The CLI writes the same keys into an inputs.conf file.

CLI Data Onboarding Commands & Workflows

While graphical data onboarding is accessible via Splunk Web, CLI commands are standard for headless installations, automated server provisioning, and forwarder deployments.

1. Persistent Monitoring: splunk add monitor

The splunk add monitor command instructs Splunk to monitor a file or directory continuously.

# Monitor a specific log file with explicit index and sourcetype
splunk add monitor /var/log/secure -index os_security -sourcetype linux_secure

# Monitor an entire directory (add a whitelist later in inputs.conf if needed)
splunk add monitor /var/log/httpd/ -index web -sourcetype apache_access

Under the Hood: What splunk add monitor Modifies

• The command writes a new monitor stanza directly into $SPLUNK_HOME/etc/apps/search/local/inputs.conf (or $SPLUNK_HOME/etc/system/local/inputs.conf depending on context):

  [monitor:///var/log/secure]
  index = os_security
  sourcetype = linux_secure
  disabled = 0
  

• The Fishbucket Registration: splunkd reads the first 256 bytes of the target file to compute a beginning CRC fingerprint. That CRC, the seek address it has read up to, and a CRC of the data at that point are stored in Splunk's tracking database, the fishbucket ($SPLUNK_DB/fishbucket/splunk_private_db, exposed as the _thefishbucket index). This state tracking guarantees that if Splunk restarts, it resumes reading from the exact byte offset where it left off, avoiding duplicate events.

2. Batch One-Time Loading: splunk add oneshot

When an administrator needs to index an archived log file or historical dump without establishing continuous file tracking, splunk add oneshot provides a synchronous batch mechanism:

splunk add oneshot /var/log/historical_dump_2025.log -index legacy_data -sourcetype legacy_app

Key Differences: add monitor vs. add oneshot

3. Forwarder Target Configuration: splunk add forward-server

Configuring forwarder egress to route data to downstream indexers is performed using forward-server CLI commands:

# Add an active indexer receiving target
splunk add forward-server indexer01.corp.internal:9997 -auth admin:<password>

# Add a second indexer target for automatic load balancing
splunk add forward-server indexer02.corp.internal:9997 -auth admin:<password>

# List all configured forwarder destinations and their connection status
splunk list forward-server

# Remove a decommissioned indexer from the forwarder configuration
splunk remove forward-server indexer01.corp.internal:9997

Under the Hood: What add forward-server Modifies

• The command writes target group stanzas into $SPLUNK_HOME/etc/system/local/outputs.conf:

  [tcpout]
  defaultGroup = default-autolb-group
  
  [tcpout:default-autolb-group]
  server = indexer01.corp.internal:9997, indexer02.corp.internal:9997
  

• splunk list forward-server queries splunkd over port 8089 to report the status of TCP connections, identifying whether each indexer is Active (currently receiving data) or Configured (available in the failover/load-balancing pool).

---

Configuration Verification & Inspection via CLI

Because Splunk evaluates multiple layers of .conf files across system, apps, and user directories, diagnosing the effective runtime configuration is impossible by reading a single file. The CLI provides specialized tools to inspect merged configuration state:

1. Inspecting Resolved Settings with splunk btool

btool is the premier diagnostic command for Splunk administrators. It simulates the exact configuration merging logic that splunkd executes at startup:

# View the merged settings for a configuration file type (e.g. inputs.conf)
splunk btool inputs list

# Trace the exact file path and line number where each attribute was set
splunk btool inputs list --debug

# Filter for a specific stanza within the resolved configuration
splunk btool inputs list --debug | grep -A 5 "monitor:///var/log/secure"

# Inspect server.conf across all apps to verify the license manager setting (manager_uri)
splunk btool server list --debug | grep -A 4 "\[license\]"

2. Validating Configuration Syntax with btool check

To detect typos, illegal attribute names, and syntax errors before restarting the production daemon:

splunk btool check

• If an administrator misspelled an attribute (for example, typing sourctyp = custom instead of sourcetype = custom in inputs.conf), splunk btool check reports an "Invalid key in stanza" line naming the file, line number, and key.

3. Hot-Reloading Subsystems Without Daemon Restart

While changes to low-level networking or cluster peering require a full restart (splunk restart), several operational subsystems can be reloaded dynamically via the CLI:

# Reload Deployment Server classes and bundles
splunk reload deploy-server

# Reload authentication settings (e.g. after editing LDAP configurations)
splunk reload auth

# Force splunkd to reload monitor inputs without dropping active connections
splunk _internal call /services/data/inputs/monitor/_reload -auth admin:password

---

Administrative Pitfalls & Best Practices

1. Credential Exposure in Command Line Arguments: Avoid typing -auth username:password in scripts or shared terminal sessions. Use an interactive splunk login session (which caches a session token for the OS user) or a dedicated automation account with the minimum capabilities it needs.
2. Local Stanza Pollution: CLI commands like splunk add monitor default to writing into the search app's local directory ($SPLUNK_HOME/etc/apps/search/local/inputs.conf) or $SPLUNK_HOME/etc/system/local/inputs.conf. In structured enterprise environments, administrators should package inputs inside modular, version-controlled Technology Add-ons (e.g., TA-linux-inputs/default/inputs.conf) distributed centrally via the Deployment Server rather than running manual CLI input additions on production servers.
3. Fishbucket Collision after File Moves: If a log file is indexed via splunk add oneshot and later moved into a monitored directory, Splunk will index it again, because a oneshot upload does not leave a monitor checkpoint in the fishbucket.
4. Missing Service User Context: Running splunk start as the root superuser after Splunk was installed to run under an unprivileged user (e.g. splunk) alters file permissions in $SPLUNK_HOME/var/. When the unprivileged user subsequently attempts to start Splunk, splunkd fails with permission-denied errors on files it can no longer write. Always execute CLI commands as the dedicated splunk service user.