13.3 Scripted Inputs & PowerShell Data Collection

Scripted Inputs & PowerShell Data Collection

Quick Summary: Scripted inputs run a command on a schedule and index whatever it writes to standard output, which lets Splunk collect data from APIs, command-line tools, and system utilities. Know where scripts must live, how interval works (seconds, cron, 0 for continuous, -1 for once), that standard error goes to splunkd.log, how passAuth hands a session key to the script, and how Windows [powershell://] inputs are scheduled.

---

Anatomy of a Scripted Input in inputs.conf

A scripted input instructs Splunk's background daemon (splunkd) to periodically invoke an external script or executable binary, capture its generated output, and stream that output into the data ingestion pipeline.

Where Scripts Must Live

inputs.conf.spec restricts the command to one of these directories:

• $SPLUNK_HOME/etc/system/bin/
• $SPLUNK_HOME/etc/apps/<app_name>/bin/
• $SPLUNK_HOME/bin/scripts/

The path can be absolute, can use $SPLUNK_HOME, or can start with . to mean "inside the current app" (for example ./bin/collect.sh). To run a program located elsewhere, place a .path file in one of the allowed directories. It is a one-line pointer to the real command and its arguments, and it may point anywhere on the file system.

# App-relative script (recommended)
[script://./bin/collect_cloud_metrics.sh]
interval = 300
sourcetype = cloud:metrics
index = ops_metrics

# Pointer file for a program outside the allowed directories
[script://$SPLUNK_HOME/etc/apps/org_net_audit/bin/network_audit.path]
interval = 0 2 * * *
sourcetype = net:audit
index = security

---

Scheduling Mechanisms: Seconds, Cron Expressions, and Continuous Scripts

The interval setting in inputs.conf controls when splunkd runs the script:

Periodic vs. Continuous Scripts

1. Periodic scripts (interval > 0 or cron): the script runs, writes events to standard output, and exits. This suits polling an API, collecting system statistics, or checking disk space.
2. Continuous scripts (interval = 0): the script is expected to keep running, for example streaming from a message queue. If it exits or crashes, Splunk starts it again immediately, so a failing script can restart in a tight loop. Make sure errors are logged to standard error.

---

Execution Environment & Process Model

Scripted inputs are managed by splunkd's internal execProcessor subsystem. The runtime environment possesses specific operating system characteristics that administrators must take into account:

1. Operating System User Context & Permissions

• A scripted input runs as the same operating system account as splunkd: typically the splunk account on Linux, and on Windows the account the Splunk service runs as (a virtual account, Local System, or a domain account).
• The script has no more and no less permission than that OS user:
  • If a script attempts to read /var/log/messages or query hardware devices that require root permissions, the execution will fail with permission denied errors.
  • Sudo privileges should be avoided; if required, configure strict /etc/sudoers rules for the specific command.

2. Runtime Environment Variables

Scripts run with Splunk's environment, including SPLUNK_HOME, so they can build paths relative to the installation instead of hard-coding them. Run splunk envvars to see the variables Splunk sets, and use splunk cmd <program> to run a program with that environment when testing.

3. Python Runtime & Environment Isolation

Splunk Enterprise (including heavy forwarders) bundles its own Python 3 runtime ($SPLUNK_HOME/bin/splunk cmd python3). The universal forwarder does not. On a UF, a scripted input must be an executable, shell, batch, or PowerShell script, or must call an interpreter already installed on the host.

• For Python scripts on Splunk Enterprise, python.version in the input stanza can select the Python version.
• Splunk's environment points library paths at Splunk's own libraries, which can conflict with system programs or third-party Python packages that a script calls.
• The Dependency Conflict Hazard: If a script relies on external Python modules (e.g., requests, boto3, or database drivers) that are not bundled with Splunk, administrators must either:
  1. Vendor the third-party libraries directly into the application's bin/lib/ directory and append that directory to sys.path within the script.
  2. Use a shell wrapper script that invokes the host system's native Python virtual environment (venv), sanitizing LD_LIBRARY_PATH and PYTHONPATH prior to execution.

4. REST API Authentication via passAuth

Often, a scripted input needs to query Splunk's own internal REST API (TCP port 8089) to fetch configuration settings, trigger saved searches, or inspect cluster state. Hard-coding an admin password in a script file is a serious security risk.

Splunk solves this using the passAuth parameter in inputs.conf:

[script://./bin/audit_forwarders.py]
disabled = false
interval = 3600
passAuth = splunk-system-user

• When passAuth is configured, splunkd generates a temporary, short-lived session token with the privileges of the specified user (e.g., splunk-system-user or admin).
• Prior to executing the script, splunkd writes this session token to the child process's standard input (stdin).
• The script reads the token from stdin and passes it in the Authorization header when querying https://localhost:8089:

  import sys, urllib.request, ssl
  
  # Read session token injected via stdin by passAuth
  session_key = sys.stdin.readline().strip()
  
  req = urllib.request.Request("https://localhost:8089/services/server/info")
  req.add_header("Authorization", f"Splunk {session_key}")
  # Execute authenticated REST call...
  

---

Standard Streams: stdout vs. stderr and The Buffering Trap

The interaction between splunkd and the script's output streams is rigidly segregated:

                               +----------------------------+
                               |   Scripted Input Process   |
                               | (Python / Bash / Binary)   |
                               +----------------------------+
                                  |                      |
                    stdout Stream |                      | stderr Stream
                                  v                      v
        +-----------------------------------+   +-----------------------------------+
        |        Splunk execProcessor       |   |       Splunk Logging Engine       |
        |  (Injects into Ingestion Pipeline)|   |  (Captures diagnostic messages)   |
        +-----------------------------------+   +-----------------------------------+
                          |                                       |
                          v                                       v
                inputQueue -> parsingQueue               $SPLUNK_HOME/var/log/splunk/
                (Indexed as Event Data)                  splunkd.log (WARN / ERROR)

1. Standard Output (stdout) -> Ingestion Pipeline

Everything the script writes to stdout is captured by execProcessor and injected directly into the Splunk ingestion pipeline:

• The byte stream enters inputQueue tagged with the source, sourcetype, index, and host defined in the inputs.conf stanza.
• It then passes to parsingQueue, where line breaking (LINE_BREAKER), timestamp extraction (TIME_PREFIX, TIME_FORMAT), and transformations occur.
• Best practice: Format script output as clean, single-line structured strings (e.g., key-value pairs or JSON objects) containing explicit timestamps to streamline indexing.

2. Standard Error (stderr) -> splunkd.log

Anything written to stderr bypasses the ingestion pipeline completely:

• splunkd intercepts stderr and writes it directly to $SPLUNK_HOME/var/log/splunk/splunkd.log.
• Log entries appear under the ExecProcessor component with a log level of ERROR or WARN:

  09-23-2026 14:35:10.124 -0400 ERROR ExecProcessor [14205 ExecProcessor] - 
  message from "/opt/splunk/etc/apps/TA-cloud/bin/collect.py" urllib.error.URLError: <urlopen error [Errno 110] Connection timed out>
  

• Administrators troubleshooting failing scripted inputs should immediately inspect splunkd.log for ExecProcessor messages: index=_internal sourcetype=splunkd component=ExecProcessor

3. The Critical Developer Trap: Output Buffering

In many programming languages (especially Python and C/C++), standard output is block-buffered rather than line-buffered when redirected to an operating system pipe.

• When splunkd executes a script, stdout is connected to a pipe, not a terminal (TTY).
• If a Python script prints events in a loop, Python may hold the data in an internal 4 KB or 8 KB memory buffer instead of writing it immediately to the pipe.
• A periodic script flushes its buffer when it exits, but a continuous script (interval = 0) that never exits delivers events only when the buffer fills, so they arrive late and in bursts.
• The Fix: Scripts must explicitly flush stdout after printing events:
  • In Python 3: print(event_data, flush=True) or sys.stdout.flush().
  • Or set PYTHONUNBUFFERED=1 inside a wrapper script before launching the interpreter.

---

Script Error Handling, Timeouts, and Resource Leaks

Failing to design scripted inputs defensively can destabilize the entire host system:

1. Non-Zero Exit Codes

A script that fails usually leaves evidence in two places: the messages it wrote to standard error, and the ExecProcessor entries in splunkd.log. A scheduled script simply runs again at its next interval; a continuous script (interval = 0) is restarted as soon as it exits.

2. Execution Overlap Prevention

If a script scheduled with interval = 60 sometimes takes 90 seconds, do not rely on Splunk to manage the overlap.

• Design for it: Keep each run shorter than the interval, and give network calls explicit timeouts. A script that hangs keeps its process and resources until it exits or Splunk stops.

3. Hanging Processes and Resource Leaks

Splunk does not know when a script is "stuck". A script that waits forever on the network keeps running.

• If a script initiates a network connection to a third-party server that hangs indefinitely without a TCP timeout, the script process remains alive in the operating system process table indefinitely.
• Over time, hung scripts can hold process slots, file handles, sockets, and memory on the host.
• Defensive Coding Requirements:
  • Always specify explicit timeouts on all network socket connections, HTTP requests, and database queries (e.g., requests.get(url, timeout=15)).
  • Implement signal handlers (SIGTERM, SIGINT) in scripts to ensure graceful shutdown when splunkd stops or restarts.

---

Windows PowerShell Inputs

On Microsoft Windows platforms, administrative data collection frequently relies on PowerShell. While administrators can execute PowerShell via generic [script://...] stanzas (e.g., calling powershell.exe -File script.ps1), Splunk on Windows provides a native mechanism: the PowerShell input ([powershell://<name>]), which runs Windows PowerShell (version 3 or later) commands or scripts.

Stanza Syntax in inputs.conf

[powershell://InstalledSoftware]
script = Get-ItemProperty HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall\* | Select-Object DisplayName, DisplayVersion, Publisher, InstallDate
schedule = 0 0 * * *
sourcetype = Windows:InstalledSoftware
index = windows_inventory

[powershell://QueryADUsers]
script = . "$SplunkHome\etc\apps\TA-Windows\bin\get_locked_accounts.ps1"
schedule = */15 * * * *
sourcetype = ActiveDirectory:LockedUsers
index = msad

Key Parameters for PowerShell Inputs

• script: Contains either inline PowerShell cmdlets/pipeline expressions or a dot-sourced path to an external .ps1 script file.
• schedule: How often to run, in seconds or as a cron schedule. Whichever you choose, the command also runs once when the instance starts. If schedule is not set, it runs only at startup.
• sourcetype: Stamped on all resulting output.

PowerShell Execution Policies & Security Controls

Windows enforces execution policies to prevent unauthorized script execution:

• Policies: Restricted, AllSigned, RemoteSigned, Unrestricted, Bypass.
• Enterprise Group Policy (GPO) Overrides:
  • Domain Group Policy can enforce machine-wide execution policies, so test scripts under the target machine's effective policy.
  • Environments that use AppLocker or Windows Defender Application Control (WDAC) may force PowerShell into Constrained Language Mode, which blocks some .NET and COM features that complex scripts use.

Service Account Privileges & Active Directory Data Collection

Since version 9.1, the Windows universal forwarder installer creates and uses a least-privileged virtual account, NT SERVICE\SplunkForwarder, by default. You can instead choose Local System or a domain account during installation.

• Local accounts vs. the domain: Neither the virtual account nor Local System carries a domain user's identity. Both reach network resources as the computer account (DOMAIN\COMPUTER$), which usually cannot query domain controllers or remote systems the way an administrator expects.
• Domain Service Account Configuration:
  • When configuring a forwarder or Heavy Forwarder to harvest Active Directory topology, group membership changes, DNS server records, or Hyper-V metrics across multiple domain systems, the Splunk service must be configured to run as a dedicated Domain Service Account or a Group Managed Service Account (gMSA).
  • Following the principle of least privilege, the domain account does not need Domain Admin rights; it should be granted delegated read permissions on required Active Directory organizational units (OUs) and placed in the built-in Event Log Readers and Performance Monitor Users security groups.

---

Technical Comparison: Scripted Inputs vs. PowerShell Inputs