12.3 Deploying a Remote Monitor Input, Network Shares & Archives

Deploying a Remote Monitor Input

Blueprint objective 12.3 is "Deploy a remote monitor input". In a distributed deployment you rarely add a monitor input on the machine where you are logged in. Instead you create the input centrally and let the deployment server deliver it to the forwarders that sit next to the data.

Method 1: Splunk Web "Add Data > Forward"

When a Splunk Enterprise instance is also a deployment server with connected deployment clients, Settings > Add Data > Forward walks you through a remote input:

1. Prerequisite: at least one forwarder must be configured as a deployment client of this instance, for example with splunk set deploy-poll <ds>:8089 and a restart. Otherwise the page reports that no deployment clients were found.
2. Select Forwarders: choose New or Existing server class, move hosts from Available host(s) to Selected host(s), and name the server class. A server class must contain hosts of a single platform; you cannot mix Windows and *nix hosts.
3. Select Source: choose the data source for the forwarders, for example Files & Directories with the path to monitor.
4. Input Settings: set the source type, app context, host, and destination index.
5. Review and Submit. Splunk stores the input in a deployment app mapped to the server class, and the forwarders download it the next time they phone home.

Splunk documents this page for a single Splunk Enterprise instance that acts as both the indexer and the deployment server. With several indexers, the page isn't useful, and you build the deployment app yourself (Method 2).

Method 2: A Deployment App Built by Hand

$SPLUNK_HOME/etc/deployment-apps/org_linux_web_inputs/
└── local/
    └── inputs.conf

# inputs.conf inside the deployment app
[monitor:///var/log/nginx/access.log]
index = web
sourcetype = nginx:access
disabled = false

# serverclass.conf on the deployment server
[serverClass:linux_web]
whitelist.0 = web-*
machineTypesFilter = linux-x86_64

[serverClass:linux_web:app:org_linux_web_inputs]
restartSplunkd = true
stateOnClient = enabled

Then run splunk reload deploy-server, or save the same mapping in forwarder management. Each matching forwarder downloads org_linux_web_inputs into $SPLUNK_HOME/etc/apps/, restarts, and starts monitoring the file. Data flows to the indexers through the forwarder's existing outputs.conf, which is often delivered by a separate outputs app.

Verifying the Remote Input

• Forwarder management shows that the clients downloaded the app.
• On a forwarder, splunk list monitor shows the monitored files, and splunk btool inputs list --debug shows the stanza came from the deployment app.
• On the search head, run index=web sourcetype=nginx:access host=web-* to confirm events arrive. If they do not, check the forwarder's _internal logs for the tailing processor and its output connections.

Monitoring Remote File Systems (NFS, SMB/CIFS)

Splunk's documentation allows a monitor input on a mounted or shared directory, including network file systems, as long as the forwarder can read it. It is supported, but it has trade-offs:

• The monitoring host has to open, read, and re-check files across the network, which adds latency and load on the file server.
• Network interruptions and file-system client caching can make files appear to change size or content unexpectedly, which can cause gaps or duplicate reads.
• On Windows, use UNC paths or local drives the service account can reach, and run the forwarder as a user that has access to the share.

Where you can, install a universal forwarder on the machine that writes the logs and monitor the files locally. That is simpler, avoids network file-system behavior, and lets the forwarder use indexer acknowledgment on its outbound connection.

Compressed Archive Processing Mechanics

Log archiving pipelines often package older log segments into compressed archives. Splunk's forwarders decompress archive files before processing them, and they process archives single-threaded.

Supported Archive Formats

• TAR, TAR.GZ / TGZ, TBZ / TBZ2
• GZ, BZ2
• ZIP
• Z

How Archives Are Processed

1. The monitor input recognizes the archive type, and the archive processor decompresses it before the contents enter the parsing pipeline.
2. props.conf can customize this with unarchive_cmd (in [source::...] stanzas) and unarchive_sourcetype.
3. Archive processing is single-threaded, so very large archives take time.

The Immutability Requirement for Archives

A critical operational distinction between plain text logs and compressed archives is that Splunk cannot tail an active or appending compressed archive.

• Plain text files can be tailed because new bytes are appended after data already read, and the seek address stays valid.
• A compressed archive has to be decompressed as a whole, so there is no "new data after the seek address" to read.
• Splunk's documentation is explicit: if you add new data to an existing archive file, the forwarder reprocesses the entire file rather than just the new data, which can duplicate events.

Practical rule: Build and compress archives outside the monitored directory, then move each finished archive in. Never append to an archive that Splunk has already read.

Batch Inputs & The Sinkhole Policy

While [monitor://...] keeps watching files as they grow, the batch input ([batch://<path>]) loads files once and destructively. For a single static file you can also use the CLI commands splunk add oneshot or splunk spool.

The Batch Ingestion Architecture

A batch input is defined in inputs.conf using the [batch://<path>] stanza:

[batch:///var/spool/splunk_drop/*.csv]
disabled = false
index = historical_data
sourcetype = csv:billing
move_policy = sinkhole

Unlike monitor inputs, which keep tracking files, a batch input is a one-time, destructive loader. Splunk also has a built-in spool directory, $SPLUNK_HOME/var/spool/splunk, where any file you drop in is processed and deleted.

The Mandatory move_policy = sinkhole

The defining characteristic of a batch input is the move_policy setting. The spec marks it as required: you must include move_policy = sinkhole in every batch stanza:

move_policy = sinkhole

What the Sinkhole Policy Does

When move_policy = sinkhole is configured:

1. Splunk discovers the file in the designated batch directory.
2. Splunk reads the whole file and sends its events through the pipeline like any other input.
3. After reading the file, Splunk deletes it. That is what "sinkhole" means, and inputs.conf.spec requires move_policy = sinkhole for batch inputs.

Why It Matters

Batch inputs are meant for files that should disappear once loaded. Because the policy is destructive, point them only at a dedicated drop directory, and make sure the Splunk service account can delete files there.

[!CAUTION] Filesystem Permissions and Staging Hazards:

1. Delete Permissions: The operating system user running splunkd (e.g., splunk) must possess both read AND write/delete permissions on the target directory and files. If Splunk cannot delete a file, the file stays in the drop directory, and the input cannot work as intended.
2. Never Point Batch at Master Copies: Because sinkhole permanently destroys the source file upon ingestion, never point a [batch://...] input at a directory containing original, master, or live log files. Batch inputs must only monitor dedicated drop folders (such as /var/spool/splunk_drop/) where automated ETL workflows deposit disposable copies of historical archives.