12.1 File & Directory Monitoring Mechanics

File & Directory Monitoring Mechanics

Quick Summary: Ingesting log data from local files and directory hierarchies is the most foundational task in Splunk Enterprise administration. Configured through [monitor://<path>] stanzas in inputs.conf, file monitoring is driven by the splunkd tailingProcessor pipeline. This engine continuously monitors active files, maintains byte-level read offsets in a specialized tracking database known as the fishbucket, identifies log rotations without duplicating data, and navigates complex directory structures using precise single-level and recursive wildcards.

---

In-Depth Syntax and Structure of [monitor://<path>]

In Splunk Enterprise and the Universal Forwarder, the primary configuration stanza for continuous data ingestion from the filesystem is [monitor://<path>], defined within inputs.conf. Unlike batch or one-time inputs, a monitor input establishes a continuous watch over the target file or directory tree, ingesting existing contents and tailing newly appended data in near real time.

Stanza Path Conventions and Normalization

When defining monitor paths, administrators must adhere to strict platform syntax and path normalization rules:

# Linux / Unix absolute file monitoring
[monitor:///var/log/messages]
disabled = false
index = os
sourcetype = syslog

# Linux / Unix directory tree monitoring
[monitor:///var/log/httpd]
disabled = false
index = web
sourcetype = access_combined

# Windows local filesystem monitoring
[monitor://C:\inetpub\logs\LogFiles]
whitelist = \.log$
disabled = false
index = iis
sourcetype = ms:iis:auto

Key Path Rules

1. Windows Paths: On Windows, the path starts with the drive letter, as in [monitor://C:\inetpub\logs\LogFiles]. Remember that whitelist and blacklist values are regular expressions, so a literal backslash inside them must be escaped as \\.
2. Leading Slashes on POSIX Systems: On Linux and Unix platforms, an absolute path must begin with three forward slashes: two slashes indicating the protocol scheme (monitor://) and the third denoting the filesystem root directory (/), yielding [monitor:///path/to/log].
3. Directory vs. File Targets: If <path> references an individual file, Splunk opens and tails that specific file. If <path> references a directory, Splunk automatically monitors every readable file within that directory and, by default, recurses through all child directories unless constrained by wildcards or tuning settings.
4. Metadata Assignment: Within each monitor stanza, administrators assign critical indexing and parsing metadata:
   • index: Specifies the destination target index on the indexers (e.g., index = security). If omitted, data routes to the default index (typically main).
   • sourcetype: Assigns the format classification that governs event breaking, timestamp parsing, and field extraction (e.g., sourcetype = cisco:asa). If omitted, Splunk assigns a source type automatically, which may not be the one you want.
   • source: Overrides the default source value, which is the file's full path. Overriding it is rarely needed for monitored files.
   • host: Sets the originating host value, overriding the server's local hostname when ingesting third-party or relayed telemetry.
   • disabled: Boolean toggle (false enables the input; true disables it without deleting the configuration stanza).

---

Wildcard Syntax: Single-Segment (*) vs. Recursive Traversal (...)

File systems in enterprise environments frequently partition logs by application instance, date, tenant, or customer ID. Splunk provides two distinct wildcard mechanisms in monitor stanzas to traverse these directory structures: the single asterisk (*) and the recursive ellipsis (...). Mastering the precise behavioral difference between these operators is crucial for avoiding unintended data ingestion and performance degradation.

Wildcard Comparison:
/var/log/*/*.log    --> Matches exactly ONE directory level deep
                        Matches: /var/log/apache/access.log
                        Does NOT match: /var/log/apache/ssl/access.log

/var/log/.../*.log  --> Matches ANY number of nested subdirectories
                        Matches: /var/log/apache/access.log
                        Matches: /var/log/apache/ssl/access.log
                        Matches: /var/log/apache/ssl/external/access.log

The Single Asterisk (*)

The single asterisk * represents a single-segment wildcard. It matches zero or more characters within a single directory segment or within a filename, but it never crosses directory separator boundaries (/ or \).

• In a filename position: [monitor:///var/log/app_*.log] matches /var/log/app_web.log and /var/log/app_api.log, but does not match /var/log/app_sub/other.log.
• In a directory position: [monitor:///var/log/*/catalina.out] matches /var/log/tomcat1/catalina.out and /var/log/tomcat2/catalina.out. However, it strictly refuses to match /var/log/cluster/node1/catalina.out because that path requires crossing two directory separators.

The Recursive Ellipsis (...)

The triple-dot ellipsis ... represents a recursive directory traversal wildcard. It matches zero, one, or multiple nested directory levels deep, recursing down through an arbitrary directory tree until it finds files matching the subsequent path pattern.

• In an intermediate path position: [monitor:///opt/apps/.../*.log] discovers and monitors .log files in /opt/apps/, /opt/apps/auth/, /opt/apps/auth/svc/, and arbitrarily deep directory hierarchies.
• In a trailing position: [monitor:///var/log/audit/...] monitors every file and child directory beneath /var/log/audit/, regardless of nesting depth.

Wildcard Matching Matrix

To understand how * and ... evaluate real-world filesystem paths, consider the following matrix:

[!WARNING] Using broad recursive monitor stanzas such as [monitor:///var/log/...] without whitelists or blacklists can inadvertently ingest binary system dumps, rotated compressed archives, or rapidly churning temporary files, exhausting indexing licenses and degrading search performance.

---

Tailing Processor Mechanics: Continuous File Tracking

monitor inputs are handled by the tailing processor (TailingProcessor in splunkd.log). It discovers files under the monitored paths, reads new data as it is appended, and records its progress in the fishbucket so that restarts neither lose nor repeat data.

How the Monitor Processor Works

• When you name a file, Splunk consumes new data written to it. When you name a directory, it recursively examines the subdirectories (recursive = true by default) for new files it can read.
• A file or directory that does not exist when Splunk starts is checked again every 24 hours from the last restart. Subdirectories of monitored directories are scanned continuously.
• Splunk uses memory for every file it tracks, including ignored files. On directories with huge numbers of old files, use a narrower path, a whitelist/blacklist, or ignoreOlderThan.
• Monitor inputs may overlap. As long as the stanza names differ, each is independent, and a file is handled by the most specific matching stanza.
• Disabling or deleting a monitor input does not stop files already being read. It only stops Splunk from checking those files again. Restart to stop in-process indexing.
• Limits: files whose path is longer than 1,024 characters (256 on Windows) are not monitored, and files with a .splunk extension are skipped.

How Progress Is Stored: The Fishbucket

The fishbucket ($SPLUNK_DB/fishbucket/splunk_private_db, visible as the _thefishbucket index) stores, for each file:

• a beginning CRC of the first 256 bytes (initCrcLength), which is the lookup key,
• the seekAddress (how many bytes have been read), and
• a seekCRC of the data at that position.

After a restart, the processor recomputes each file's beginning CRC, finds the record, checks that the data at the seekAddress still matches the seekCRC, and continues from there.

---

Log Rotation Detection Mechanics

Log rotation (Linux logrotate, Log4j/Logback rolling) keeps files from growing without limit. Splunk recognizes rotated files by content (CRC), not by file name or inode.

Move-and-Recreate (Rename) Rotation

Step 1: Active logging to /var/log/app.log
Step 2: logrotate renames it: mv /var/log/app.log /var/log/app.log.1
Step 3: a new, empty /var/log/app.log is created and the application writes to it

1. The renamed app.log.1 has the same first 256 bytes as the file Splunk was reading, so its beginning CRC matches an existing fishbucket record. Its content at the seekAddress matches the seekCRC, and it has no data beyond what was read, so Splunk does not read it again.
2. The new app.log starts with different content, so its beginning CRC is new. Splunk treats it as a new file and reads it from the start.
3. This works only if the rotated file stays within a monitored path, or if Splunk finished reading it before the rename. Data that was still unread when the file moved out of the monitored path is not collected.

Compressed rotated copies (.gz, .tar) are the documented exception: Splunk processes renamed archives. That is why many teams blacklist \.gz$ in directories where rotation compresses old files.

Copy-Truncate (copytruncate) Rotation

Step 1: /var/log/app.log is copied to /var/log/app.log.1
Step 2: /var/log/app.log is truncated in place and the application keeps writing to it

• The copy app.log.1 has the same beginning CRC as the data already read, so it is recognized and not read again.
• The truncated app.log soon holds new opening content with a new beginning CRC, so Splunk reads it as a new file.
• Anything written between the copy and the truncate can be lost, and a partially read file can be read twice. Prefer move-and-recreate for important audit logs.

[!CAUTION] Do not add crcSalt = <SOURCE> to monitors of rotating files. It puts the path into the CRC, so a renamed file looks brand new and is indexed again.

---

Symbolic Link Handling: Hazards, Loops & Traversal Risks

Symbolic links (symlinks) are pointers in a filesystem that reference another file or directory path. In inputs.conf, the followSymlink setting controls whether Splunk traverses symlinks encountered during directory monitoring:

[monitor:///var/log/apps]
followSymlink = false

Default Behavior and Operational Hazards

followSymlink defaults to true in inputs.conf. While symlink traversal provides administrative convenience when logs are distributed across multiple storage volumes, it introduces severe operational hazards:

1. Symlinks That Point Back Into the Tree

If a monitored directory contains a symbolic link that points back to itself or to an ancestor directory (for example, /var/log/apps/archive pointing to /var/log/apps), the same files are reachable through ever-longer paths:

/var/log/apps/archive/archive/archive/archive/...

At best this wastes effort walking the same directories; at worst it makes file tracking confusing and hard to troubleshoot. Keep monitored trees free of such links, or set followSymlink = false.

2. Duplicate Data Ingestion

If an administrator configures a monitor stanza for /var/log/httpd/access.log and another monitor stanza for /var/www/logs/, where /var/www/logs/access.log is a symlink pointing to /var/log/httpd/access.log, Splunk can ingest the same data through both stanzas if crcSalt = <SOURCE> makes the two paths look like different files. This doubles indexing volume, artificially inflates licensing usage, and corrupts search results with duplicate events.

3. Security and Directory Traversal Vulnerabilities

Following symlinks can also expose files outside the intended directory. If an unprivileged application user creates a symlink inside a monitored directory pointing to sensitive system files:

ln -s /etc/shadow /var/log/myapp/debug.log

If the forwarder runs as root and followSymlink = true (the default), Splunk follows the link and can index sensitive password hashes. Per the spec, whitelists and blacklists also apply to files at the symlink destination, so a tight whitelist limits the damage.

Recommended Best Practices for Symlinks

1. Disable Symlink Traversal by Default: Set followSymlink = false on broad directory monitor stanzas unless symlinks are strictly necessary for the application architecture.
2. Explicit Target Monitoring: If specific symlinked targets must be ingested, create dedicated, tightly scoped [monitor://<explicit_target>] stanzas pointing directly to the real underlying path rather than enabling recursive symlink following on parent directories.