5.2 User Account Administration, Search Restrictions & Limits

User Account Administration, Search Restrictions & Limits

Quick Summary: Safeguarding Splunk Enterprise resources and enforcing multi-tenant isolation requires a combination of user lifecycle governance, index access boundaries, search filter scoping, and concurrency quotas. Using authorize.conf and authentication.conf, administrators can restrict which indexes a role can search (srchIndexesAllowed), control default search targets (srchIndexesDefault), enforce row-level data filtering transparently (srchFilter), and throttle historical and real-time search concurrency (srchJobsQuota, cumulativeSrchJobsQuota).

---

User Account Administration & Lifecycle Management

Splunk Enterprise maintains local user accounts directly on the instance, managed either through the Splunk Web user interface or via the Command Line Interface (CLI).

Account Lifecycle via Splunk Web

Administrators manage local accounts by navigating to Settings > Users:

• New User Creation: Requires specifying a Username, Full Name, Email Address, Time Zone, Default App (e.g., search), and assigning one or more initial Roles.
• Locked Accounts: Native accounts lock automatically after repeated failed logins (see the lockout settings below). An administrator unlocks them from Settings > Users.
• Password Reset: Administrators can set a new password and tick Require password change on first login.

CLI User Administration

The CLI provides automated, scriptable commands for managing local accounts via $SPLUNK_HOME/bin/splunk:

# Create a new local user with role assignments
splunk add user jdoe -password "TemporaryP@ss123" -role user -role soc_tier1 -full-name "John Doe" -email "jdoe@example.com" -default-app search

# Modify existing user role assignments (repeat -role for each role)
splunk edit user jdoe -role user -role soc_lead

# Change a user password (requires admin authentication credentials)
splunk edit user jdoe -password "NewSecureP@ssword!" -auth admin:AdminPassword123

# Display detailed attributes of a specific user
splunk list user jdoe

# Remove a local user account
splunk remove user jdoe

Local Credential Storage: $SPLUNK_HOME/etc/passwd

Native accounts are stored in $SPLUNK_HOME/etc/passwd, one colon-separated record per user holding the username, password hash, full name, roles, and email address.

• Passwords are stored as salted hashes. passwordHashAlgorithm in authentication.conf controls the algorithm, and the options are SHA512-crypt (the current default family), SHA256-crypt, and legacy MD5-crypt.
• Warning: Direct manual editing of $SPLUNK_HOME/etc/passwd is strongly discouraged while splunkd is running. Use the CLI or REST API to ensure atomic file locking and configuration cache reloading.

Account Lockout and Password Policy in authentication.conf

Native-authentication lockout and password rules live in the [splunk_auth] stanza, not in [authentication]:

[splunk_auth]
lockoutUsers         = true   # default: lockout enabled
lockoutAttempts      = 5      # failed logins allowed (default 5, max 64)
lockoutThresholdMins = 5      # window for counting failures (default 5, max 120)
lockoutMins          = 30     # how long the account stays locked (default 30, 1-1440)
minPasswordLength    = 8      # default 8
expirePasswordDays   = 90     # default 90

lockoutMins has a minimum of 1 minute. There is no "lock forever until an admin clears it" value; administrators can unlock an account sooner from Settings > Users. In a search head cluster, lockout state applies per member, not cluster-wide.

---

Restricting Index Access: srchIndexesAllowed vs. srchIndexesDefault

Controlling which data stores an analyst can search is managed through two primary parameters in authorize.conf under the [role_<role_name>] stanza:

[role_security_analyst]
srchIndexesAllowed = security;firewall;wineventlog;*
srchIndexesDefault = security;firewall

1. srchIndexesAllowed (Search Boundary)

• Specifies the white-list of indexes that a user assigned to this role has permission to search.
• If a user submits a query targeting an index not present in their srchIndexesAllowed list (e.g., index=payroll), Splunk blocks access, returning either a permission error or zero search results.

2. srchIndexesDefault (Unqualified Target)

• Specifies which indexes Splunk queries when a user submits an unqualified search—meaning a search that does not explicitly contain an index=... clause.
• For example, if a user runs fail* OR error, Splunk automatically routes that search exclusively across the indexes defined in srchIndexesDefault.
• Keep every index in srchIndexesDefault inside srchIndexesAllowed; a default index that the role is not allowed to search returns nothing.

Wildcard Mechanics: Non-Internal (*) vs. Internal (_*)

A critical distinction exists in how Splunk evaluates wildcards within srchIndexesAllowed:

• The Asterisk Wildcard (*): Matches all non-internal (user-created and application) indexes. It matches main, security, firewall, web, os, etc. Crucial Rule: The asterisk wildcard * never matches internal indexes beginning with an underscore (_).

• The Internal Wildcard (_*): Matches all internal system indexes: _internal, _audit, _introspection, _telemetry, _thefishbucket.

• Combined Wildcard: To permit access to all data on the Splunk instance (both public data and internal system diagnostic logs), the administrator must explicitly declare both:

  srchIndexesAllowed = *;_*
  

Security Implications of Internal Indexes (_internal & _audit)

Granting access to _* or _audit to non-administrative users carries significant operational and compliance risks:

• _audit Index: Contains the complete history of all SPL queries executed by all users in the organization. If users search for sensitive parameters (such as user=jdoe password=Secret or SSNs), those queries are recorded in plaintext in _audit. Analysts with access to _audit can view those sensitive strings.
• _internal Index: Contains system paths, license keys, forwarder hostnames, internal stack traces, and REST API access logs.

Index Access Parameter Matrix

---

Data Scoping with srchFilter

The srchFilter directive in authorize.conf enables administrators to enforce row-level security and data segregation without relying on users to write accurate filtering clauses.

How srchFilter Operates at Search Parse Time

When a user assigned a role with a srchFilter submits a search, the Splunk search parser automatically injects the filter into the search string using a boolean conjunction (AND):

# Stanza in authorize.conf
[role_emea_analyst]
srchFilter = region=EMEA AND sourcetype!=cisco:debug

1. User submits query in search bar: index=network status=blocked
2. Splunk search parser compiles the effective execution query:

   ( index=network status=blocked ) ( region=EMEA AND sourcetype!=cisco:debug )
   

3. The conjoined query is distributed to indexers over TCP 8089.
4. Events that do not match region=EMEA or that contain sourcetype=cisco:debug are discarded during the indexer filtering phase before results are returned to the Search Head.

Multi-Role Evaluation Semantics (Logical OR)

When a user account is assigned multiple roles, Splunk combines their respective search filters using a logical OR disjunction. This ensures that users receive the least-restrictive union of data across all their assigned functional roles:

• Role role_us_sales: srchFilter = department=sales country=US
• Role role_uk_support: srchFilter = department=support country=UK
• If user asmith is assigned both roles, Splunk constructs the effective filter:

  ( ( original_search ) ) AND ( ( department=sales country=US ) OR ( department=support country=UK ) )
  

Performance Hazards of srchFilter

While powerful, poorly configured srchFilter definitions can severely degrade cluster search performance:

• Prefer Indexed Terms: Filters that use indexed fields (index, source, sourcetype, host, event types, tags) are efficient. The spec also allows search fields, but those filter later and cost more.
• The Wildcard Penalty: Using leading wildcards or unindexed regex in a search filter (e.g., srchFilter = *Confidential*) prevents indexers from leveraging .tsidx time-series indexes. The indexers are forced to decompress and scan the raw data journal of every bucket, generating massive disk I/O bottlenecks.

---

Search Concurrency Limits & Resource Allocation

Uncontrolled search activity can quickly exhaust Search Head CPU cores, saturate memory, and fill up local disk volumes. Splunk provides granular throttling directives in authorize.conf:

[role_contractor]
# Individual user limits (a value of 0 means unlimited)
srchJobsQuota = 2
srchDiskQuota = 250

# Collective group limit (requires enable_cumulative_quota = true in limits.conf)
cumulativeSrchJobsQuota = 10

Resource Directives Explained

1. srchJobsQuota (Per-User Historical Search Limit): Defines the maximum number of concurrent historical (disk-based) searches that an individual user belonging to this role can execute simultaneously. If a user exceeds this limit, subsequent searches enter a queued state or fail with an error.

2. rtSrchJobsQuota (Per-User Real-time Search Limit): Defines the maximum number of concurrent real-time searches an individual user can run. Real-time searches run continuously and consume resources on the search head and indexers. A value of 0 means no limit. To stop a role from running real-time searches, do not give it the rtsearch capability (the built-in user role lacks it).

3. cumulativeSrchJobsQuota (Team-Wide Historical Limit): Establishes an aggregate ceiling for all users belonging to this role combined. It takes effect only when enable_cumulative_quota = true in limits.conf. For example, if a team has 50 contractors each allowed 2 searches (srchJobsQuota = 2), setting cumulativeSrchJobsQuota = 10 ensures that the contractor team as a whole cannot consume more than 10 concurrent search slots on the Search Head, protecting system capacity for core operational teams.

4. cumulativeRTSrchJobsQuota (Team-Wide Real-time Limit): Limits the total number of real-time searches running concurrently across all members of the role.

5. srchDiskQuota (Dispatch Storage Limit per User): Specifies the maximum total disk space (in megabytes) that an individual user's search artifacts can consume in $SPLUNK_HOME/var/run/splunk/dispatch/. Defaults are 100 MB for user, 500 MB for power, and 10,000 MB for admin; 0 means unlimited.

System Search Concurrency Capacity Formula

Individual role quotas carve out slices of the Search Head's global concurrency capacity, configured in limits.conf under the [search] stanza:

$\text{Total Max Concurrent Searches} = (\text{max\_searches\_per\_cpu} \times \text{CPU Cores}) + \text{base\_max\_searches}$

• Default values: max_searches_per_cpu = 1, base_max_searches = 6.
• On a 16-core Search Head, the baseline capacity is: $(1 \times 16) + 6 = 22 \text{ concurrent historical searches}$
• If administrators grant high srchJobsQuota values to too many users without setting cumulativeSrchJobsQuota, concurrent scheduled searches and dashboard refreshes will quickly saturate the 22 available execution threads.

---

The Dispatch Directory Lifecycle

When a search is dispatched, Splunk allocates a dedicated directory on the Search Head at: $SPLUNK_HOME/var/run/splunk/dispatch/<search_id>/

This directory holds search artifacts, including status.csv, results.csv.gz, execution timelines, and search logs (info.csv).

• Artifact Lifetime (TTL): Ad-hoc search artifacts expire after 10 minutes by default (ttl = 600 in limits.conf). Scheduled search artifacts default to twice the schedule period (dispatch.ttl = 2p), and triggered-alert artifacts follow alert.expires (default 24 hours).
• Quota Enforcement: Before dispatching a new search, splunkd measures the cumulative size of all active directories owned by that user in the dispatch path. If the total already exceeds srchDiskQuota, the new search is refused with a disk-quota message. A running search that crosses the quota is auto-finalized.

---

Administrative Anti-Patterns & Pitfalls

Anti-Pattern 1: Granting * in srchIndexesDefault

• Why this fails: Setting srchIndexesDefault = * means that whenever a user types a simple search like error, Splunk queries every single non-internal index in the enterprise (web, proxy, dns, metrics, endpoints). This causes massive disk I/O and CPU spikes across all indexers.
• The Correct Pattern: Set srchIndexesDefault strictly to the one or two primary indexes relevant to that role (e.g., main or security).

Anti-Pattern 2: Forgetting That * Excludes Internal Indexes

• Why this fails: An administrator creates a junior administrator role with srchIndexesAllowed = * and expects them to troubleshoot Splunk forwarders. When the junior admin searches index=_internal sourcetype=splunkd, no results appear, leading to false troubleshooting steps.
• The Correct Pattern: Explicitly specify _internal or _* in srchIndexesAllowed if internal log inspection is required.