8.2 Roles of the Search Head and Search Peers

The Roles of the Search Head and Search Peers

Blueprint objective 8.2 asks you to explain what the search head does and what the search peers do. The two roles split the work, and the knowledge bundle is what lets peers search on the search head's behalf.

Architectural Separation: Search Heads vs. Search Peers

Distributed search divides computational workloads between two distinct server tiers: Search Heads and Search Peers.

+-------------------------------------------------------------+
|                        SEARCH HEAD                          |
|  - User Interface (TCP 8000) & REST API (TCP 8089)          |
|  - SPL Parsing, AST Generation, & Execution Planning        |
|  - Knowledge Object Management & Bundle Creation            |
|  - Reduce Phase: Merging, Sorting, & Presentation           |
+-------------------------------------------------------------+
                               |
          TCP 8089 (REST API / HTTPS Management)
          - Knowledge Bundle Distribution (tar.gz)
          - Search Dispatch & Peer Status Checks
                               |
        +----------------------+----------------------+
        |                                             |
        v                                             v
+-------------------------------+   +-------------------------------+
|      SEARCH PEER (INDEXER 1)  |   |      SEARCH PEER (INDEXER 2)  |
|  - Ingestion (TCP 9997 s2s)   |   |  - Ingestion (TCP 9997 s2s)   |
|  - Time-series Index (*.tsidx)|   |  - Time-series Index (*.tsidx)|
|  - Rawdata Storage & Journal  |   |  - Rawdata Storage & Journal  |
|  - Map Phase: Filter & Eval   |   |  - Map Phase: Filter & Eval   |
+-------------------------------+   +-------------------------------+

1. The Search Head (Query Coordinator & Reducer)

A Search Head is a Splunk Enterprise instance dedicated to user interaction and analytical orchestration. It does not store user-indexed events locally. Its core administrative responsibilities include:

• Client Servicing: Hosting the Splunk Web graphical interface (TCP 8000) and serving REST API client requests.
• Authentication & Authorization: Authenticating users (via internal backends, LDAP, or SAML) and enforcing Role-Based Access Controls (RBAC) defined in authorize.conf.
• Query Compilation: Parsing user SPL strings, building execution plans, and splitting the search into a remote part (run on peers) and a local part (run on the search head).
• Knowledge Object Repository: Housing user-created and application-level knowledge objects—including saved searches, alerts, field extractions, lookups, macros, tags, and data models.
• Knowledge Bundle Packaging: Compressing active search-time knowledge objects into tarball archives and distributing them to connected search peers.
• The Reduce Phase: Aggregating, deduplicating, sorting, and calculating final statistical calculations on intermediate result streams received from search peers.

2. Search Peers (Data Indexers & Mappers)

A Search Peer is an Indexer that has established a trusted communication channel with one or more Search Heads. In distributed search, Search Peers act as workers executing localized data retrieval. Their responsibilities include:

• Data Storage: Writing raw compressed journals and time-series inverted index files (*.tsidx) across bucket directories (hot, warm, cold).
• Knowledge Bundle Execution: Storing the knowledge bundles received from search heads in $SPLUNK_HOME/var/run/searchpeers/ and using them (not the peer's own knowledge objects) for search-time field extractions and lookups.
• The Map Phase: Scanning local index buckets within the query's designated time boundary (earliest and latest), retrieving candidate events matching index-time keys, evaluating distributable streaming commands (such as eval, where, and rex), and computing partial summary aggregates.
• Streaming Results: Returning compressed intermediate results back to the Search Head over the management REST channel.

Search Head vs. Search Peer Responsibilities

Knowledge Bundle Replication Mechanics

Search peers run the search on the search head's behalf, so they need the search head's search-time knowledge: field extractions, lookups, event types, tags, macros, and so on. When a peer runs a distributed search it ignores its own local knowledge objects and uses only what is in the search head's knowledge bundle.

SEARCH HEAD                                      SEARCH PEER (INDEXER)
$SPLUNK_HOME/etc/system, apps, users             $SPLUNK_HOME/var/run/searchpeers/
          │  (subset of files)                                 ▲
          ▼                                                    │
$SPLUNK_HOME/var/run/<bundle>.bundle  ── TCP 8089 (full, then deltas) ──┘

1. What the Bundle Contains

• Bundles typically contain a subset of files (configuration files and assets) from $SPLUNK_HOME/etc/system, $SPLUNK_HOME/etc/apps, and $SPLUNK_HOME/etc/users on the search head.
• By default, peers receive nearly the entire contents of the search head's apps, including lookup files. Large lookups are the usual cause of oversized bundles.
• [replicationAllowlist] and [replicationDenylist] in distsearch.conf control which files go into the bundle. The denylist wins when a file matches both.

2. Replication Cycle

1. The search head builds bundles under $SPLUNK_HOME/var/run.
2. It replicates the bundle periodically in the background or when it starts a search.
3. The first replication to a new peer sends the full bundle. Later replications usually send only the delta since the last one.
4. Each peer stores bundles under $SPLUNK_HOME/var/run/searchpeers, one set per search head. Because the path differs from the search head's, search scripts must not hard-code paths to resources.

3. Replication Policies (replicationPolicy in [replicationSettings])

Keeping the Knowledge Bundle Small

If very large or frequently changing files are part of the bundle, replication slows, peers use more disk under searchpeers, and searches can run against stale knowledge. Once the bundle is larger than maxBundleSize (default 2048 MB), replication stops and Splunk logs an error and shows an admin message.

Excluding Files with [replicationDenylist]

On the search head, add patterns to distsearch.conf. The older stanza name [replicationBlacklist] still works but is deprecated:

[replicationSettings]
maxBundleSize = 2048

[replicationDenylist]
# Exclude one very large lookup that searches do not need on the peers
big_threat_feed = apps/TA-threat-intel/lookups/massive_feed.csv
# Exclude every CSV lookup in one app
noisy_lookups = apps/legacy_app/lookups/*.csv

Pattern Rules

1. Patterns are matched against file paths relative to $SPLUNK_HOME/etc, such as apps/<app>/lookups/<file> or users/<user>/<app>/....
2. The syntax is Splunk's pattern style, which is mostly a regular expression: ... matches anything, * matches anything except a directory separator, and . matches a literal dot.
3. If you exclude a lookup that a search still needs, run that lookup on the search head instead, for example with lookup local=true, because the peers no longer have the file.
4. After changing the lists, watch the next replication in the Monitoring Console or splunkd.log to confirm the bundle shrank below maxBundleSize.