8.3 Configuring Search Peers & Distributed Search Groups

Configuring Search Peers & Distributed Search Groups

In a distributed Splunk deployment, establishing secure, resilient connections between Search Heads and Indexers is essential for search execution. Administrators must actively register search peers, configure communication timeouts, monitor peer availability, and logically partition indexer fleets using Distributed Search Groups.

All distributed search connections are governed by configuration stanzas within distsearch.conf. Understanding how to manipulate these settings through Splunk Web, the Command Line Interface (CLI), and direct configuration edits ensures operational stability across complex network topologies.

---

Establishing Search Peer Connections

Before a Search Head can dispatch queries to an indexer, an administrator must register the indexer as a Search Peer. When the relationship is established, the search head's public key is placed on the peer over the management port (TCP 8089), and the peer uses it to authenticate the search head's later requests.

+-----------------------+                         +-----------------------+
|      SEARCH HEAD      |                         |      SEARCH PEER      |
|                       |                         |       (INDEXER)       |
| 1. splunk add         |  HTTPS REST (TCP 8089)  |                       |
|    search-server with | ----------------------> | 2. Authenticates the  |
|    -remoteUsername /  |                         |    remote admin creds |
|    -remotePassword    |                         |                       |
|                       |  SH public key (trusted.pem)                    |
| 3. Sends its public   | ----------------------> | 4. Stores it under    |
|    key                |                         |    etc/auth/distServerKeys/<SH serverName>/ |
| 5. Adds peer to       |                         |                       |
|    distsearch.conf    |                         |                       |
+-----------------------+                         +-----------------------+

Method 1: Configuring Search Peers via Splunk Web

1. Log in to the Search Head's Splunk Web interface with administrative privileges.
2. Navigate to Settings > Distributed Search > Search Peers.
3. Click New Search Peer.
4. Enter the complete URI of the target indexer's management interface, including protocol and port (e.g., https://idx01.corp.internal:8089).
5. Provide credentials for an admin user on the remote indexer (Remote username and Remote password). The search head uses them once to set up trust by installing its public key on the peer.
6. Click Save, and repeat for each peer. The peer appears in the Search peers table.

Method 2: Configuring Search Peers via the CLI

Run this on the search head, once per peer:

splunk add search-server https://idx01.corp.internal:8089 \
    -auth admin:SearchHeadPassword \
    -remoteUsername admin -remotePassword IndexerPassword

splunk list search-server -auth admin:SearchHeadPassword
splunk remove search-server https://idx01.corp.internal:8089 -auth admin:SearchHeadPassword

• -auth: credentials for the search head (where the command runs).
• -remoteUsername / -remotePassword: credentials for an admin-level user on the search peer. The peer's admin password must have been changed from the default, or the search head cannot authenticate to it.
• The URI must include the scheme (https:// or http://) and the peer's management port.

How the Trust Is Established (Public Key Authentication)

Access to peers is controlled through public key authentication, not stored passwords:

• The search head has a key pair under $SPLUNK_HOME/etc/auth/distServerKeys/, and its public key is trusted.pem.
• When you add a peer with Splunk Web or the CLI, Splunk automatically copies the search head's trusted.pem to the peer at $SPLUNK_HOME/etc/auth/distServerKeys/<searchhead_serverName>/trusted.pem.
• If you add peers by editing distsearch.conf instead, you must copy that key file to each peer yourself and restart the peers.
• A peer that serves several search heads keeps one directory per search head under distServerKeys/.

Important: A search head cannot also act as a search peer. The only exception is the Monitoring Console, which works as a "search head of search heads". Indexer clusters connect their search heads to the peers automatically, and search head clusters have their own rules for connecting to peers.

---

Core distsearch.conf Settings

$SPLUNK_HOME/etc/system/local/distsearch.conf on the search head holds the peer list and the distributed search options:

[distributedSearch]
servers = https://idx01.corp.internal:8089,https://idx02.corp.internal:8089,https://idx03.corp.internal:8089
statusTimeout = 10
connectionTimeout = 10
sendTimeout = 30
receiveTimeout = 600
disabled_servers = https://idx04.corp.internal:8089

Exam trap: Older material lists checkTimedOutServersFrequency, removedTimedOutServers, and autoAddServers. The current spec says the first two are no longer supported and are ignored, and autoAddServers is deprecated. Heartbeat multicast settings are deprecated as well.

---

Monitoring Search Peer Status

Splunk Web shows each peer's state under Settings > Distributed search > Search peers. You can also query it from the search head:

| rest splunk_server=local /services/search/distributed/peers
| table title status replicationStatus version

What Happens When a Peer Is Unreachable

• The search head marks the peer as Down in the peer list and keeps searching the peers that are Up.
• The search still completes, but with incomplete results, and Splunk Web warns the user that some peers could not be searched.
• Peers listed in disabled_servers, or disabled in Splunk Web, are neither monitored nor searched.
• If a peer lacks the current knowledge bundle, it is skipped by default. bestEffortSearch = true changes that behavior.

Administrative Monitoring Tools

1. Monitoring Console: its distributed search dashboards show peer health and knowledge bundle replication.
2. _internal: search splunkd.log on the search head for distributed search and bundle replication warnings.

---

Distributed Search Groups: Logical Fleet Partitioning

In large enterprise deployments with tens or hundreds of indexers, treating all search peers as a single homogenous pool is often inefficient or non-compliant with data governance requirements. Splunk allows administrators to organize search peers into Distributed Search Groups.

Architectural Rationale for Search Groups

• Data Sovereignty & Geographic Boundaries: Isolating indexers by region (e.g., EU_Indexers vs. US_Indexers) to ensure European queries do not dispatch across transatlantic links.
• Regulatory Compliance: Restricting access to high-security environments (such as PCI-DSS cardholder environments or HIPAA repositories) so that only compliance auditors query those dedicated indexers.
• Workload Tiering: Separating high-performance indexers hosting real-time security data from dense storage indexers hosting archived operational metrics.

Defining Search Groups in distsearch.conf

Groups are defined in distsearch.conf on the search head with [distributedSearch:<group_name>] stanzas. Each group's servers list must be a subset of the peers in the main [distributedSearch] stanza, and it uses host:port peer identifiers:

[distributedSearch]
servers = https://10.1.1.11:8089,https://10.1.1.12:8089,https://10.2.1.11:8089,https://10.2.1.12:8089

[distributedSearch:PCI_Indexers]
default = false
servers = 10.1.1.11:8089,10.1.1.12:8089

[distributedSearch:EU_Region]
default = false
servers = 10.2.1.11:8089,10.2.1.12:8089

The default Attribute

• Only one group may have default = true. A search that names no group then runs against that group's peers.
• If no group is the default (or the default group has no servers list), a search that names no group runs against the full [distributedSearch] peer list.
• Groups can overlap. A peer can belong to several groups.
• Indexer clusters: Splunk says search groups are generally not valid for indexer clustering. The cluster places primary bucket copies arbitrarily, so a group-limited search can return incomplete results. Groups are useful with clusters only in special cases, such as a search head that searches several clusters, or a cluster plus standalone indexers.

Restricting Search Execution with splunk_server_group and splunk_server

Users and scheduled searches target specific search groups or individual indexers using internal search fields:

1. Filtering by Search Group (splunk_server_group)

index=card_payments splunk_server_group=PCI_Indexers
| stats count by payment_status

The search head sends this search only to the peers in [distributedSearch:PCI_Indexers]. The other peers receive nothing, which saves their CPU, disk reads, and network bandwidth.

Multiple groups can be combined using boolean logic:

index=network (splunk_server_group=EU_Region OR splunk_server_group=PCI_Indexers)
| stats sum(bytes) by src_ip

2. Filtering by Specific Indexer (splunk_server)

If an administrator needs to investigate or troubleshoot a specific indexer's local storage:

index=_internal splunk_server=idx-us01.corp.internal
| stats count by sourcetype

This directs the query exclusively to the named indexer host.