8.4 Search Head Scaling & Search Head Clustering (SHC)

Search Head Scaling & Search Head Clustering (SHC)

As user adoption and analytical workloads expand across an enterprise, a single Search Head inevitably encounters resource constraints. Maximum concurrent search capacity is bound to physical CPU cores, and an unexpected hardware outage or operating system restart halts dashboard access and scheduled alerting. To eliminate single points of failure and scale search throughput horizontally, Splunk provides Search Head Clustering (SHC).

An SHC coordinates a pool of search heads that share search workloads, synchronize configurations, replicate search artifacts, and orchestrate scheduled reports without human intervention. Mastering the underlying architecture, consensus mechanisms, and administrative workflows of Search Head Clusters is an advanced competency for Splunk administrators.

---

Search Tier Scaling Models: Standalone vs. Clustered

Enterprise architectures evaluate three distinct operational models for the search tier:

1. SINGLE SEARCH HEAD (Lab / Dev)
   [Users] --> [Single Search Head] --> [Indexers]
   * Single Point of Failure (SPOF)
   * Strict Concurrency Ceiling

2. MULTIPLE INDEPENDENT SEARCH HEADS (Departmental)
   [SecOps Users] --> [Search Head A] --\
                                         --> [Shared Indexers]
   [NetOps Users] --> [Search Head B] --/
   * Siloed Knowledge Objects (Saved searches on A do not exist on B)
   * Duplicate Alert Executions
   * Operational Divergence

3. SEARCH HEAD CLUSTER (Enterprise Production)
                 +----------------------+
                 | Load Balancer (ADC)  |
                 +----------------------+
                     /      |       \
                    v       v        v
                 +-----+ +-----+ +-----+
                 | SH1 | | SH2 | | SH3 | (Min 3 Members)
                 +-----+ +-----+ +-----+
                    \       |       /
               Raft Consensus & Replicated Artifacts
                     \      |      /
                      v     v     v
                 +----------------------+
                 |   Shared Indexers    |
                 +----------------------+

Model Comparison

---

Architecture of a Search Head Cluster

A Search Head Cluster consists of multiple interconnected components operating under explicit architectural rules:

1. The 3-Member Minimum Requirement

An SHC requires a minimum of three Search Head members. A two-member cluster is strictly unsupported in production because two nodes cannot achieve a majority quorum in the event of a single node failure or network partition.

2. The Dynamic Captain

At any given time, exactly one member in the cluster functions as the Search Head Captain. The captain is elected dynamically by the members through a Raft-based election. The Captain's responsibilities include:

• Scheduling Coordination: Managing the cluster-wide scheduler and assigning scheduled searches and alerts across members based on current node resource utilization.
• Search Artifact Synchronization: Orchestrating the replication of search artifacts across members to meet the cluster's configured replication_factor.
• Knowledge Object Replication: Coordinating changes to runtime knowledge objects (dashboards, alerts, lookups, field extractions) across all cluster members.
• Cluster State Tracking: Monitoring member heartbeats and health metrics.

[!IMPORTANT] The Captain does not sit idle waiting to schedule jobs; it functions as a fully capable search head that also serves user queries and executes searches.

3. Cluster Members

All other search heads in the cluster function as Members. Members:

• Authenticate users and serve Splunk Web sessions distributed by an upstream load balancer.
• Execute ad-hoc searches initiated by users.
• Execute scheduled searches assigned to them by the Captain.
• Replicate search artifacts and knowledge object modifications to peer members.
• Participate in Raft elections whenever the Captain becomes unreachable.

4. The Deployer Role

The Deployer is a specialized, non-clustered Splunk Enterprise instance dedicated to distributing apps, configuration baselines, and Technology Add-ons to SHC members.

• Staging Directory: Staged inside $SPLUNK_HOME/etc/shcluster/apps/.
• Crucial Boundary Rule: The deployer must never be a member of the search head cluster it manages. It is a separate Splunk Enterprise instance. Members find it through conf_deploy_fetch_url in their [shclustering] stanza. It can share an instance with some other management components, and a deployment server with 50 or fewer clients can host it.
• Anti-Pattern Warning: The Deployment Server must never be used to manage Search Head Cluster members. Baseline apps must be deployed exclusively via the Deployer.

Inter-Node Network Ports

---

The Raft Consensus Algorithm & Quorum Dynamics

Search Head Clusters rely on the Raft consensus algorithm to elect the Captain and maintain state consistency across members without risk of split-brain corruption.

Calculating Quorum: The Majority Formula

To elect a Captain and perform write operations (such as scheduling searches or modifying knowledge objects), an SHC must achieve Quorum. Quorum represents a strict majority of the total configured cluster members:

$\text{Quorum} = \left\lfloor \frac{N}{2} \right\rfloor + 1$

Where $N$ is the total number of registered cluster members.

Fault Tolerance by Member Count

Why Odd Numbers of Members Are Preferred

The quorum table shows why architects prefer odd numbers of cluster members (3, 5, or 7):

• Adding a 4th member to a 3-member cluster increases hardware, licensing, and network costs, but provides zero additional fault tolerance. Both a 3-member and a 4-member cluster can only survive a single node failure.
• If a 4-member cluster loses 2 nodes, remaining members (2) cannot reach the required quorum of 3, causing the cluster to freeze.
• In contrast, a 5-member cluster withstands 2 simultaneous node failures while maintaining the quorum of 3.

Handling Network Partitions (Split-Brain Prevention)

Consider a 5-member SHC deployed across two physical data centers connected by an inter-site link:

• Data Center East: Contains Member 1, Member 2, Member 3.
• Data Center West: Contains Member 4, Member 5.

If the inter-site fiber link is severed:

1. Data Center East (3 nodes): Calculates its active count against $N=5$. It has 3 nodes available, satisfying Quorum ($3 \ge 3$). It retains or elects an active Captain and continues full operations: scheduling searches, modifying dashboards, and servicing users.
2. Data Center West (2 nodes): Evaluates its active count (2 nodes). Because $2 < 3$, it fails quorum. The nodes immediately step down to a restricted operational state: they refuse to elect a Captain, halt all scheduled search executions, and reject knowledge object writes.
3. Outcome: The cluster completely prevents a "split-brain" catastrophe where both data centers independently schedule duplicate alerts and diverge in state.

Dynamic Captaincy vs. Static Captaincy

• Dynamic Captaincy (Default): Members continuously elect and re-elect the Captain dynamically using Raft. If the active Captain crashes, the remaining quorum members automatically elect a new Captain within seconds.
• Static Captaincy: A disaster-recovery override. When too many members are lost for a majority to exist, an administrator reconfigures the survivors with election = false (in [shclustering]), making one member a static captain (mode = captain) and pointing the others at it (captain_uri). Searching and scheduling can then resume. Return to dynamic election once the cluster is healthy again.
• Captain tuning: preferred_captain = true marks the members that should hold captaincy. captain_is_adhoc_searchhead = true stops the captain from running scheduled jobs, and adhoc_searchhead = true does the same for a regular member.

Scheduled Search Execution & Search Artifact Replication

One of the greatest operational advantages of an SHC is its ability to scale scheduled reporting and alerting without duplication.

                           +-------------------------+
                           |  SHC CAPTAIN (MEMBER 1) |
                           |  - Central Scheduler    |
                           |  - Tracks Node Load     |
                           +-------------------------+
                                 /             \
      Assigns Job 101           /               \   Assigns Job 102
      (Low CPU Load)           /                 \  (Low CPU Load)
                              v                   v
                   +-------------------+ +-------------------+
                   |     MEMBER 2      | |     MEMBER 3      |
                   |  Executes Job 101 | |  Executes Job 102 |
                   +-------------------+ +-------------------+
                              \                   /
                               \                 /
               Replicates Search Artifacts (replication_factor = 3)
                                 \             /
                                  v           v
                   +-----------------------------------------+
                   | DISPATCH DIRECTORY REPLICATION:         |
                   | $SPLUNK_HOME/var/run/splunk/dispatch/   |
                   +-----------------------------------------+

1. Centralized Scheduled Search Orchestration

In a standalone search head, the local savedsearches.conf scheduler dispatches searches directly. In an SHC, scheduling is centralized exclusively on the Captain:

• The Captain reads the consolidated schedule for all enabled alerts and reports across all apps.
• The Captain monitors the current search concurrency, active search jobs, and CPU load of all members.
• The Captain delegates execution to the least loaded member in the cluster.
• Failover Protection: If a member executing a scheduled alert terminates unexpectedly before finishing, the Captain detects the broken heartbeat, re-claims the job, and re-dispatches it to a healthy member. Alerts are never dropped or silently abandoned.

2. Search Artifact Replication Mechanics

When a member completes a search, the results are written to disk as a search artifact inside $SPLUNK_HOME/var/run/splunk/dispatch/<search_id>/.

• In an SHC, artifacts are replicated between members over the cluster's configured replication port.
• The number of copies maintained is determined by the replication_factor attribute in server.conf ([shclustering] stanza; default value: 3).
• Why this matters for users: When users access Splunk Web through a load balancer, subsequent page refreshes or dashboard views may land on different cluster members. Because artifacts are replicated, Member 2 can instantly render the cached dashboard panels generated moments earlier by Member 1 without re-running expensive queries against the indexers.

---

Deployer Administration & App Lifecycle Management

Because SHC members are ephemeral compute nodes that replicate dynamic runtime configurations, administrative configurations—such as installing Technology Add-ons, creating base apps, or deploying pre-built dashboards—must follow a controlled deployment pipeline via the Deployer.

1. The Staging Repository on the Deployer

On the Deployer host, application packages and configurations destined for cluster members are placed inside:

$SPLUNK_HOME/etc/shcluster/apps/
├── Splunk_TA_windows/
├── Splunk_TA_nix/
├── enterprise_security_app/
└── custom_corporate_dashboards/

2. Pushing App Bundles via the CLI

To deploy the staged applications to the cluster, the administrator executes the apply shcluster-bundle command directly from the Deployer:

splunk apply shcluster-bundle \
    -target https://sh01.corp.internal:8089 \
    -auth admin:DeployerPassword \
    -preserve-lookups true

• -target: The URI of any active member of the cluster (standard practice targets the current Captain, though pointing to any member redirects appropriately).
• -auth: Local administrative credentials on the Deployer authorizing the push.
• -preserve-lookups true|false: A critical administrative flag. If set to true, the deployment process ensures that lookup table CSV files modified by users or lookups populated by searches on the cluster members are not overwritten by the static files residing on the Deployer.

3. Deployment Flow and Rolling Restarts

When splunk apply shcluster-bundle executes:

1. The Deployer packages $SPLUNK_HOME/etc/shcluster/apps/ into a compressed bundle and transmits it to the target member over TCP 8089.
2. The Captain coordinates the distribution of the bundle to all active members.
3. The Captain inspects the updated configurations. If the changes require a restart (e.g., updates to server.conf, web.conf, or specific modular inputs), the Captain orchestrates a rolling restart.
4. During a rolling restart, the Captain instructs members to restart one at a time. The remaining members absorb incoming user traffic from the load balancer, ensuring uninterrupted availability throughout the application upgrade.