1.1 Splunk Enterprise Architecture & Core Components

Splunk Enterprise Architecture & Core Components

Quick Summary: Splunk Enterprise operates as a modular, distributed platform organized across four distinct functional tiers: Ingestion, Indexing, Search, and Management. Scaling an enterprise deployment requires understanding component responsibilities—such as the discrete roles of Indexers, Search Heads, the Cluster Manager, Deployer, and Deployment Server—as well as the specific TCP ports (8000, 8089, 9997, 9887, 8191) facilitating inter-node orchestration.

The Four Processing Tiers of Splunk Enterprise

At enterprise scale, Splunk decouples its processing stages into dedicated tiers to ensure independent scalability, fault tolerance, and predictable performance.

1. Ingestion Tier (Collection & Egress): Composed of Universal Forwarders (UFs) and Heavy Forwarders (HFs) deployed across endpoints, application servers, network devices, and intermediate perimeter gateways. This tier monitors files, captures network streams (syslog, metrics), runs scripted inputs, and forwards raw or pre-processed events to the indexing layer.

2. Indexing & Storage Tier (Parsing & Persistence): Composed of Indexers (also termed peer nodes in a cluster). This tier receives data streams from forwarders, breaks raw bitstreams into discrete events, extracts timestamps, assigns default metadata (source, sourcetype, host, index), parses data transformations, writes compressed raw data and time-series index files (*.tsidx) into directory buckets, and runs the search requests that Search Heads dispatch to it.

3. Search & Visualization Tier (SPL Compilation & Presentation): Composed of Search Heads (standalone or in a Search Head Cluster). This tier serves the Splunk Web user interface (TCP 8000), authenticates users, parses Search Processing Language (SPL) queries, compiles execution plans, dispatches search requests to indexers over the management port (TCP 8089), aggregates and post-processes returned result sets, and renders dashboards, reports, and alerts. Search heads store knowledge objects (saved searches, lookups, field extractions, data models) but do not store indexed application data.

4. Management & Orchestration Tier (Governance & Topology Control): Composed of specialized management instances that govern configuration distribution, cluster state, and licensing:

   • Deployment Server (DS): Centralized deployment tool distributing configuration packages (deployment apps) to forwarders.
   • License Manager (LM): Meters daily uncompressed data ingestion volume against purchased licensing quotas.
   • Cluster Manager (CM): Orchestrates Indexer Clusters, managing bucket replication factor (RF) and search factor (SF).
   • Deployer: Distributes configuration bundles and apps to Search Head Cluster (SHC) members.

---

Detailed Core Component Responsibilities

Each component in the Splunk Enterprise ecosystem performs specialized tasks with distinct boundary constraints:

1. Indexers (Search Peers)

An Indexer performs two primary workloads: data ingestion and search servicing.

• Ingestion Pipeline: When unparsed data arrives (typically on the TCP 9997 receiving port), the indexer moves it through a chain of queues and pipelines: parsingQueue -> parsing pipeline (UTF-8 conversion, line breaking, header handling) -> aggQueue -> merging pipeline (line merging and timestamp extraction) -> typingQueue -> typing pipeline (regex replacement for TRANSFORMS/SEDCMD, punct annotation) -> indexQueue -> indexing pipeline, which writes buckets.
• Bucket Storage: Events are written into rawdata blocks inside buckets (hot, warm, cold, frozen), accompanied by .tsidx inverted index files that map search keywords to specific event locations.
• Search Execution: When a Search Head dispatches a query, the indexer evaluates its local .tsidx files, retrieves matching raw events from disk, evaluates streaming search commands (such as eval, where, fields), and streams the filtered intermediate results back to the Search Head.

2. Search Heads

Search Heads isolate analytical workloads from storage operations.

• A Search Head translates user SPL into distributed search queries. For example, in index=web status=500 | stats count by clientip, the filtering step (index=web status=500) is distributed to all indexers to run in parallel.
• The reporting command (stats count by clientip) is partially evaluated on indexers and finalized on the Search Head.
• In distributed environments, Search Heads maintain search peer configurations in distsearch.conf and issue distributed requests over the management port (TCP 8089).

3. Forwarders (Universal & Heavy)

• Universal Forwarder (UF): The lightweight agent installed on data-producing endpoints. It collects data and streams unparsed raw chunks over TCP 9997. It lacks Splunk Web and the parsing pipeline.
• Heavy Forwarder (HF): A full Splunk Enterprise instance configured to forward data rather than index it locally. It contains full parsing and typing queues, allowing it to perform regex-based data masking, event routing, filtering to nullQueue, or terminate HTTP Event Collector (HEC) inputs.

4. Deployment Server (DS)

• The Deployment Server provides centralized configuration management for forwarders and non-clustered instances.
• Deployment clients (forwarders) run the deploymentclient.conf engine, polling the Deployment Server over TCP 8089 at a default phoneHomeIntervalInSecs (default: 60 seconds).
• The Deployment Server groups clients into Server Classes based on hostname, IP address, DNS domain, or machine architecture, downloading matching apps from $SPLUNK_HOME/etc/deployment-apps/ into $SPLUNK_HOME/etc/apps/ on the client.

5. License Manager (formerly License Master)

• The License Manager hosts one or more Splunk Enterprise license keys and meters daily indexing consumption.
• Each indexer that reports to a central license manager is a license peer. It sets manager_uri in the [license] stanza of server.conf (the older master_uri name is deprecated) to the manager's management port, usually TCP 8089.
• License usage is the raw, uncompressed volume of data entering the indexing pipeline, not the compressed size on disk. Peers report it to the manager every minute, and the license day runs from midnight to midnight on the license manager's clock.
• Exceeding a pool's or stack's daily quota generates a license warning. If a license type accumulates too many warnings in its rolling window, it goes into violation. For an Enterprise stack under 100 GB/day, that is 45 warnings in 60 days. A violation blocks search, never indexing (see Chapter 3).

6. Cluster Manager (formerly Cluster Master)

• The Cluster Manager coordinates Indexer Clusters (peer nodes).
• It maintains the cluster's Replication Factor (RF) (the number of raw data copies maintained across distinct peers) and Search Factor (SF) (the number of searchable .tsidx index copies maintained).
• It tracks which peers hold each bucket copy, designates the primary (searchable) copies, and orchestrates bucket fix-up when a peer goes down so that the replication factor and search factor are restored.
• Crucial Architectural Rule: The Cluster Manager does NOT sit in the data path. Peer nodes replicate bucket data directly to one another over a replication port that is configured on each peer (Splunk's examples use TCP 9887). The Cluster Manager merely communicates metadata and instructions over TCP 8089.

7. Deployer & Search Head Cluster (SHC) Captain

• Deployer: A dedicated management instance that pushes configuration baseline apps to Search Head Cluster members using the command splunk apply shcluster-bundle. Staging files reside in $SPLUNK_HOME/etc/shcluster/apps/.
• SHC Captain: In a Search Head Cluster (minimum 3 members), one member is elected Captain by a Raft-based election that members run over their management ports (TCP 8089). Port 8191 belongs to the KV store and is a separate replication channel. The Captain assigns scheduled search jobs across cluster members, balances search concurrency, and coordinates knowledge object synchronization. If the Captain fails, the remaining members dynamically elect a new Captain.

---

Network Communication & Port Topology

Configuring firewalls and verifying network reachability between Splunk components is a fundamental administrative responsibility. The table below defines the standard network ports utilized across Splunk Enterprise components:

---

Topology Architectures: Single-Instance vs. Clustered Distributed Tiers

Splunk Enterprise deployments evolve across distinct architectural maturity models based on ingestion throughput, search concurrency, and availability requirements:

Architecture Models Compared

Exam note: Splunk does not publish one fixed daily-volume cut-off for moving from one model to the next. Sizing comes from Splunk Validated Architectures and capacity planning, not from a number you memorize for this exam.

Topology Breakdown

1. Single-Instance Deployment: A single Splunk Enterprise installation executes all roles simultaneously: input ingestion, parsing, indexing, search execution, and web hosting. While simple to deploy, any service restart halts both data collection and analytical searches. There is no separation between user query overhead and data write performance.

2. Distributed (Non-Clustered) Deployment: Splunk decouples Search Heads from Indexers. Forwarders route data to one or more standalone indexers over TCP 9997. A dedicated Search Head connects to indexers via distributed search definitions (distsearch.conf). However, if an indexer fails, the data residing on that specific node becomes inaccessible to search until the host recovers, and data in transit may be lost if indexer acknowledgment is disabled.

3. Clustered Distributed Deployment: Enterprise production environments deploy redundant clusters:

   • Indexer Cluster: Managed by a Cluster Manager. Contains peer nodes holding replicated bucket copies across site boundaries (Single-Site or Multi-Site Indexer Clustering). Provides high availability, disaster recovery, and continuous searchability even during hardware failure or operating system patching.
   • Search Head Cluster (SHC): A pool of at least three Search Heads coordinated by a dynamic Captain and provisioned by a Deployer. Delivers search redundancy, load balances scheduled alerts, and prevents search service interruptions.

---

Administrative Traps & Architecture Anti-Patterns

When designing and maintaining a Splunk environment, administrators must avoid common architectural traps:

Anti-Pattern 1: Managing Clustered Nodes with the Deployment Server

A frequent administrative error is configuring the Deployment Server to push configuration apps directly to Indexer Cluster peer nodes or Search Head Cluster members.

• Why this fails: Indexer peer nodes require coordinated bundle deployment via the Cluster Manager ($SPLUNK_HOME/etc/manager-apps/, formerly master-apps/) to validate the bundle, distribute it consistently to every peer, and perform rolling restarts when needed. Bypassing the Cluster Manager causes configuration drift and cluster instability.
• The Correct Pattern: Use the Deployment Server strictly for Universal Forwarders, Heavy Forwarders, and standalone instances. Use the Cluster Manager for indexer peers, and the Deployer for Search Head Cluster members.

Anti-Pattern 2: Colocating the Deployment Server on High-Load Instances

Placing the Deployment Server role on an active Search Head or primary Indexer in an environment with hundreds of forwarders.

• Why this fails: Forwarders poll the Deployment Server over TCP 8089 every 60 seconds by default. When hundreds of forwarders phone home simultaneously, the splunkd management thread pool becomes saturated, causing search execution timeouts on Search Heads or delayed ingest acknowledgment on Indexers.
• The Correct Pattern: Splunk's planning guidance is to run the deployment server on a dedicated instance once it has more than 50 clients. Only with 50 or fewer clients may it share an instance with another role, and it must never share one with an indexer cluster manager.

Anti-Pattern 3: Misunderstanding Cluster Manager Network Flows

Assuming that all data replicated between indexers passes through the Cluster Manager.

• Why this fails: Administrators frequently misconfigure network access control lists (ACLs) by allowing replication port 9887 only between the Cluster Manager and indexers, rather than between all peer indexers.
• The Correct Pattern: Open TCP 9887 bidirectionally among all indexer peer nodes. The Cluster Manager only requires TCP 8089 access to communicate state instructions.

Anti-Pattern 4: Assuming License Manager Downtime Immediately Halts Ingestion

Believing that if the License Manager goes offline, indexers immediately reject forwarder connections.

• Why this fails: Splunk incorporates a resilient 72-hour grace period. If a license peer loses network connectivity to the License Manager, it continues ingesting, parsing, and indexing data uninterrupted. Only if communication remains severed after 72 continuous hours is search capability restricted; indexing operations never cease.