11.1 Deployment Management & the Deployment Server

Deployment Server Architecture & Deployment Apps

Quick Summary: Centralized configuration management across distributed Splunk environments is anchored by the Deployment Server (DS) and Deployment Client (DC) framework. Rather than manually configuring hundreds or thousands of forwarders, administrators maintain modular configuration packages called deployment apps in $SPLUNK_HOME/etc/deployment-apps/ on a centralized server. Through client-driven polling over HTTPS management port 8089, deployment clients automatically discover, download, and stage configuration updates based on assigned server classes.

Centralized Management: The Poll-and-Download Paradigm

A critical architectural distinction in Splunk Enterprise is that the Deployment Server does not push configurations to clients. Instead, the architecture functions on a client-initiated poll-and-download model (often termed "phone-home").

+-----------------------+              TCP 8089 (HTTPS)             +-----------------------+
|   Deployment Client   | ----------------------------------------> |   Deployment Server   |
| (Universal Forwarder) | <---------------------------------------- |  (Splunk Enterprise)  |
+-----------------------+   1. Handshake & Checksum Manifest        +-----------------------+
                            2. Compute Delta & Download Apps

Why Pull Outperforms Push in Enterprise Networks

1. Perimeter Firewall & NAT Traversal: Forwarders are frequently deployed across strict network segments, such as DMZs, cloud VPCs, remote branch offices, and isolated internal subnets. Firewalls typically permit outbound connections from endpoints to central management networks but block inbound connections initiated by central servers. Because the forwarder initiates the outbound connection to the Deployment Server over TCP 8089, no inbound firewall pinholes into secured host environments are required.
2. Ephemeral & Dynamic Host Support: In modern containerized and cloud auto-scaling environments, forwarders spin up and tear down dynamically with ephemeral IP addresses. A push server would need an exhaustive, real-time registry of all active forwarder IPs. With the polling model, newly provisioned forwarders automatically register themselves upon initial boot.
3. Resilience to Offline Endpoints: When laptops, branch offices, or development VMs go offline, a push server wastes CPU cycles retrying connections and generating timeout faults. Under the poll-and-download model, offline forwarders simply check in and download required configuration updates whenever network connectivity is restored.

The Phone-Home Handshake Sequence

Every polling cycle follows a deterministic four-phase exchange over HTTPS (TCP 8089):

1. Connection: The client connects over HTTPS to the deployment server's management port at targetUri. If the deployment server requires client certificates (requireClientCert), the client must present one.
2. Client Identity: The client identifies itself (client name, IP address, DNS name, host name, instance ID, and machine type) and reports the deployment apps it holds, with their checksums.
3. Server Class Evaluation & Delta Calculation: The Deployment Server evaluates the client's metadata against all stanzas in serverclass.conf. It compiles the set of apps the client should possess and compares the server's current app checksums against the checksums reported by the client.
4. Download & Execution: If the Deployment Server detects missing apps or checksum mismatches, it responds with an instruction set directing the client to download the updated archive bundles. The client streams the compressed app packages from the Deployment Server, unpacks them into $SPLUNK_HOME/etc/apps/, and initiates any required post-download actions (such as restarting splunkd or reloading specific configuration subsystems).

Planning and Placing the Deployment Server

The deployment server does no indexing, but it works hard when many clients phone home and download apps at the same time. Splunk's planning guidance:

• Dedicated instance: Because app downloads use a lot of CPU and memory, run the deployment server on a dedicated Splunk Enterprise instance that is not an indexer or search head.
• The 50-client exception: Only when the deployment server has 50 or fewer clients may an indexer or search head double as the deployment server. In that small case the deployment server may also host some management components, such as a search head cluster deployer.
• Never with the cluster manager: Do not co-locate a deployment server and an indexer cluster manager under any circumstances. Both need reliable resources, and the deployment server can starve the manager.
• Not its own client: A deployment server cannot be a deployment client of itself. Splunk logs "This DC shares a Splunk instance with its DS: unsupported configuration".
• Pace large fleets: Clients phone home every phoneHomeIntervalInSecs (default 60 seconds). With thousands of clients, a longer interval spreads out handshakes and downloads.

Illustrative Bandwidth Estimate

Pushing a 15 MB add-on to 4,000 forwarders sends about 60 GB in total. If every client downloaded it within the same minute, that would take about 8 Gbps:

$\text{Bandwidth} = \frac{4{,}000 \times 15\text{ MB} \times 8}{60\text{ s}} = 8{,}000\text{ Mbps}$

In practice clients phone home at different moments, but the math shows why large app updates and short phone-home intervals on big fleets need planning.

Architectural Boundaries: What the Deployment Server Must Not Manage

1. Never Use the Deployment Server for Indexer Cluster Peers

Do not use the deployment server or forwarder management to distribute configuration to indexer cluster peer nodes. Peers receive common configuration only through the cluster manager's configuration bundle: files placed in $SPLUNK_HOME/etc/manager-apps/ (formerly master-apps/) and pushed with splunk apply cluster-bundle. You may use the deployment server to update the cluster manager itself, which then pushes the bundle to the peers.

2. Never Use the Deployment Server for Search Head Cluster Members

The deployment server is not supported for distributing apps or configuration to search head cluster members. Use the deployer ($SPLUNK_HOME/etc/shcluster/apps/ and splunk apply shcluster-bundle).

3. Keep the Deployment Server Off Busy Instances

Once there are more than 50 clients, keep the deployment server off indexers and search heads, and never place it on the indexer cluster manager. Heavy phone-home and download traffic on those instances competes with indexing, searching, and cluster management.