11.2 Deployment Apps & Deployment Clients

Deployment Apps and Deployment Clients

Blueprint objectives 11.3 and 11.4 cover managing forwarders with deployment apps and configuring deployment clients. The deployment server stores apps in $SPLUNK_HOME/etc/deployment-apps/, and each client pulls the apps its server classes assign to it into $SPLUNK_HOME/etc/apps/.

Deployment Apps Anatomy & Packaging Best Practices

On the Deployment Server, configuration packages staged for distribution are known as deployment apps. They reside in a specialized repository directory:

$SPLUNK_HOME/etc/deployment-apps/<app_name>/

When a client downloads a deployment app, the Deployment Server strips the deployment-apps namespace and writes the directory directly into the client's production runtime directory:

$SPLUNK_HOME/etc/apps/<app_name>/

Standard Directory Structure of a Deployment App

A deployment app conforms to the standard Splunk application hierarchy:

$SPLUNK_HOME/etc/deployment-apps/org_all_forwarder_outputs/
├── default/
│   ├── app.conf
│   └── outputs.conf
├── local/            (optional; replaced on clients at each update)
├── bin/
│   └── (Optional: custom collection scripts, modular inputs, binaries)
└── metadata/
    └── default.meta

What Happens to an App on the Client During an Update

When a deployment app changes, the client replaces its copy of the app with the downloaded version. Any content created locally inside that app on the client, including its local/ directory and lookup files, is deleted or overwritten, unless you protect it with excludeFromUpdate in serverclass.conf on the deployment server:

[serverClass:linux_web:app:org_app_apache_inputs]
excludeFromUpdate = $app_root$/local,$app_root$/lookups/cidr.csv

excludeFromUpdate can be set at the global, server class, or app level. Each entry must start with $app_root$/. Also note: if you remove an app from the deployment server's repository, every client that received it deletes its copy. Once the deployment server manages an app, it always manages it.

Creating Modular "Micro-Apps"

A severe anti-pattern in Splunk forwarder management is deploying a single monolithic app containing outputs, multiple input types, and parsing properties (e.g., all_forwarder_config). This approach tightly couples unrelated configurations and forces widespread forwarder restarts whenever a single input changes.

Instead, enterprise administrators construct atomic, modular micro-apps dedicated to single operational domains:

$SPLUNK_HOME/etc/deployment-apps/
├── org_all_forwarder_outputs/      # outputs.conf (indexer cluster target, autoLB, SSL)
├── org_all_license_client/        # server.conf ([license] points to License Manager)
├── org_nix_base_inputs/           # inputs.conf (Linux /var/log/messages, secure, audit)
├── org_win_base_inputs/           # inputs.conf (Windows Security, System, Application event logs)
├── org_app_apache_inputs/         # inputs.conf (Apache access_log, error_log)
└── org_app_oracle_inputs/         # inputs.conf (Oracle alert logs and listener telemetry)

By composing server classes from these modular units, a Linux web server can receive org_all_forwarder_outputs, org_nix_base_inputs, and org_app_apache_inputs. If the Apache log path changes, only org_app_apache_inputs is updated and distributed, eliminating unnecessary changes to baseline system or output configurations.

Deployment Client Setup: deploymentclient.conf

To transform any standard Splunk instance (Universal Forwarder, Heavy Forwarder, or standalone server) into a Deployment Client, an administrator defines deploymentclient.conf in $SPLUNK_HOME/etc/system/local/ or within a bootstrap deployment client application ($SPLUNK_HOME/etc/apps/org_base_deploymentclient/local/deploymentclient.conf).

Detailed Configuration Syntax

# $SPLUNK_HOME/etc/system/local/deploymentclient.conf

[deployment-client]
# Optional logical name the deployment server can filter on
clientName = corp-web-prod-01

# How often (seconds) to check for new content (default 60)
phoneHomeIntervalInSecs = 60

[target-broker:deploymentServer]
# Fully qualified domain name or IP and management port (TCP 8089) of the Deployment Server
targetUri = splunk-ds.corp.internal:8089

Key Attributes Breakdown

• [deployment-client]: The global client stanza.
  • clientName: An optional logical name, such as pci-cardholder-web, that server class filters can match. When filters are evaluated, the client name is tried first, then IP address, DNS name, host name, and instance ID.
  • phoneHomeIntervalInSecs: The time between phone-home checks. Defaults to 60 seconds.
• [target-broker:deploymentServer]: Defines the connection target broker.
  • targetUri: The deployment server's host name or IP address and management port, in host:port form (for example splunk-ds.corp.internal:8089).
  • handshakeRetryIntervalInSecs: How often to retry a failed handshake (default: one fifth of phoneHomeIntervalInSecs).
  • After changing deploymentclient.conf, restart the client. splunk set deploy-poll also requires a restart.

CLI Configuration Workflow

Administrators can configure the deployment client via the command-line interface without manually creating files:

# Point forwarder to the Deployment Server and configure deploymentclient.conf
splunk set deploy-poll splunk-ds.corp.internal:8089

# Restart splunkd to initiate immediate handshake
splunk restart

# Display current deployment client status and target broker configuration
splunk display deploy-client

# Check the configured phone-home polling target URI
splunk show deploy-poll