14.2 HTTP Event Collector (HEC) Architecture & Administration

HTTP Event Collector (HEC)

Quick Summary: The HTTP Event Collector (HEC) lets applications send events to Splunk over HTTP or HTTPS with a token instead of a user name and password, so no forwarder is needed on the client. HEC listens on port 8088 by default, is disabled globally until you enable it, and is configured with a global [http] stanza plus one [http://<token_name>] stanza per token. Know the JSON event endpoint versus the raw endpoint, how tokens restrict indexes, the status codes clients receive, and how HEC indexer acknowledgment uses channels.

What HEC Is and Where It Runs

HEC is an input built into splunkd. Any client that can make an HTTP POST can send data, for example applications, container logging drivers, serverless functions, and automation scripts.

• Token authentication: each token is a GUID. Clients present it in the Authorization header, and tokens can be enabled, disabled, or restricted to specific indexes without sharing user credentials.
• Where to run it: HEC can run on a standalone instance, on indexers, or on heavy forwarders. At scale it is common to place several HEC instances behind a load balancer.
• Where tokens are stored: tokens created in Splunk Web are written to $SPLUNK_HOME/etc/apps/splunk_httpinput/local/inputs.conf. To manage tokens centrally, useDeploymentServer = 1 writes the HEC configuration to the deployment server repository instead.
• Enabling it: in Splunk Web, open Settings > Data inputs > HTTP Event Collector > Global Settings and enable all tokens. This sets disabled = 0 in the [http] stanza.

Global Settings: the [http] Stanza

A separate limit, max_content_length in the [http_input] stanza of limits.conf, caps the size of a request body (default 838,860,800 bytes, about 800 MB).

[http]
disabled = 0
port = 8088
enableSSL = 1
dedicatedIoThreads = 2

Token Stanzas: [http://<token_name>]

Restricting Indexes with indexes

An event can name its own index in the JSON metadata. That index must be in the token's indexes list if the list is set; otherwise the request fails with status code 7, Incorrect index. The key trap: if indexes is not set, the list is empty and any index can be used. Set indexes on every token that untrusted or shared clients use.

[http://k8s_production_ingress]
disabled = 0
token = 7a8b9c0d-1e2f-3a4b-5c6d-7e8f9a0b1c2d
index = prod_k8s
indexes = prod_k8s, prod_k8s_security
sourcetype = kube:ingress
useACK = 1

[http://cicd_build_runner]
disabled = 0
token = 99887766-5544-3322-1100-aabbccddeeff
index = cicd_events
indexes = cicd_events
sourcetype = gitlab:pipeline

Authentication Options

If a token is sent in both the query string and the header, the query-string token takes precedence.

Event Endpoint vs. Raw Endpoint

/services/collector (also /services/collector/event): JSON Events

{
  "time": 1727082400.451,
  "host": "k8s-worker-node-04",
  "source": "auth-svc",
  "sourcetype": "auth:json",
  "index": "prod_k8s",
  "event": {"action": "login_attempt", "user": "jsmith", "status": "success"},
  "fields": {"environment": "production", "region": "us-east-1"}
}

• event holds the data: a string, a number, or a JSON object. The metadata keys time, host, source, sourcetype, index, and fields are optional; anything left out takes the token's values.
• time is epoch time in <sec>.<ms> format.
• fields defines a flat set of indexed fields. It is accepted only by the event endpoint and does not apply to raw data.
• Batching: put several event objects one after another, or in a JSON array, in one request to improve throughput.

/services/collector/raw: Raw Text

curl -H "Authorization: Splunk 7a8b9c0d-1e2f-3a4b-5c6d-7e8f9a0b1c2d" \
  -H "X-Splunk-Request-Channel: 18654C68-B28B-4450-9CF0-6E7645CA60CA" \
  "https://hec.corp.example:8088/services/collector/raw?index=net_fw&sourcetype=cisco:asa&host=edge-fw01" \
  -d "Mar 23 03:00:00 edge-fw01 %ASA-6-302013: Built inbound TCP connection 10423"

• The body is plain text. HEC breaks it into events and sends them through the pipeline, which extracts timestamps and applies the source type's props.conf rules.
• Metadata comes from the query string, the token, or the global settings, and applies to all events in the request.
• Splunk's examples for the raw endpoint include a channel identifier (X-Splunk-Request-Channel header or ?channel=), and a channel is mandatory whenever the token uses indexer acknowledgment.

Status Codes Returned to Clients

HEC replies with an HTTP status and a JSON body containing a Splunk status code. The codes most worth knowing:

Newer codes cover the health endpoint (17–20, for example 18 "HEC is unhealthy, queues are full") and capacity warnings (24–27, including HTTP 429 when the queue or ACK channel is at capacity). Splunk cautions that clients must act on these codes, for example by retrying after 503 or 429, or data can be lost.

HEC Indexer Acknowledgment and Channels

By default, HEC returns HTTP 200 as soon as the request appears valid, before the data enters the processing pipeline. An outage after that point can lose the data. HEC indexer acknowledgment lets the client check what happened. It is a different feature from forwarder indexer acknowledgment (useACK in outputs.conf), even though the names are the same.

1. Enable it per token with useACK = true (or the Enable indexer acknowledgment checkbox when creating the token).
2. Send a channel with every request: a GUID in X-Splunk-Request-Channel or ?channel=. Without it, HEC returns code 10, Data channel is missing. One channel per client stops a fast client from slowing a slow one down.
3. Receive an ack ID: each request returns an identifier such as {"ackId": 42}.
4. Query the status by posting {"acks": [40, 41, 42]} to /services/collector/ack on the same channel.
5. Interpret the reply: true means the events for that ID were replicated at the desired replication factor. It does not guarantee indexing, because the parsing pipeline can still drop events it cannot parse. false means there is no status yet or no information for that ID.
6. Stop querying after true: HEC deletes the status once it has returned true, so asking again returns false. Resend data whose ID never becomes true within a reasonable time.

Load Balancers

Acknowledgment status is kept by the HEC instance that received the request, so ACK queries must reach the same instance. Configure the load balancer for sticky sessions on the channel. For load balancers that can only stick on cookies (Splunk's example is AWS ELB), set channel_cookie so that HEC returns the channel in a cookie.