13.1 Creating Network Inputs: TCP & UDP

Creating Network Inputs: TCP and UDP

Quick Summary: Network inputs let a Splunk instance listen on a TCP or UDP port for data from network devices, syslog servers, and applications. Know the difference between raw network stanzas ([tcp://<port>], [udp://<port>]) and the Splunk-to-Splunk receiving port ([splunktcp://<port>]), the reliability limits of UDP, and why syslog is usually collected by a syslog server that writes to disk, with a universal forwarder sending the files on.

Network Input Stanzas in inputs.conf

Splunk defines network listeners in inputs.conf using dedicated protocol scheme stanzas. Understanding what each stanza expects is crucial for proper data onboarding:

1. Raw TCP Inputs: [tcp://<port>] and [tcp://<remote_server>:<port>]

• A raw TCP stanza configures Splunk to open a listening TCP socket on the specified port.
• It expects an unformatted, uncompressed raw byte stream (such as raw syslog or application logs streamed over TCP).
• If no source type is set, events get sourcetype = tcp-raw and source = tcp:<port>.
• The stanza can name a remote server ([tcp://10.10.20.5:1514]) so that it applies to connections from that host. For access control, the spec recommends the acceptFrom setting instead.
• For TLS-encrypted raw TCP inputs, configure [tcp-ssl:<port>] (note: no //) together with the [SSL] stanza in inputs.conf.
• Stream termination: A raw TCP connection carries many events, and line-breaking rules (LINE_BREAKER in props.conf) separate them. If the connection stays idle for rawTcpDoneTimeout seconds (default 10) after receiving data, Splunk adds a Done key, declaring that the last event has been completely received.

2. Raw UDP Inputs: [udp://<port>] and [udp://<remote_server>:<port>]

• A raw UDP stanza opens a listening UDP socket on the specified port. Only one stanza per UDP port is supported.
• UDP is packet/datagram-based. Each arriving UDP packet is treated as a discrete datagram, though multiple log events may be packed into a single datagram or fragmented across packets depending on the sender MTU.
• Common use cases include legacy syslog (port 514), SNMP traps (port 162), and NetFlow/sFlow collectors.
• There is no TLS option for UDP inputs.
• If you do not set a source type on a UDP input, events get the source type udp:<port>.

3. Cooked Splunk-to-Splunk Inputs: [splunktcp://<port>]

• Configures the Splunk-to-Splunk (S2S) receiving port, conventionally TCP 9997. The spec describes it as the same as [tcp://], except the sender is assumed to be a Splunk instance, usually a forwarder.
• Cooked Data Protocol: Unlike raw TCP, splunktcp expects data packaged in Splunk's proprietary s2s framing protocol. The stream carries metadata (host, source, sourcetype, index) with the data. Compression is optional (compressed in outputs.conf).
• Parsed vs. unparsed: Both universal and heavy forwarders send cooked S2S data by default (sendCookedData = true). A heavy forwarder has already parsed the data, so the indexer does not parse it again. A universal forwarder sends unparsed data, so the indexer runs the parsing pipelines.
• For encrypted forwarder-to-indexer traffic, configure [splunktcp-ssl:<port>].

TCP vs. UDP Transport Mechanics & Buffer Drops

Choosing between TCP and UDP for network data collection involves significant architectural trade-offs:

The UDP Socket Buffer Drop Hazard

When network devices (switches, routers, firewalls) emit UDP syslog packets, they stream packets into the network interface of the Splunk server. The operating system kernel places incoming datagrams into a socket receive buffer (SO_RCVBUF, requested by Splunk's _rcvbuf setting), and Splunk's UDP input reads them from there.

If a sudden burst of network logs occurs (such as during a network storm, security incident, or interface flap):

1. Datagrams arrive in the receive buffer faster than Splunk reads them.
2. Because UDP has no flow control or backpressure mechanism, the kernel cannot signal the network devices to slow down.
3. Once the kernel buffer reaches capacity, the operating system kernel silently drops all subsequent incoming UDP packets.
4. Neither the sending network device nor Splunk receives an error. The data simply vanishes.

On Linux hosts, you can spot these drops in the UDP counters from netstat -su (abridged example):

$ netstat -su
Udp:
    24891240 packets received
    142 packets to unknown port received
    892410 packet receive errors       # <-- rising counter = datagrams dropped
    24001150 packets sent
    892410 receive buffer errors       # <-- socket buffer was full

Enterprise Syslog Architecture: Aggregators vs. Direct Ingestion

In enterprise environments, thousands of routers, switches, firewalls, and server appliances produce continuous syslog traffic. A fundamental architectural question is whether to direct this traffic straight to Splunk indexers or route it through an intermediate aggregation layer.

The Direct-to-Indexer Anti-Pattern

Novice administrators frequently configure network devices to point their syslog output directly at the IP addresses of Splunk indexers (inputs.conf [udp://514]). This approach causes severe operational failure modes:

1. No High Availability or Maintenance Tolerance: If an indexer is restarted for maintenance, OS updates, or indexer clustering rolling upgrades, its UDP port closes. Because network devices use connectionless UDP, all syslog generated during that window is permanently lost.
2. Imbalanced ingestion: Network devices usually send to one fixed address. Without an external load balancer, the indexer they point at receives all of that data, so data is spread unevenly across the indexers.
3. Data loss during pipeline congestion: If the indexer's queues fill (for example, because of slow disk I/O), the blockage propagates back to the input. A TCP sender is slowed down, but UDP datagrams that cannot be read are dropped.

The Common Best Practice: Dedicated Syslog Collection Tier

To reduce data loss and improve availability, many production architectures deploy a dedicated Syslog Aggregator Tier using tools such as syslog-ng or rsyslog paired with Splunk Universal Forwarders.

+-----------------------+       UDP/TCP 514       +--------------------------------------+
|  Network Appliances   | ----------------------> |      Syslog Aggregator Cluster       |
| (Firewalls, Switches) |                         | (syslog-ng / rsyslog behind VIP/NLB) |
+-----------------------+                         +--------------------------------------+
                                                                     |
                                                          Writes to Local Disk Spool
                                                          /var/log/syslog/<device>/
                                                                     |
                                                                     v
+-----------------------+        TCP 9997 s2s     +--------------------------------------+
| Indexer Cluster Peers | <---------------------- |     Splunk Universal Forwarder       |
| (IDX1, IDX2, IDX3)    |   Load-Balanced Stream  |  (Monitors disk spool & sends s2s)   |
+-----------------------+   with Indexer Ack      +--------------------------------------+

Architectural Workflow of the Aggregator Tier

1. Network Ingestion via High-Availability Syslog Relays:
   • Network devices send syslog traffic to a virtual IP (VIP) managed by a Layer 4 Network Load Balancer (or Keepalived with VRRP).
   • The VIP distributes incoming UDP/TCP 514 traffic across a redundant cluster of dedicated syslog servers running syslog-ng or rsyslog.
2. Local Disk Spooling:
   • The syslog daemon receives the raw packets, parses the originating device header, and writes the logs to dedicated, fast local disk partitions organized dynamically by device category, IP, or date: /var/log/syslog/firewalls/cisco_asa/10.200.1.1/2026-09-23.log
   • Writing to disk decouples network reception from Splunk. If Splunk or the indexers are down, the files keep accumulating, and the forwarder catches up later from its fishbucket position.
3. Splunk Universal Forwarder Collection:
   • A Splunk Universal Forwarder (UF) installed directly on each syslog aggregator server monitors the spool directories using inputs.conf:

     [monitor:///var/log/syslog/firewalls/cisco_asa/*/*.log]
     sourcetype = cisco:asa
     index = netfw
     

4. Reliable, Load-Balanced Transmission over TCP 9997:
   • The Universal Forwarder monitors the files, maintains read progress in its fishbucket, and transmits data over the cooked Splunk-to-Splunk protocol (TCP 9997).
   • The UF automatically load-balances across the indexers, switching to a randomly chosen indexer every 30 seconds by default (autoLBFrequency).
   • With indexer acknowledgment (useACK = true in outputs.conf), the UF keeps each block until an indexer confirms it was written. If no acknowledgment arrives, the UF resends the block, possibly to another indexer. This protects data in flight, although resent blocks can produce occasional duplicates.

Technical Comparison: Direct Syslog vs. Dedicated Aggregator Architecture