16.3 Validating Event Parsing with Data Preview

Validating Event Parsing with Data Preview

Quick Summary: Before onboarding a new source everywhere, preview it on the Set Source Type page of Settings > Add Data. The page shows how Splunk will break the sample into events and which timestamps it will extract, and lets you adjust the settings until they are right. Then save the settings as a source type and deploy the props.conf stanza to the tier where each setting takes effect, usually the indexers or heavy forwarders. Afterward, watch splunkd.log for DateParserVerbose, LineBreakingProcessor, and AggregatorMiningProcessor warnings.

---

The Data Preview Interface in Splunk Web

Onboarding a new source without validation risks badly broken events, wrong timestamps, or blocked queues caused by an expensive regex. The data preview on the Set Source Type page lets you check all of this on a sample first.

Navigation and Workflow

1. Log into Splunk Web with administrative credentials.
2. Navigate to Settings > Add Data.
3. Choose Upload to send a representative sample file from your workstation, or Monitor to point at a file the instance can read.
4. Select the file and click Next to reach the Set Source Type page, which is the data preview.

Architectural Mechanics of Data Preview

• Nothing is indexed yet: the preview shows how the sample would be broken into events and timestamped. Data is indexed only if you continue through Input Settings, Review, and Submit.
• Preview on a test instance: preview uses the parsing settings of the instance you are logged in to. Many teams preview on a non-production instance and then onboard the data for real with the saved settings.

The Configuration Tabs in Data Preview

The left pane of the Set Source Type page has four collapsible panels. The Event Breaks panel appears only when Splunk cannot determine how to break the file, or when the chosen source type does not define line breaking. Delimited settings appears only for structured data such as CSV. Click Apply settings to refresh the preview. Changes made in Advanced take precedence over conflicting Event Breaks or Timestamp choices.

---

Validating Event Breaks, Timestamps, Delimiters & Field Extractions

When inspecting the visual rendering in Data Preview, administrators must verify three core criteria before approving the configuration:

1. Visual Verification of Event Breaks

• Event separation: the preview lists events one by one, numbered, with the extracted time next to each.
• Multiline Integrity: Verify that multiline events—such as Java stack traces or XML blocks—remain grouped within a single event block rather than fragmenting into dozens of individual single-line events.
• No Runaway Merging: Ensure that distinct single-line events do not merge into a single massive block due to an overly restrictive break regex.

2. Visual Verification of Timestamp Extraction

• Highlighted timestamp: the text Splunk used as the timestamp is highlighted in the event.
• Time column: the parsed time appears next to each event. Compare it with the text; a mismatch means the wrong string was used or the time zone is wrong.
• Warnings: the preview flags events for which it could not find a timestamp.
• Integrity Check: If the highlighted text does not match the actual date string (e.g., if an IP address or customer ID is highlighted instead), the TIME_PREFIX or TIME_FORMAT is incorrect.

3. Delimited Field Mapping and Column Structure

• For CSV or TSV data, verify that the column header row properly maps to each data column below it.
• Check that embedded commas inside quoted strings (e.g., "San Francisco, CA") do not split into separate columns.

[!IMPORTANT] Use a representative sample: a few hand-made lines prove little. Use real data that includes the awkward cases, such as multi-line stack traces, daylight saving changes, lines without timestamps, and the longest events you expect.

---

Exporting Generated Stanzas and Enterprise Deployment Architecture

When the preview looks right, click Save As to save the settings as a named source type (for example corp:app:json), with a description, category, and app. The Advanced panel lists the exact settings, so you can copy them into a props.conf stanza for deployment elsewhere.

Example Generated props.conf Stanza

[corp:app:json]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)\{"timestamp":
TIME_PREFIX = \{"timestamp":"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%z
MAX_TIMESTAMP_LOOKAHEAD = 32
TRUNCATE = 200000

The Deployment Tier Rule: Where Does props.conf Belong?

A frequent architectural mistake is deploying parsing stanzas to the wrong tier in a distributed environment:

Rule: Parse-time settings (LINE_BREAKER, SHOULD_LINEMERGE, TRUNCATE, TIME_PREFIX, TIME_FORMAT, TZ, DATETIME_CONFIG, TRANSFORMS) belong on the first full Splunk instance that handles the data. Input-time settings (CHARSET, NO_BINARY_CHECK, EVENT_BREAKER, INDEXED_EXTRACTIONS) belong on the instance that reads the data, which can be a universal forwarder. Search-time settings (EXTRACT, REPORT, KV_MODE, FIELDALIAS) belong on the search heads.

Tier Placement Scenarios

1. Universal Forwarder (UF) -> Indexer Cluster (Standard Enterprise Model):

   • Universal forwarders do not run the parsing pipelines, so LINE_BREAKER or TIME_FORMAT placed on them has no effect.
   • Search heads do not break lines or extract timestamps at index time either.
   • Action: put the stanza in an app under $SPLUNK_HOME/etc/manager-apps/<app_name>/ on the cluster manager (the older master-apps directory is deprecated), then distribute it to all peers with splunk apply cluster-bundle.

2. Universal Forwarder (UF) -> Heavy Forwarder (HF) -> Indexer Cluster:

   • A heavy forwarder is a full Splunk instance, so it parses the data it receives before forwarding it. The indexers do not parse it again.
   • Action: deploy the app containing props.conf to the heavy forwarders, typically with the deployment server. The indexers need the parse-time settings only for data that reaches them unparsed.

---

Troubleshooting Parsing Errors in splunkd.log

When parsing issues occur in production, splunkd records diagnostic warnings and errors in $SPLUNK_HOME/var/log/splunk/splunkd.log. Administrators can search these events using the _internal index.

Diagnostic SPL Query

index=_internal sourcetype=splunkd component IN (DateParserVerbose, LineBreakingProcessor, AggregatorMiningProcessor) log_level IN (WARN, ERROR)
| stats count by component, event_message
| sort - count

Key Ingestion Processors and Error Signatures

Queue Backpressure Diagnostics

If metrics.log (group=queue) shows blocked=true for parsingQueue or aggQueue, or the Monitoring Console shows those queues full:

• Check for catastrophic backtracking in custom LINE_BREAKER or TIME_PREFIX regular expressions.
• Nested quantifiers (such as (.*a)+) can keep a CPU core busy for a long time on long lines, and the blocked pipeline then backs up every input feeding it.
• Also check line merging: a source type left on SHOULD_LINEMERGE = true puts extra work on aggQueue.