4.1 Kibana Discover, Security Data Views & Runtime Fields

In modern security operations centers (SOCs), rapid triage depends on an analyst's ability to pivot seamlessly from macroscopic threat indicators to microscopic raw event records. Kibana Discover serves as the ground-truth investigation workspace within the Elastic Stack, enabling security engineers and analysts to inspect unstructured and structured telemetry, uncover event spikes across time boundaries, and formulate exploratory threat hypotheses.

Understanding how Discover interacts with Elasticsearch indices through Security Data Views and dynamically parses incoming data via Runtime Fields is essential for effective security monitoring, rapid incident response, and performance-conscious query design.

---

Navigating Kibana Discover in Security Operations

Discover exposes the raw event stream indexed into Elasticsearch. While curated dashboards and the Elastic Security app offer specialized analytical abstractions, Discover provides unfiltered access to fields defined in the Elastic Common Schema (ECS).

+---------------------------------------------------------------------------------------+
|                                 KIBANA DISCOVER UI                                    |
+---------------------------------------------------------------------------------------+
| [Data View: logs-* v]  [KQL: host.name: "srv-web01" and event.outcome: "failure"     ] |
| [Time Picker: Last 24 Hours (Absolute: 2026-09-29T00:00:00Z to 2026-09-30T00:00:00Z)] |
+---------------------------------------------------------------------------------------+
| EVENT DISTRIBUTION HISTOGRAM (Binned by 30-minute intervals)                          |
|  ||   |||        ||||||||||| (Anomaly Spike: 14:30 UTC)        ||                     |
|  ||   |||   ..   |||||||||||                              ..   ||     ..              |
+---------------------------------------------------------------------------------------+
| FIELD SIDEBAR       | CUSTOM DOCUMENT TABLE                                           |
| Selected Fields:    | @timestamp           | host.name  | user.name | process.name    |
| - @timestamp        | 2026-09-29T14:30:12Z | srv-web01  | root      | sshd            |
| - host.name         | 2026-09-29T14:30:14Z | srv-web01  | admin     | sshd            |
| - user.name         | 2026-09-29T14:30:17Z | srv-web01  | test      | sshd            |
| - process.name      |                                                                 |
| Available Fields... | [> Expand Doc] -> Flyout (JSON / Single-Doc / Surrounding Docs) |
+---------------------------------------------------------------------------------------+

Time Picker Mechanics

Accurate temporal scoping prevents query timeouts and cluster resource exhaustion while ensuring analysts capture relevant evidence:

1. Quick Intervals: Pre-configured dynamic windows (e.g., Last 15 minutes, Last 24 hours, Last 7 days). These automatically calculate relative offsets from the current execution moment.
2. Relative Time: Configured with custom lookbacks, such as now-4h to now, or rounding boundaries like now-1d/d (looking back one full calendar day rounded to midnight).
3. Absolute Time: Fixed ISO 8601 start and end boundaries (e.g., 2026-09-29T08:00:00.000Z to 2026-09-29T12:00:00.000Z). Absolute ranges are mandatory when reconstructing forensic evidence during incident containment.
4. Auto-Refresh: Live-polling interval (e.g., refresh every 5 seconds or 1 minute) used on SOC overview monitors tracking active intrusion scenarios.

Analyst Tip on Timezones: Elasticsearch natively stores all dates internally in Coordinated Universal Time (UTC) formatted as ISO 8601. Kibana translates timestamps into the browser's local timezone by default. In multi-region incident response, analysts should standardize Kibana's Advanced Settings (dateFormat:tz) to UTC to avoid temporal misalignments across distributed investigation teams.

The Event Distribution Histogram

The histogram positioned directly above the document table plots event frequency over the selected time range. It relies on an Elasticsearch date_histogram aggregation, automatically binning counts into intervals (seconds, minutes, hours, or days) determined by the total query span.

• Spike Detection: Sudden vertical surges often indicate high-velocity events such as brute-force authentication attacks, port scans, or network fuzzing.
• Ingestion Troughs & Blackouts: Complete drops in event counts indicate ingestion pipeline failures, log shipper crashes, or network partitioning between endpoints and Fleet Server.
• Interactive Temporal Zooming: Clicking and dragging across an anomalous histogram peak immediately constrains the Discover time picker to that exact micro-window, filtering millions of background records down to the intrusion burst.

Document Table Customization

By default, Discover renders matching documents with the @timestamp column and the full unparsed _source document. In high-volume triage, this creates visual noise. Analysts customize the view by:

• Adding Columns from the Field Sidebar: Hovering over ECS fields (e.g., host.name, user.name, event.action, process.name, source.ip, destination.ip) and selecting Add creates a clean, tabular presentation.
• Column Sorting: Clicking column headers toggles ascending or descending sort orders. Elasticsearch executes this sorting against the indexed values of keyword, date, or numeric fields.
• Field Statistics: Selecting a field in the sidebar shows its top values and their share of the current documents, allowing analysts to spot outlier usernames or unusual destination ports immediately.

Document Flyout & Surrounding Documents

Clicking the expand icon next to any event opens the Document Flyout, displaying structured key-value pairs categorized by ECS domain, as well as the raw JSON payload in _source.

From the flyout, analysts can apply instantaneous inclusion filters (+) or exclusion filters (-), check field mapping definitions, or copy field values. Crucially, the flyout offers View Surrounding Documents:

• In incident triage, an alert typically fires on a single isolated event (e.g., an unauthorized PowerShell execution).
• Selecting View surrounding documents opens a context view showing a configurable number of documents immediately before and after the selected event, in time order, from the same data view.
• The context view does not automatically limit results to the same host. Add a filter such as host.name: "srv-dc01" to focus it on one entity. Kibana sorts by the data view's time field and uses tiebreaker fields (the context:tieBreakerFields advanced setting, _doc by default) to keep a stable order when events share a timestamp.

---

Security Data Views Architecture

A Data View (formerly known as an Index Pattern in older Elastic Stack versions) informs Kibana which Elasticsearch indices, data streams, or index aliases contain the telemetry an analyst wishes to explore.

Stream Matching and Wildcards

Data Views use glob-style wildcards to target specific subsets of security data:

• logs-*: Matches all general log data streams across Windows, Linux, network appliances, and cloud workloads.
• logs-endpoint.events.*: Targets endpoint sensor activity generated by the Elastic Defend integration.
• .alerts-security.alerts-default: The hidden alias holding detection alerts for the default space (one alias per Kibana space, .alerts-security.alerts-<space-id>).
• auditbeat-*,packetbeat-*,filebeat-*: Targets legacy Beat shippers.

Multiple comma-separated streams can be combined in a single Data View, enabling cross-source correlation across disparate infrastructure layers.

The Security app uses its own default data view (security-solution-<space-id>), built from the securitySolution:defaultIndex advanced setting. By default it includes Beats and Elastic Agent patterns such as logs-*, filebeat-*, auditbeat-*, packetbeat-*, winlogbeat-* and endgame-*, plus -*elastic-cloud-logs-*, which excludes Elastic Cloud's own logs.

Default Timestamp Configuration

Every time-based Data View requires a primary time field, almost universally configured as @timestamp in ECS-compliant environments. This field governs:

• The chronological axis of the Discover histogram.
• The range filter applied by the Kibana time picker.
• Time-based shard skipping: Elasticsearch can skip shards whose time range cannot match the selected window, which keeps long-retention searches fast.

Field Formatters

Data Views allow administrators to define presentation rules for fields without modifying underlying documents:

• String Formatters: Enforcing uppercase, lowercase, or regex substring transforms.
• URL Templates: Transforming IP addresses, domain names, or file hashes into clickable hyperlinks directing analysts to internal asset databases or external threat intelligence lookups (e.g., VirusTotal or internal ticketing systems).
• Bytes & Duration Formatters: Formatting raw integer values (e.g., 1048576 bytes to 1 MB, or 3500 ms to 3.5s) for rapid operational interpretation.

---

Runtime Fields in Security Operations

A Runtime Field is an Elasticsearch field evaluated dynamically at query time using a Painless script. Unlike indexed fields, which are parsed and written to immutable inverted indices or BKD trees during ingestion, runtime fields calculate their values on the fly whenever a query executes.

+---------------------------------------------------------------------------------+
|                      INDEXED FIELDS VS RUNTIME FIELDS                           |
+---------------------------------------------------------------------------------+
| INGEST-TIME PATH (Indexed Fields):                                              |
| Log Ingestion -> Ingest Pipeline -> Inverted Index / Doc Values -> Shard Storage |
| [Disk footprint high; Ingest CPU cost; Instantaneous query execution at scale]   |
+---------------------------------------------------------------------------------+
| QUERY-TIME PATH (Runtime Fields):                                               |
| Query Execution -> Shard Scan -> Painless Script Evaluated -> Results Emitted   |
| [Zero disk storage; Zero reindexing; Compute CPU overhead during search]        |
+---------------------------------------------------------------------------------+

Scopes: Ephemeral vs. Saved Runtime Fields

1. Saved Runtime Fields in Data Views: Configured directly in Kibana under Stack Management > Data Views. Once saved, the field appears in the Discover sidebar and can be queried using KQL, visualized in Kibana Lens, and referenced in detection rules just like any pre-indexed ECS field.
2. Index-Level Runtime Fields: Defined directly inside the Elasticsearch index mapping (_mapping). Any query hitting that index across any Kibana space or API client can consume the field.
3. Ephemeral Runtime Fields: Defined dynamically within the body of a specific search request using the runtime_mappings parameter. These exist exclusively for the duration of that single query execution, perfect for ad-hoc hunting scripts.

Painless Scripting for Ad-Hoc Parameter Parsing

When new attack techniques emerge or unmapped logs arrive without proper ingest pipelines, waiting for reindexing or pipeline redeployment impedes urgent threat containment. Runtime fields allow SOC analysts to extract critical indicators immediately.

Example 1: Extracting DNS Subdomains

Consider a scenario where dns.question.name contains subdomain.attacker-c2.com, but the SOC needs to aggregate queries by the top-level domain or extract high-entropy subdomains used for DNS tunneling.

// Painless script to extract the first subdomain label
String domain = doc['dns.question.name'].value;
if (domain != null) {
  int firstDot = domain.indexOf('.');
  if (firstDot > 0) {
    emit(domain.substring(0, firstDot));
  } else {
    emit(domain);
  }
}

Example 2: Calculating Session Duration

Calculating session length in seconds across authentication sessions where event.start and event.end exist as separate timestamps:

if (doc.containsKey('event.start') && doc.containsKey('event.end') &&
    !doc['event.start'].empty && !doc['event.end'].empty) {
  long durationMs = doc['event.end'].value.toEpochMilli() - doc['event.start'].value.toEpochMilli();
  emit(durationMs / 1000.0);
}

Architectural Tradeoffs in Enterprise SOC Clusters

While runtime fields provide unprecedented agility, deploying them across high-volume security data streams introduces serious performance considerations:

Best Practice Lifecycle: Treat runtime fields as an exploratory staging ground. When a runtime field proves indispensable for automated alerting or high-frequency dashboards, migrate the parsing logic upstream into an Elastic Ingest Pipeline processor (dissect, grok, or script) so future logs are indexed natively.