5.2 Data Tables, Heatmaps & Specialized Security Charts

While time-series line charts and single-metric indicators provide essential macroscopic visibility into security event volume, deep forensic triage requires multi-dimensional decomposition. When evaluating complex threat activities—such as lateral movement, persistent command-and-control (C2) communications, or privilege abuse—analysts must analyze relationships across multiple categorical fields simultaneously.

Kibana provides a suite of specialized visualization structures designed for multi-variable correlation and threat detection. These include hierarchical Data Tables, two-dimensional temporal Heatmaps, compliance-driven Gauge and Goal charts, and exploratory Tag Clouds. Mastering these visualization types allows security engineers to build operational dashboards that rapidly transition from broad situational indicators to granular forensic evidence.

---

Data Tables with Multi-Tier Nested Bucket Aggregations

The Data Table is the most analytically granular visualization in Kibana. Unlike raw document tables in Discover, which display individual unaggregated logs, a visualization Data Table groups telemetry into hierarchical aggregation tiers, calculating statistical metrics at each layer of the hierarchy.

+---------------------------------------------------------------------------------------+
|                      MULTI-TIER NESTED SECOPS DATA TABLE                              |
+---------------------------------------------------------------------------------------+
| Tier 1: destination.ip (Top 3)                                                        |
|   --> Tier 2: destination.port (Sub-buckets per IP)                                   |
|         --> Tier 3: event.outcome (Outcome per Port)                                  |
|               --> Metric: sum(destination.bytes) & count(*)                           |
+--------------------+------------------+---------------+---------------+---------------+
| DESTINATION IP     | DESTINATION PORT | EVENT OUTCOME | TOTAL BYTES   | EVENT COUNT   |
+--------------------+------------------+---------------+---------------+---------------+
| 198.51.100.24      | 443              | success       | 485.2 MB      | 14,230        |
|                    | 443              | failure       | 12.4 KB       | 42            |
|                    | 8443             | success       | 1.8 GB [RED]  | 8,912         |
| 203.0.113.89       | 22               | failure       | 145.8 KB      | 1,204 [AMBER] |
|                    | 22               | success       | 3.2 MB        | 2             |
| 192.0.2.15         | 445              | failure       | 84.1 KB       | 650           |
+--------------------+------------------+---------------+---------------+---------------+

Constructing Multi-Tier Aggregation Trees

To build a forensic investigation table, an analyst configures successive Split Rows bucket aggregations in Kibana Lens or classic Data Tables:

1. Primary Bucket (Tier 1): Defines the primary entity under investigation. For network exfiltration hunting, this is typically destination.ip or source.ip.
2. Secondary Bucket (Tier 2): Subdivides each parent bucket by a secondary analytical dimension, such as destination.port or network.transport.
3. Tertiary Bucket (Tier 3): Further partitions the sub-buckets by operational state, such as event.outcome (success vs. failure) or user.name.
4. Metrics: Across each leaf bucket, Kibana computes aggregated metrics, including count(*), sum(destination.bytes), avg(event.duration), or cardinality(source.ip).

Shard Execution Order and Sub-Bucket Truncation Dynamics

Security analysts must understand how Elasticsearch processes nested bucket aggregations across distributed shards to avoid misinterpreting table results:

• Elasticsearch evaluates nested aggregations top-down. Shards first identify the top $N$ terms for the Tier 1 aggregation (e.g., top 10 destination IPs based on byte volume).
• The inner Tier 2 aggregation is calculated exclusively for documents that fall within those top 10 Tier 1 buckets.
• If a destination IP had high port variance or high failure counts but was ranked 11th in total byte volume, it will be completely excluded from the table, along with all of its sub-buckets.
• Operational Rule: When sorting a Data Table by a sub-bucket metric, the primary bucket must be sorted by that same metric (or configured with a sufficiently large size and shard_size) to ensure the parent entities containing the highest sub-metrics are retrieved.

Metric Formatting and Conditional Highlighting

Raw integers in security tables introduce cognitive friction during high-stress incident triage. Kibana Data Tables allow field formatters and conditional rules:

• Bytes Formatting: Automatically renders raw integers (1844674407) into human-readable units (1.8 GB, 24.5 MB).
• Duration Formatting: Converts raw nanosecond or millisecond integers into readable time units (42ms, 1.4m, 2.8h).
• Conditional Color Highlighting: Sets dynamic cell background or text colors based on numerical thresholds:
  • Red background when sum(destination.bytes) > 1073741824 (1 GB outbound transfer).
  • Amber background when count(*) of event.outcome: failure exceeds 500 (brute-force threshold).
  • Green text when compliance authentication ratios exceed 99.5%.

---

Heatmap Visualizations: Uncovering Temporal and Behavioral Threat Patterns

A Heatmap represents three dimensions of security data across a two-dimensional grid: an X-axis category, a Y-axis category, and a color intensity value representing an aggregated metric (such as event count or sum of bytes). In security operations, Heatmaps are predominantly deployed for temporal distribution analysis.

+---------------------------------------------------------------------------------------+
|                   DAY-OF-WEEK VS. HOUR-OF-DAY TEMPORAL HEATMAP                        |
+---------------------------------------------------------------------------------------+
| Y: Day of Week | X: Hour of Day (00 to 23 UTC)                                       |
|                | 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23
+----------------+----------------------------------------------------------------------+
| Monday         |  .  .  .  .  .  .  .  . ## ## ## ## ## ## ## ## ##  .  .  .  .  .  . |
| Tuesday        |  .  .  .  .  .  .  .  . ## ## ## ## ## ## ## ## ##  .  .  .  .  .  . |
| Wednesday      |  .  .  .  .  .  .  .  . ## ## ## ## ## ## ## ## ##  .  .  .  .  .  . |
| Thursday       |  .  .  .  .  .  .  .  . ## ## ## ## ## ## ## ## ##  .  .  .  .  .  . |
| Friday         |  .  .  .  .  .  .  .  . ## ## ## ## ## ## ## ## ##  .  .  .  .  .  . |
| Saturday       |  .  .  . [!!] .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  . |
| Sunday         |  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  . |
+----------------+----------------------------------------------------------------------+
| LEGEND: [.] Low/Zero (Benign)  [##] Normal Business Hours Traffic                     |
|         [!!] HIGH-DENSITY ANOMALOUS ACTIVITY (Saturday 03:00 UTC - Threat Indicator)  |
+---------------------------------------------------------------------------------------+

Constructing Temporal Heatmaps via Runtime Fields

To build a Day-of-Week vs. Hour-of-Day heatmap, the X and Y axes must extract temporal components from @timestamp:

1. X-Axis (Hour of Day): Configured as a Histogram aggregation on a runtime field extracting hour (values 0 through 23), or using a date histogram with interval offsets.
2. Y-Axis (Day of Week): Configured as a Terms aggregation on a runtime field extracting day names or numeric day values (0 to 6, where 0 is Sunday or Monday).

// Painless script to extract Day of Week for Heatmap Y-Axis
ZonedDateTime zdt = doc['@timestamp'].value;
emit(zdt.getDayOfWeek().toString());

// Painless script to extract Hour of Day for Heatmap X-Axis
ZonedDateTime zdt = doc['@timestamp'].value;
emit(zdt.getHour());

Color Palettes, Ranges and Scale

How you map values to colors decides whether an analyst sees an intrusion or overlooks it:

• Simple linear ramp: Color intensity is proportional to the count. If business hours generate 100,000 events per cell and an off-hours brute-force burst generates 800 events at 03:00 on Sunday, the burst is under 1% of peak intensity and looks almost as pale as an empty cell.
• Custom color ranges: Both Lens and aggregation-based heatmaps let you define color ranges or stops. Giving small counts (for example 1 to 1,000) their own distinct color makes any activity in quiet periods stand out, whatever happens during business hours.
• Non-linear scaling: A logarithmic scale ($y = \log(x)$) compresses large disparities in the same way. Use it where the chart offers it, or approximate it with ranges that widen as the values grow.

Whichever method you use, the goal is the same: off-hours cells with modest counts must be clearly different from cells with zero events.

Threat Hunting Signatures in Heatmaps

1. Off-Hours Administrative Logins: Filtering for user.name: ('Administrator' or 'root' or 'aws_admin') and event.outcome: success. Normal operations display tight clusters Monday through Friday between 08:00 and 18:00. Dense cells appearing on Saturday at 03:00 AM represent high-priority anomalies requiring immediate investigation.
2. Scheduled Malware C2 Beaconing: Uncompromised endpoint traffic follows diurnal human working patterns (traffic peaks during the day and falls at night). Automated malware persistence mechanisms executing cron jobs or scheduled beaconing produce horizontal stripes spanning all 24 hours continuously.
3. Automated Password Spraying Bursts: Scripted attacks targeting external portals appear as intense, localized vertical columns across brief 1-hour or 2-hour bands, cutting across days of the week.
4. Ingestion Blackouts and Agent Failures: Solid white vertical stripes spanning all days at a specific hour indicate recurring operational disruptions, such as network maintenance drops or automated snapshot freezes.

---

Gauge and Goal Charts for Governance, Compliance & SOC SLAs

While Data Tables and Heatmaps serve exploratory threat hunting, Gauge and Goal visualizations provide operational metrics for SOC leadership, shift supervisors, and compliance auditors.

+---------------------------------------------------------------------------------------+
|                             GAUGE VS. GOAL ARCHITECTURE                               |
+---------------------------------------------------------------------------------------+
| GOAL VISUALIZATION: Progress Toward Target Metric                                     |
| Target: 100% Endpoint Fleet Coverage                                                  |
| [=======================================>            ] 88.4% (Target: 100.0%)         |
| Status: In Progress | Current Active Sensors: 8,840 / 10,000 Nodes                    |
+---------------------------------------------------------------------------------------+
| GAUGE VISUALIZATION: Meter Divided into Operational Severity Bands                    |
| Metric: Open Critical Alerts Awaiting Triage                                          |
|                                                                                       |
|                     . - ~ ~ ~ - .
|                 . '    [NORMAL]   ' .
|               /     (Green: 0-10)     \
|              |                         |
|             |     /                     |  <-- Needle pointing at 28
|             |    /  [ELEVATED]          |      Band: RED (CRITICAL SLA BREACH)
|              |  / (Amber: 11-20)       |
|               \                       /
|                 ' .   [CRITICAL]    . '
|                     ' - (Red: 21+) - '
+---------------------------------------------------------------------------------------+

1. Goal Charts

A Goal chart displays progress toward a single fixed numerical target. The visualization renders a progress bar or cylinder filling up as the metric approaches the predefined goal value.

• Endpoint Sensor Rollout: Tracking the percentage of enterprise assets running the Elastic Agent with Elastic Defend enabled against a 100% compliance goal.
• Log Source Onboarding: Monitoring daily ingestion volume against a contractually licensed daily ingestion limit (e.g., 500 GB/day).
• Vulnerability Remediation: Displaying the percentage of critical patch tickets resolved within the mandated 14-day SLA window.

2. Gauge Charts

A Gauge chart displays a dial or semi-circular speedometer divided into color-coded operational severity bands (e.g., Green, Yellow, Red). A needle or filled arc indicates the current metric value relative to these thresholds.

• Mean Time to Acknowledge (MTTA): Monitoring the average time (in minutes) for Tier 1 analysts to triage incoming critical alerts. Thresholds: Green (0–15 mins), Yellow (16–30 mins), Red (>30 mins).
• Active Alert Queue Backlog: Displaying the total number of unassigned alerts currently in the open state.
• Authentication Failure Ratio: Calculating the percentage of total authentications that resulted in failure:
  $\text{Failure Rate} = \frac{\text{Failures}}{\text{Successes} + \text{Failures}} \times 100$ Thresholds: Normal (< 5% / Green), Suspicious (5–15% / Yellow), Active Attack (> 15% / Red).

---

Tag Clouds for Exploratory Threat Hunting

A Tag Cloud visualizes the output of a Terms aggregation by rendering words in varying font sizes and weights. The size of each word is proportional to the value of the associated metric (typically document count or sum of bytes).

Operational Hunting Workflows

While Tag Clouds lack the precision required for formal compliance reporting, they serve as high-speed exploratory launching pads during broad threat hunting sprints:

1. Hunting Living-Off-The-Land Binaries (LOLBins): Filtering for process executions where process.parent.name: ('cmd.exe' or 'powershell.exe' or 'wscript.exe') and generating a Tag Cloud on process.name. Dominant benign processes (such as git.exe or conhost.exe) appear large, while anomalous utilities (such as certutil.exe, vssadmin.exe, bitsadmin.exe, or whoami.exe) stand out for immediate investigation.
2. User-Agent Anomaly Detection: Generating a Tag Cloud on http.request.user_agent.original across web proxy logs. Standard corporate browsers appear as massive text blocks. Unusual, truncated, or default script User-Agents (such as curl/7.68.0, python-requests/2.25.1, Go-http-client/1.1, or Mozilla/4.0 (compatible;)) appear clearly on the periphery.
3. Suspicious Service Installations: Aggregating Windows Event ID 7045 (A new service was installed) by service.name to identify randomly generated or masqueraded service names.

Analytical Limitations of Tag Clouds in SecOps

Security analysts must exercise caution when relying on Tag Clouds:

• No Chronological Context: A Tag Cloud cannot indicate when an event occurred. 50 executions spread evenly across 30 days look identical to 50 executions concentrated within a 10-second brute-force burst.
• Suppression of Rare High-Severity Threats: The most dangerous adversary tradecraft in an enterprise network is often low-frequency (the "signal in the noise"). A sophisticated C2 agent or zero-day exploit might execute only once. In a Tag Cloud weighted by count, a single execution will be rendered in the smallest possible font or dropped entirely by the terms size limit.
• Imprecise Comparison: Human visual perception cannot reliably discern small differences in word surface area, making it difficult to determine whether term A has 15% or 30% more events than term B.

---

Reference Table: Specialized SecOps Chart Types & Operational Strengths