5.1 Metric, Aggregation & Histogram Visualizations for SOC

In high-throughput Security Operations Centers (SOCs), analysts cannot evaluate security posture or detect distributed intrusion campaigns by scanning raw, unaggregated event logs document by document. Enterprise telemetry streams generate hundreds of millions of events daily across firewalls, authentication systems, cloud audit trails, and endpoint sensors. Rapid threat detection, anomaly hunting, and operational situational awareness require statistical synthesis.

Elasticsearch delivers this analytical capability through its Aggregation Framework. Rather than merely retrieving documents that match search filters, aggregations execute distributed mathematical computations across shard segments, transforming raw logs into actionable intelligence. For security analysts, mastering aggregation primitives—particularly terms, cardinality, date histograms, and percentiles—is foundational to constructing robust Kibana dashboards, tuning threat detection thresholds, and uncovering adversary tradecraft.

---

Building Aggregation-Based Visualizations in Kibana

Elastic's exam objective asks you to create aggregation-based visualizations for security use cases. In Kibana, aggregation-based is a specific editor type in the Visualize Library, separate from Lens and TSVB. Elastic recommends Lens for most new charts, but the aggregation-based editor is still available and maps directly to Elasticsearch aggregations:

1. Open Visualize Library → Create visualization → Aggregation based.
2. Pick a chart type: Area, Data table, Gauge, Goal, Heat map, Horizontal bar, Line, Metric, Pie, Tag cloud or Vertical bar.
3. Choose the source: a data view or a saved search from Discover. A saved search carries its query and filters with it.
4. Configure Metrics (the Y-axis or cell value, such as Count, Unique Count, Sum or Percentiles).
5. Add Buckets: X-axis, Split series or Split chart for bar, line and area charts; Split rows or Split table for data tables; Split slices for pie charts.
6. Save the visualization to the library and add it to a dashboard.

Best practices for security use cases: start from a narrow data view or saved search, keep the number of buckets readable (top 5 to 10 terms), label axes with the ECS field meaning, and choose the chart type by question: how many over time (line or bar with a date histogram X-axis), which entities dominate (horizontal bar or data table), and how is it distributed (heat map). The sections below explain the aggregation mechanics behind every one of these editor choices.

---

Elasticsearch Aggregation Architecture: Metric vs. Bucket Aggregations

Elasticsearch executes aggregations directly on data nodes using columnar doc values (and in-memory data structures), bypassing the overhead of deserializing complete _source JSON documents. Aggregations fall into two primary architectural families:

+---------------------------------------------------------------------------------------+
|                         ELASTICSEARCH AGGREGATION TAXONOMY                             |
+---------------------------------------------------------------------------------------+
| BUCKET AGGREGATIONS (Document Partitioning)                                           |
| - Creates sets ('buckets') of documents matching specific criteria or values          |
| - Examples: terms, date_histogram, filter, range, ip_range, composite                 |
|                                    |                                                  |
|                                    v (Feeds documents into)                           |
| METRIC AGGREGATIONS (Mathematical Computation)                                        |
| - Computes numerical metrics across documents within each bucket                       |
| - Single-Value Metrics: count (value_count), sum, avg, min, max, cardinality          |
| - Multi-Value Metrics:  percentiles, percentile_ranks, stats, extended_stats          |
+---------------------------------------------------------------------------------------+

1. Bucket Aggregations

Bucket aggregations do not calculate statistical metrics directly; instead, they partition incoming documents into distinct collections known as buckets. Each bucket is associated with a criteria key and a document count (doc_count).

• Terms Aggregation: Dynamically builds buckets for each unique value of a field (e.g., partitioning events by user.name or process.name).
• Date Histogram Aggregation: Dynamically partitions documents into chronological time intervals (e.g., 5-minute, 1-hour, or 1-day windows) along the @timestamp axis.
• Range / IP Range Aggregations: Bins documents into user-defined numerical or subnet ranges (e.g., grouping traffic by internal vs. external CIDR blocks).
• Filter Aggregations: Evaluates a query to define a single bucket containing all matching events (e.g., isolating event.outcome: failure).

2. Metric Aggregations

Metric aggregations calculate mathematical values across the documents contained within a bucket. In Kibana visualizations, metric aggregations define the Y-axis height of bar charts, the values within metric cards, or the cell values in data tables.

• Single-Value Metrics: Produce a single numeric result, such as sum(destination.bytes), avg(event.duration), or cardinality(source.ip).
• Multi-Value Metrics: Produce multiple statistical outputs per bucket, such as percentiles(network.bytes) emitting values for the 50th, 75th, 90th, 95th, and 99th percentiles.

3. Sub-Aggregation Nesting

The true power of the aggregation framework emerges from nesting. Bucket aggregations can contain sub-aggregations (both metric and further bucket aggregations) down multiple tiers. For example, a SOC analyst can configure an outer bucket aggregation on host.name, nest an inner bucket aggregation on user.name, and nest a terminal metric aggregation calculating sum(destination.bytes). This calculates the precise outbound data volume generated by each distinct user on each individual host.

---

Terms Aggregations & The Distributed Shard Top N Challenge

The terms aggregation is one of the most frequently used aggregations in security monitoring. It extracts the most frequent values for a specific field, powering essential SecOps views such as:

• Top Talkers: The top 10 internal hosts transmitting outbound traffic (source.ip).
• Top Targeted Accounts: The top 20 usernames experiencing failed logon attempts (user.name).
• Top Queried C2 Domains: High-frequency suspicious domains in DNS logs (dns.question.registered_domain).
• Top Blocked Network Services: Most frequently targeted destination ports on perimeter firewalls (destination.port).

The Distributed Shard Pitfall

Because Elasticsearch distributes index data across multiple primary shards located on different cluster nodes, terms aggregations operate in a distributed, two-stage fashion:

1. The coordinating node receives the query and broadcasts a terms aggregation request requesting the top $N$ terms (defined by the size parameter) to every shard holding relevant index data.
2. Each shard evaluates its local Lucene segment files, tallies the document counts for that field locally, and returns only its top $N$ terms back to the coordinating node.
3. The coordinating node combines the partial results from all shards, sums the counts for matching keys, and selects the final top $N$ terms to return to Kibana.

+---------------------------------------------------------------------------------------+
|                 DISTRIBUTED TERMS AGGREGATION ESTIMATION ISSUE                        |
+---------------------------------------------------------------------------------------+
| Requested: Top 3 Destination IPs (size = 3)                                          |
|                                                                                       |
| SHARD 1 (Local Top 3):               SHARD 2 (Local Top 3):                           |
| 1. 198.51.100.1 -> 500 events        1. 203.0.113.5  -> 450 events                    |
| 2. 198.51.100.2 -> 300 events        2. 198.51.100.1 -> 400 events                    |
| 3. 198.51.100.3 -> 200 events        3. 203.0.113.8  -> 250 events                    |
| [Unreturned on Shard 1:              [Unreturned on Shard 2:                          |
|  203.0.113.5 -> 180 events (Rank 4)]  198.51.100.2 -> 220 events (Rank 4)]           |
|                                                                                       |
| COORDINATING NODE MERGE (Without Shard Size Tuning):                                 |
| - 198.51.100.1 = 500 + 400 = 900 (Accurate)                                          |
| - 203.0.113.5  = Shard 2 only (450) -> Actual is 450 + 180 = 630 (UNDERCOUNTED!)     |
| - 198.51.100.2 = Shard 1 only (300) -> Actual is 300 + 220 = 520 (UNDERCOUNTED!)     |
+---------------------------------------------------------------------------------------+

When data is distributed unevenly across shards, a term that ranks 4th on every shard might have a higher aggregate count across the entire cluster than a term that ranks 1st on one shard and does not appear on others. Because each shard only returns its top $N$ candidates, unreturned counts are omitted, leading to term count inaccuracies or missing terms in the final Top N list.

Tuning shard_size

To eliminate or minimize this distributed calculation error, Elasticsearch provides the shard_size parameter. The shard_size setting controls how many candidate terms each individual shard returns to the coordinating node for the final merge.

• By default, Elasticsearch automatically calculates shard_size as: $\text{shard\_size} = (\text{size} \times 1.5) + 10$
• While this default provides a reasonable heuristic for general search, high-cardinality security data (such as public IP addresses, URLs, or file hashes) often requires explicit manual tuning.
• In Kibana, Lens's Top values function offers an accuracy mode option that asks each shard for more candidate terms, and the aggregation-based editor accepts an explicit shard_size (e.g., 500 or 1000) through the aggregation's advanced JSON input.
• Setting shard_size equal to or greater than the number of unique terms in the index produces 100% exact results across all shards, though at the expense of increased network serialization and heap memory consumption on coordinating nodes.

Evaluating Estimation Accuracy

Elasticsearch returns two diagnostic metrics in every terms aggregation response:

1. doc_count_error_upper_bound: The maximum possible number of documents that could have been missed for any term included in the final Top N list. If this value is 0, the returned counts are guaranteed to be exact.
2. sum_other_doc_count: The total number of documents that matched the query but belonged to terms that fell outside the Top N list. In threat hunting, an unusually massive sum_other_doc_count indicates that event activity is heavily dispersed across thousands of low-frequency entities (long-tail behavior).

---

Count vs. Cardinality: Tracking Unique Entities in SecOps

In security monitoring, there is a fundamental difference between evaluating the frequency of events (value_count) and evaluating the uniqueness of entities (cardinality):

• Value Count (count): Tallies the total number of documents containing a field. An analyst observing 10,000 failed logon events knows that an attack occurred, but cannot determine whether it was a single user being brute-forced or thousands of accounts being targeted.
• Cardinality (cardinality): Calculates the approximate count of unique, distinct values for a field. Evaluating cardinality(user.name) over those 10,000 failed logons immediately reveals whether 1 account was targeted 10,000 times (targeted brute force) or 10,000 accounts were targeted once each (distributed password spraying).

SecOps Detection Scenarios Driven by Cardinality

1. Distributed Password Spraying: Filtering for event.outcome: failure and bucketing by source.ip, then calculating cardinality(user.name). A single external IP address attempting authentication against 50+ distinct usernames within 10 minutes represents a high-confidence password spray.
2. Horizontal Port Scanning: Bucketing network connection events by source.ip and calculating cardinality(destination.ip) and cardinality(destination.port). A host contacting hundreds of distinct destination ports across internal subnets within seconds indicates active reconnaissance.
3. DNS DGA (Domain Generation Algorithm) Detection: Bucketing DNS queries by host.id or source.ip and calculating cardinality(dns.question.name). Infected endpoints executing DGA algorithms generate thousands of distinct, algorithmically randomized domain lookups per hour.

The HyperLogLog++ (HLL++) Algorithm & precision_threshold

Computing exact distinct counts across hundreds of millions of records requires keeping every unique observed term in memory, which quickly exhausts JVM heap space in large clusters. To prevent out-of-memory crashes, Elasticsearch implements the HyperLogLog++ (HLL++) probabilistic algorithm for cardinality aggregations.

• Logarithmic Memory Footprint: HLL++ hashes incoming values using a 64-bit hash function and monitors the distribution of leading zeros in the hash outputs. It estimates distinct counts with high accuracy while consuming a fixed, tiny memory footprint.
• precision_threshold Parameter: Governs the trade-off between memory consumption and statistical accuracy. The parameter defines the threshold below which counts are expected to be close to 100% exact.
  • The default value is 3000 (maximum configurable is 40000).
  • For cardinalities below the precision_threshold, accuracy is typically within fractions of a percent.
  • Memory overhead is bounded: at the maximum threshold of 40000, the aggregation consumes approximately $40000 \times 8\text{ bytes} \approx 320\text{ KB}$ of heap memory per bucket.
  • For low-volume threat detection (e.g., tracking unique admin accounts logging into a domain controller, typically < 100), counts far below the default precision_threshold of 3000 are expected to be very close to exact, at negligible resource cost.

---

Percentiles & Median Metrics for Threat Detection

When analyzing numeric security telemetry—such as connection duration (event.duration), network transfer sizes (network.bytes), or query response times—analysts frequently make the mistake of using the arithmetic average (avg). In security operations, arithmetic averages are deeply deceptive.

The Arithmetic Average Trap

Consider 1,000 network connections between an internal workstation and an external server:

• 999 connections are benign keep-alive packets transferring 200 bytes each (total: 199,800 bytes).
• 1 connection is an adversary staging and exfiltrating a compressed database backup transferring 500 megabytes (524,288,000 bytes).
• The arithmetic mean (avg) is:
  $\frac{199,800 + 524,288,000}{1000} \approx 524,488\text{ bytes} \approx 512\text{ KB}$

If an analyst creates a baseline alert triggering when outbound connections exceed an average of 100 KB, the aggregate metric will show 512 KB, completely obscuring whether every session transferred 500 KB or whether a massive half-gigabyte data theft occurred. Conversely, if 10,000 sessions transferred 200 bytes and one transferred 50 MB, the average is only 5 KB, completely hiding the 50 MB exfiltration event.

Percentile Analysis via T-Digest

Elasticsearch calculates percentiles using the T-Digest algorithm, an online clustering algorithm that generates approximate rank-based statistics with high accuracy at the extreme tails (the 1st and 99th percentiles):

+---------------------------------------------------------------------------------------+
|                   PERCENTILE DISTRIBUTION IN NETWORK FORENSICS                        |
+---------------------------------------------------------------------------------------+
| Metric: network.bytes per connection (Workstation -> Unknown External Host)           |
|                                                                                       |
| p50 (Median) :     210 bytes  -> 50% of connections transferred <= 210 bytes          |
| p75          :     240 bytes  -> 75% of connections transferred <= 240 bytes          |
| p90          :     350 bytes  -> 90% of connections transferred <= 350 bytes          |
| p95          :     800 bytes  -> 95% of connections transferred <= 800 bytes          |
| p99          : 524,000 bytes  -> Top 1% of connections transferred up to 524 KB!      |
|                                                                                       |
| INTERPRETATION: Sharp divergence between p95 and p99 indicates low-frequency,          |
| high-volume anomalous data transfer characteristic of staging or exfiltration.        |
+---------------------------------------------------------------------------------------+

SecOps Threat Hunting Applications for Percentiles

1. Command and Control (C2) Jitter Analysis: Calculating percentiles on delta-times between consecutive outbound connections. Malware beacons programmed with 10% jitter exhibit tightly clustered percentiles (p50 and p95 are almost identical), whereas human browsing exhibits massive variance.
2. Authentication Latency Spikes: Monitoring p99 of authentication response times. Credential attacks targeting directory services cause heavy queueing, creating sharp spikes in p95 and p99 response times while p50 remains relatively stable.
3. DNS Tunneling Inspection: Calculating percentiles of dns.question.name length. Normal lookups tend to be short, while tunneling tools pack encoded data into long labels, so the upper percentiles of query length jump sharply. Set thresholds from your own baseline.

---

Date Histogram Aggregations & Interval Tuning

The date_histogram aggregation is the temporal engine of Kibana, binning time-stamped events along the @timestamp axis to reveal attack trends, event surges, and behavioral shifts.

Calendar vs. Fixed Intervals

Elasticsearch supports two distinct interval definitions:

1. Fixed Intervals (fixed_interval): Fixed increments of time defined by SI units: seconds (s), minutes (m), hours (h), or days (d as exactly 24 hours / 86,400 seconds). For example, 1m, 5m, or 1h. Fixed intervals do not adjust for daylight saving transitions or calendar anomalies.
2. Calendar Intervals (calendar_interval): Understand calendar boundaries: day (1d), week (1w), month (1M), or year (1y). A calendar day bucket starts at midnight in the target timezone and accommodates 23, 24, or 25 hours during daylight saving transitions.

Kibana "Auto" Interval Behavior

In Kibana Lens and Discover, the interval is frequently set to Auto. When Auto is selected, Kibana inspects the active time picker range (e.g., Last 15 minutes vs. Last 30 days) and automatically selects an interval that produces a readable number of buckets (guided by the histogram:barTarget and histogram:maxBars advanced settings, 50 and 100 by default).

• Over Last 15 minutes, Auto selects an interval of 10s or 30s.
• Over Last 24 hours, Auto selects 30m or 1h.
• Over Last 90 days, Auto selects 1d or 1w.

This dynamic scaling prevents the browser from attempting to render 500,000 distinct bars (which would freeze the browser DOM) while preventing cluster coordination nodes from merging millions of temporal buckets.

The Danger of Implicit Gap-Skipping: Zero-Event Intervals

By default, a date_histogram fills gaps inside the range of the data with empty buckets (min_doc_count defaults to 0), but it only builds buckets from the first matching document to the last one. Intervals with no events at the start or end of the selected time range are missing unless extended_bounds is set. Gaps can also disappear if someone sets min_doc_count: 1. In Lens, the date histogram's Include empty rows option (on by default) sets min_doc_count: 0 and extends buckets to the full time range. Turning it off hides empty intervals.

In security operations, empty intervals represent vital intelligence:

• An endpoint sensor suddenly transmitting zero events indicates that an adversary may have executed net stop on the Elastic Agent, tampered with Sysmon, or severed network connectivity.
• A firewall interface showing zero dropped packets over a 2-hour window indicates an upstream routing failure, a policy bypass, or log shipping disruption.

If empty buckets are omitted, the visualization can draw a continuous line between Friday evening and Monday morning, or simply end early, hiding a 48-hour logging blackout. To guarantee a continuous timeline across the whole query window, the aggregation needs both of these settings:

{
  "aggs": {
    "events_over_time": {
      "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "1h",
        "min_doc_count": 0,
        "extended_bounds": {
          "min": "now-24h/h",
          "max": "now/h"
        }
      }
    }
  }
}

• min_doc_count: 0: Instructs Elasticsearch to emit buckets even when zero documents match that interval.
• extended_bounds: Defines the explicit minimum and maximum time boundaries for the histogram, ensuring that if zero events occurred at the beginning or end of the query window, empty buckets are still rendered.

---

Comparison Table of Aggregation Types in Security Visualizations

The following reference table summarizes the primary aggregation types utilized across SecOps visualizations and threat hunting workflows: