6.2 Lens Formulas, Ratios & Anomaly Baseline Metrics

Standard aggregation counts and sums provide essential visibility into security event volume, but in modern enterprise threat monitoring, raw numbers can be deeply misleading. A surge of 1,000 failed authentication events during Monday morning peak login hours might represent a completely normal failure rate of $0.5\%$. Conversely, an identical count of 1,000 failed authentications occurring at 3:00 AM on a Sunday during a total volume of 1,020 attempts represents a catastrophic $98\%$ failure rate indicative of an automated credential stuffing attack.

To detect sophisticated adversaries, security analysts must normalize telemetry into dynamic ratios, evaluate cumulative trends, smooth volatile baselines, and measure rate-of-change velocity. Kibana Lens Formulas provide an expressive mathematical computation layer evaluated directly across aggregated buckets, enabling SIEM analysts to derive advanced anomaly metrics on the fly.

---

Custom Metric Engineering via Lens Formulas

Unlike Elasticsearch runtime fields (which execute Painless scripts on a per-document basis during query execution), Lens formulas operate on aggregated bucket values returned by Elasticsearch. This architectural separation delivers two major benefits:

1. Zero Indexing or Reindexing Overhead: Formulas do not alter index mappings or require CPU-intensive per-record parsing on data nodes.
2. Client-Side Analytical Agility: Formulas execute within Kibana's visualization pipeline, allowing analysts to prototype complex mathematical relationships and immediately render results across millions of underlying documents.

+---------------------------------------------------------------------------------------------------+
|                         LENS FORMULA EVALUATION PIPELINE IN SEC OPS                               |
+---------------------------------------------------------------------------------------------------+
| 1. RAW TELEMETRY SHARDS (Elasticsearch Hot/Warm Tiers)                                            |
|    [Doc 1: outcome=failure] [Doc 2: outcome=success] [Doc 3: outcome=failure] [Doc 4: ...]       |
+---------------------------------------------------------------------------------------------------+
                                                  |
                                                  v (Elasticsearch Aggregation Phase)
+---------------------------------------------------------------------------------------------------+
| 2. AGGREGATED BUCKETS (date_histogram over @timestamp)                                           |
|    Bucket [14:00 - 15:00]: total_docs = 10,000 | failure_docs = 2,500                             |
+---------------------------------------------------------------------------------------------------+
                                                  |
                                                  v (Kibana Lens Formula Evaluation Phase)
+---------------------------------------------------------------------------------------------------+
| 3. LENS FORMULA EXECUTION                                                                         |
|    Formula: count(kql='event.outcome: "failure"') / count() * 100                                |
|    Computation: (2,500 / 10,000) * 100 = 25.0% Failure Rate                                      |
+---------------------------------------------------------------------------------------------------+
                                                  |
                                                  v
+---------------------------------------------------------------------------------------------------+
| 4. RENDERED ANOMALY METRIC ON ANALYST CANVAS (Normalized Failure Percentage Curve)                |
+---------------------------------------------------------------------------------------------------+

Formula Bar Syntax Mechanics

To create a custom metric in Lens, an analyst clicks on a metric slot (such as the Y-axis) and selects Formula. The formula editor supports:

• Standard Arithmetic Operators: + (addition), - (subtraction), * (multiplication), / (division), and () (parenthetical grouping).
• Elasticsearch Metric Functions: count(), sum(field), average(field), min(field), max(field), median(field), percentile(field, percentile=95), standard_deviation(field), unique_count(field) and last_value(field).
• Column (Time-Series) Functions: cumulative_sum(), differences(), moving_average(), counter_rate(), normalize_by_unit() and the overall_sum(), overall_average(), overall_min() and overall_max() family. These need a date histogram (or other ordered) dimension.
• Time Shift: count(shift='1w') evaluates the same metric one week earlier, so count() - count(shift='1w') compares each bucket with last week.
• Filtered Metrics with KQL: The syntax metric(field, kql='...') allows an analyst to evaluate an aggregation against a specific subset of events within the same bucket.

---

Calculating Security Ratios and Dynamic Rates

Ratios normalize volatile raw counts against total activity, neutralizing the visual distortion caused by organic diurnal (day/night) enterprise activity swings.

1. Dynamic Authentication Failure Rate

In authentication telemetry (logs-auth.* or logs-system.*), tracking raw failure counts produces continuous false alarms during high-traffic shifts. A dynamic failure rate formula calculates the exact percentage of authentications that failed:

count(kql='event.category: "authentication" and event.outcome: "failure"') / 
count(kql='event.category: "authentication"') * 100

• If count(kql='event.category: "authentication"') is zero in a given bucket, Lens automatically handles division by zero by rendering a null or empty gap rather than terminating with an error.
• An analyst can apply horizontal threshold reference lines on the chart (e.g., at $15\%$) to trigger visual alerts when the failure proportion crosses anomalous thresholds.

2. Perimeter Block Efficiency Ratio

For firewall and proxy telemetry (logs-network.*), security engineers monitor whether perimeter filtering rules are successfully intercepting threat traffic:

count(kql='event.action: "blocked" or event.action: "denied"') / count() * 100

Sudden drops in this ratio during sustained inbound connection volume indicate potential firewall rule bypasses, misconfigurations, or routing changes that expose internal subnets to raw ingress.

3. Outbound-to-Inbound Byte Asymmetry

During data staging and command-and-control (C2) operations, compromised hosts often transmit significantly more data than they receive. The byte ratio identifies asymmetric endpoints:

sum(destination.bytes, kql='network.direction: "outbound"') / 
sum(source.bytes, kql='network.direction: "inbound"')

A ratio substantially greater than $1.0$ on a workstation endpoint indicates anomalous data transmission worthy of immediate forensic investigation.

---

Cumulative Sum for Progressive Exfiltration Detection

Advanced persistent threat (APT) actors frequently employ "low-and-slow" exfiltration techniques designed to evade volumetric threshold detection rules. Instead of transferring a 100 GB database dump in a single massive burst (which would trigger high-severity SIEM alerts), an attacker transfers 50 MB every 15 minutes over a period of weeks.

On a standard hourly or daily bar chart, these 50 MB transfers blend completely into background network noise. However, by applying the Cumulative Sum function, the analyst transforms the visualization into an aggregated trajectory curve:

cumulative_sum(sum(destination.bytes))

+---------------------------------------------------------------------------------------------------+
|                         HOURLY BURST VS. CUMULATIVE EXFILTRATION TRAJECTORY                       |
+---------------------------------------------------------------------------------------------------+
| STANDARD HOURLY BAR CHART (Exfiltration blips blend into normal traffic):                         |
| 100 MB |   ||       ||        |||       ||        ||       |||       ||                         |
|  50 MB |   ||   ..  ||   ..   |||  ..   ||   ..   ||  ..   |||  ..   ||                         |
|        +-----------------------------------------------------------------                         |
|          Day 1    Day 2     Day 3     Day 4     Day 5    Day 6     Day 7                          |
+---------------------------------------------------------------------------------------------------+
| CUMULATIVE SUM CURVE (Exposes steady, monotonic multi-gigabyte exfiltration):                     |
|  50 GB |                                                             ...---/ (Total: 48 GB)       |
|  40 GB |                                                   ...---///                              |
|  30 GB |                                         ...---///                                        |
|  20 GB |                               ...---///                                                  |
|  10 GB |                     ...---///                                                            |
|   0 GB +-----------...---///---------------------------------------------                         |
|          Day 1    Day 2     Day 3     Day 4     Day 5    Day 6     Day 7                          |
+---------------------------------------------------------------------------------------------------+

Operational Applications of cumulative_sum()

1. Data Leak Quantification: During active containment, incident responders need to report the total volume of data staged or exfiltrated by a compromised host. cumulative_sum() provides an immediate running total without requiring manual spreadsheet calculations.
2. Ransomware Encryption Velocity: Tracking cumulative_sum(count(kql='event.action: "file-rename" or event.action: "file-modified"')) across endpoint file logs visualizes the rapid, exponential curve of automated ransomware encrypting local file systems.

---

Moving Averages for Baseline Smoothing

Security telemetry is inherently noisy. Normal business operations cause high-frequency fluctuations that obscure underlying threat trajectories. To separate signal from noise, analysts use the Lens moving average function.

Function Syntax and Arguments

moving_average(metric, window=5)

• metric: The aggregated metric to smooth, such as count() or sum(network.bytes).
• window: Optional number of buckets to average. Lens averages the last window values to calculate each point, and the default window is 5.
• Like other column functions, moving_average needs a date histogram dimension, so the order of buckets is defined.

The same smoothing is available without writing a formula: choose the Moving average quick function on a metric and set its window.

Constructing an Operational Baseline Curve

Consider an analyst monitoring DNS query volume to identify DNS tunneling or Domain Generation Algorithms (DGA):

• Raw Metric Layer: unique_count(dns.question.name) plotted as a light grey bar chart with 10-minute intervals.
• Smoothed Baseline Layer: A line overlay configured with the Lens formula:

  moving_average(unique_count(dns.question.name), window=6)
  

  Because the interval is 10 minutes, a window of 6 gives a rolling one-hour average.
• Anomaly Identification: When a raw bar rises well above the smoothed line, the analyst sees an acute anomaly rather than routine variance.

Measuring Volatility

Lens has no moving standard-deviation function, but two formulas show burstiness well:

count() - moving_average(count(), window=12)

plots how far each bucket sits above or below its recent average, and standard_deviation(network.bytes) per interval shows how uneven individual connections are within each bucket. Repeated large positive gaps point to automated scanning or bursty C2 traffic.

---

Difference Over Time for Burst and Tampering Detection

While moving averages smooth out rapid changes, security analysts often need to detect the exact opposite: abrupt, violent velocity shifts between consecutive time buckets. The differences() function computes the delta between the current bucket and the preceding bucket:

$\Delta_t = \text{Metric}_t - \text{Metric}_{t-1}$

Formula Syntax

differences(count())

+---------------------------------------------------------------------------------------------------+
|                         VELOCITY DELTA VISUALIZATION VIA differences()                            |
+---------------------------------------------------------------------------------------------------+
|                                                                                                   |
| +20,000 Delta |       ||| (Positive Spike: Acute Brute-Force Attack Initiation)                   |
| +10,000 Delta |       |||                                                                         |
|             0 +-------|||-------------------------------------------------------| Zero Baseline   |
| -10,000 Delta |                                                         |||                       |
| -20,000 Delta |                                                         ||| (Negative Plunge:     |
| -30,000 Delta |                                                         |||  Agent Tampering/     |
|               |                                                         |||  Ingest Drop)         |
+---------------------------------------------------------------------------------------------------+

SecOps Operational Scenarios

1. Acute Attack Initiation (Positive Delta): When an adversary launches a multi-threaded brute-force tool (such as Hydra or Medusa), authentication requests surge from 50 per minute to 25,000 per minute. A positive spike in differences(count()) instantly highlights the exact minute the attack commenced.
2. Defense Evasion and Service Tampering (Negative Delta): Adversaries frequently attempt to conceal their presence by terminating log collectors (e.g., executing net stop Sysmon64 or killing the Elastic Agent process). A sudden, massive negative plunge in differences(count()) on a critical server alerts the SOC to an immediate log ingestion blackout, prompting urgent investigation.

---

Percent of Total and Overall Aggregations

When presenting categorical breakdowns (e.g., breaking down alerts by threat.technique.id or host.name), analysts need to display each entity's relative contribution to the total threat volume across the entire environment.

Lens provides the overall_sum() function to calculate the grand total across all breakdown buckets:

count() / overall_sum(count()) * 100

In a donut chart or horizontal bar chart, this normalizes entity counts into exact percentages of the total observed alert pool, allowing SOC leads to prioritize remediation resources on the techniques causing the highest proportion of incidents.

---

Comprehensive Lens Formula Reference Guide

The following reference table summarizes the essential Lens formula functions, syntax structures, and practical SecOps applications:

Exam Preparation Tip: Know how to read and choose Lens formulas. Filtered metrics use a named kql argument, written in Elastic's examples as count(kql='event.outcome: "failure"'). Column functions such as cumulative_sum(), differences() and moving_average() need a date histogram dimension, and moving_average() takes an optional window (default 5). The metric function is average(), not avg().