14.1 Hypothesis-Driven Threat Hunting & Baselining in Elastic

The Philosophy & Mandate of Proactive Threat Hunting

Modern enterprise security architectures generate massive volumes of telemetry across endpoints, networks, cloud infrastructures, and identity providers. Detection engines continuously evaluate this streaming data against thousands of behavioral rules, threat intelligence feeds, and machine learning models. However, automated detection rules possess inherent structural limitations: they are predominantly reactive, designed to identify known indicators of compromise (IOCs), specific signature patterns, or previously modeled behavioral thresholds. Advanced Persistent Threats (APTs), targeted cyber espionage operators, and sophisticated cybercrime syndicates intentionally develop tradecraft designed to evade automated alerting—leveraging zero-day exploits, fileless in-memory execution, stolen administrative credentials, and native system utilities.

Threat hunting is the proactive, iterative, and human-driven search through networks and endpoints to detect malicious adversary activities that have bypassed existing automated security controls. Rather than waiting for the Detection Engine to trigger an alert in .alerts-security.alerts-*, a threat hunter operates under the fundamental premise of assumed breach: the adversary is already inside the environment, possesses valid credentials, executes commands using trusted system tooling, and leaves faint anomalies within normalized Elastic Common Schema (ECS) data streams.

To conduct effective hunts without becoming overwhelmed by millions of benign events, analysts must apply structured methodologies. In Elastic Security, this requires formulating testable hypotheses based on threat intelligence, constructing behavioral baselines using Kibana Lens and ES|QL, and performing statistical outlier analysis to expose hidden attacker tradecraft.

---

Proactive Threat Hunting vs. Reactive Alert Triage

While alert triage and threat hunting are complementary functions within a Security Operations Center (SOC), their operational triggers, analytical mindsets, and workflows differ fundamentally:

---

The Hypothesis-Driven Threat Hunting Lifecycle

Unfocused, ad-hoc log browsing rarely uncovers sophisticated threats; it rapidly degrades into aimless data scrolling. Elastic Security supports a repeatable, rigorous Hypothesis-Driven Threat Hunting Lifecycle comprising six distinct phases:

+-------------------------------------------------------------------------+
|             Hypothesis-Driven Threat Hunting Lifecycle in Elastic       |
|                                                                         |
|   +--------------------------+                                          |
|   | 1. Formulate Hypothesis  | <------------------------------------+   |
|   +--------------------------+                                      |   |
|                | (Threat Intel, MITRE ATT&CK, Asset Criticality)     |   |
|                v                                                     |   |
|   +--------------------------+                                      |   |
|   | 2. Telemetry Validation  |                                      |   |
|   +--------------------------+                                      |   |
|                | (Verify ECS Streams: logs-endpoint.*, Network, Auth)|   |
|                v                                                     |   |
|   +--------------------------+                                      |   |
|   | 3. Baseline Construction |                                      |   |
|   +--------------------------+                                      |   |
|                | (Lens, Discover, ES|QL: Profile Normal Admin Usage) |   |
|                v                                                     |   |
|   +--------------------------+                                      |   |
|   | 4. Iterative Hunting     |                                      |   |
|   +--------------------------+                                      |   |
|                | (Long-Tail Analysis, LOLBins, EQL Sequences)        |   |
|                v                                                     |   |
|         Adversary Found?                                            |   |
|         /              \                                            |   |
|     [Yes]              [No]                                         |   |
|       |                  |                                          |   |
|       v                  v                                          |   |
|  +----------+     +-------------------+                             |   |
|  | 5. Scope |     | Document Baseline |                             |   |
|  | & Case   |     | Insights & Gaps   |                             |   |
|  +----------+     +-------------------+                             |   |
|       |                  |                                          |   |
|       +--------+---------+                                          |   |
|                |                                                    |   |
|                v                                                    |   |
|   +--------------------------+                                      |   |
|   | 6. Detection Engineering | -------------------------------------+   |
|   |    Rule Automation       | (Feed custom EQL/KQL rules back into SIEM)|
|   +--------------------------+                                          |
+-------------------------------------------------------------------------+

Phase 1: Hypothesis Formulation

A hunt hypothesis is an educated, testable proposition asserting how an adversary might currently be operating within the network without triggering an alert. High-quality hypotheses derive from:

• Threat Intelligence: CISA advisories, threat actor profiling, or Mandiant reports detailing newly observed techniques.
• MITRE ATT&CK Matrix: Tactical focus on under-monitored sub-techniques (e.g., T1059.001 PowerShell, T1218 Signed Binary Proxy Execution).
• High-Value Assets: Targeting mission-critical infrastructure, domain controllers, or payment gateways where stealthy persistence would be catastrophic.

Example of a poor hypothesis: "Attackers are hacking our servers." Example of an actionable hypothesis: "Adversaries are executing living-off-the-land binaries such as certutil.exe or bitsadmin.exe with command-line arguments specifying external HTTP/HTTPS endpoints to download second-stage payloads into user-writable directories without triggering existing web gateway alerts."

Phase 2: Telemetry & Data Coverage Validation

Before writing queries, hunters must verify that the requisite data is actually collected, normalized, and accessible within the active Security Data View. For the certutil.exe hypothesis, the hunter validates that:

• Endpoint data streams (logs-endpoint.events.process-* or winlogbeat-*) actively ingest process creation events.
• Key ECS fields are consistently populated: process.name, process.executable, process.command_line, process.parent.name, and user.name.
• Script block logging or command-line capture has not been suppressed by endpoint configuration policies.

Phase 3: Baseline Construction

To identify an anomaly, the hunter must first quantify what constitutes normal operations. Hunting queries applied to a raw dataset will return thousands of hits from legitimate IT management tools, software deployments, and system administrators. Establishing baselines defines the expected operational profile over 30 to 90 days.

Phase 4: Iterative Hunting & Query Execution

The hunter executes targeted queries using Discover, ES|QL, and Kibana Lens, progressively eliminating verified administrative patterns. When an interesting event appears, the hunter pivots: inspecting the parent process in the visual event analyzer, examining associated network sockets in Timeline, and expanding the temporal window.

Phase 5: Incident Escalation & Response

If malicious activity is confirmed, the hunter transitions immediately into incident handling: opening an Elastic Security Case, pinning timeline evidence, noting indicators of compromise (IOCs), and isolating affected endpoints via Elastic Defend.

Phase 6: Detection Engineering & Rule Automation

The ultimate deliverable of a successful threat hunt is operational obsolescence: once a technique has been hunted and scoped, the hunter collaborates with detection engineers to translate the hunt query into an automated Detection Engine rule (e.g., an EQL correlation or custom threshold rule), ensuring the SOC never has to manually hunt for that exact behavior again.

---

Establishing Behavioral Baselines with Lens, Discover, and ES|QL

A baseline is a statistical or behavioral model of expected activity across enterprise entities. Without baselines, threat hunters mistake legitimate administrative tooling for active cyber attacks. Elastic Security provides specialized tools to establish baselines across several key dimensions:

1. Baselining Dimensions

• Administrative Tooling: Profiling which service accounts and administrative workstations execute utilities like powershell.exe, psexec.exe, net.exe, or dsquery.exe. Legitimate administrative tooling follows predictable schedules (e.g., automated backup scripts running daily at 02:00 UTC from an IT subnet).
• Parent-Child Process Execution: In standard operating environments, process execution lineages adhere to strict hierarchies. Normal parents for cmd.exe or powershell.exe include explorer.exe, code.exe, or deployment agents (CcmExec.exe). Conversely, office productivity suites (winword.exe, excel.exe), web server daemons (w3wp.exe, httpd), or print spoolers (spoolsv.exe) spawning command interpreters are severe behavioral anomalies.
• Network Egress Ports & Destinations: Establishing normal outbound destination ports per network segment. Workstation subnets routinely communicate externally over ports 80, 443, and 53. Unprecedented egress on ports 4444, 8080, 8443, or 1337 originating from internal database servers represents an immediate anomaly.
• Scheduled Tasks & Services: Documenting persistent system tasks. Legitimate tasks point to binaries within C:\Windows\System32\ or C:\Program Files\. Tasks pointing to user-writable paths such as C:\Users\*\AppData\Local\Temp\ or C:\ProgramData\ warrant immediate scrutiny.

2. Baselining with ES|QL

Elasticsearch Query Language (ES|QL) provides native data transformation and aggregation capabilities, allowing hunters to compute baselines over extended time horizons directly in the query pipeline.

FROM logs-endpoint.events.process-*
| WHERE event.category == "process" AND event.type == "start"
| STATS 
    execution_count = COUNT(),
    host_count = COUNT_DISTINCT(host.name),
    user_count = COUNT_DISTINCT(user.name)
  BY process.parent.name, process.name
| SORT execution_count DESC
| LIMIT 50

This query produces a ranked baseline of parent-child relationships across the fleet. High execution_count pairs (such as services.exe -> svchost.exe with thousands of executions across hundreds of hosts) represent established baseline activity. Relationships appearing with a host_count of 1 and an execution_count under 3 represent statistical outliers.

3. Visual Baselines in Kibana Lens

Kibana Lens enables visual baselining by plotting multi-layer time-series metrics:

• Top N Process Execution Ratios: A Lens stacked area chart displaying the ratio of native Windows administrative executables executed per department.
• Heatmaps of Off-Hours Activity: A Lens heatmap tracking authentication events or process execution volume grouped by user.name across the 24-hour day. Administrative scripts executed during regular business hours establish the visual baseline; executions occurring at 03:30 local time immediately stand out as visual anomalies.

---

Hunting for Hidden Adversary Tradecraft

When adversaries attempt to evade detection, they frequently manipulate execution mechanisms. Hunters apply specific analytical patterns in Elastic to unmask these activities:

1. Long-Tail (Least-Frequent Value) Analysis

Automated detection rules typically trigger when an event count exceeds an upper threshold. Attackers counter this by executing malicious commands only once or twice across an entire campaign. Long-tail analysis inverts standard threshold monitoring by searching for the least-frequent values in a dataset—finding the rare, low-frequency "needle" in the corporate haystack.

Using ES|QL, a hunter isolates rare command-line invocations across thousands of endpoints:

FROM logs-endpoint.events.process-*
| WHERE event.category == "process" AND event.type == "start"
| STATS 
    total_runs = COUNT(),
    unique_hosts = COUNT_DISTINCT(host.name)
  BY process.executable, process.command_line
| WHERE total_runs < 5 AND unique_hosts == 1
| SORT total_runs ASC
| LIMIT 100

By filtering for total_runs < 5 and unique_hosts == 1, the hunter eliminates standard enterprise software and exposes bespoke adversary commands, such as one-off PowerShell download cradles or custom compiled binaries.

2. Hunting Living-off-the-Land Binaries (LOLBins)

Adversaries increasingly avoid dropping custom compiled malware onto disk. Instead, they weaponize pre-installed, digitally signed operating system utilities—known as Living-off-the-Land Binaries (LOLBins) or LOLBAS (Living Off The Land Binaries and Scripts). Because these executables are signed by Microsoft or trusted vendors, traditional signature antivirus ignores them.

Common LOLBins hunted in Elastic Security include:

• certutil.exe: Native certificate management utility frequently abused to download remote malware payloads using -urlcache -split -f or to base64-decode payloads via -decode.
• bitsadmin.exe: Background Intelligent Transfer Service utility abused to asynchronously download malicious files (/transfer).
• mshta.exe: Microsoft HTML Application host abused to execute inline VBScript, JScript, or remote .hta files via HTTP/HTTPS URLs.
• regsvr32.exe: Abused via "Squiblydoo" techniques to download and execute remote COM scriptlets (.sct) using /s /n /u /i:http://... scrobj.dll.
• rundll32.exe: Abused to execute exported functions from arbitrary DLLs, bypassing application allowlisting.

An EQL hunting query (run it in Timeline's Correlation tab, or save it as an event correlation rule) that isolates LOLBin download activity. EQL's : operator is case-insensitive and accepts a list of wildcard patterns:

process where event.type == "start" and
  process.name in~ ("certutil.exe", "bitsadmin.exe", "mshta.exe", "regsvr32.exe") and
  process.command_line : ("*http:*", "*https:*", "*ftp:*", "*-urlcache*", "*scrobj.dll*", "*javascript:*", "*vbscript:*")

The KQL equivalent needs or between values and unquoted, escaped wildcards, for example process.name: ("certutil.exe" or "bitsadmin.exe") and process.command_line: (*http\:* or *-urlcache*).

3. Hunting Unquoted Service Paths

An unquoted service path vulnerability occurs when a Windows service binary path containing spaces is registered without surrounding quotation marks (e.g., C:\Program Files\Enterprise Software\service.exe). When the Service Control Manager starts the service, Windows interprets spaces as command delimiters, sequentially attempting to execute:

1. C:\Program.exe
2. C:\Program Files\Enterprise.exe
3. C:\Program Files\Enterprise Software\service.exe

If an unprivileged user has write permissions to C:\ or C:\Program Files\, they can drop a malicious Program.exe to achieve local privilege escalation to NT AUTHORITY\SYSTEM upon service restart.

Hunters look for newly installed services (System log Event ID 7045) whose image path contains a space but does not start with a quote. An ES|QL example over the System integration's System channel data:

FROM logs-system.system-*
| WHERE event.code == "7045"
    AND winlog.event_data.ImagePath LIKE "* *"
    AND NOT winlog.event_data.ImagePath LIKE "\"*"
| KEEP @timestamp, host.name, winlog.event_data.ServiceName, winlog.event_data.ImagePath

Review the results by hand: paths such as C:\Windows\system32\svchost.exe -k netsvcs contain spaces only in their arguments and are not vulnerable.

4. Command-Line Obfuscation Analysis

Adversaries routinely obfuscate command lines using environment variable concatenation, string reversals, tick marks (`), carats (^), or base64 encoding to evade basic keyword filters. Hunters search for execution patterns with high entropy, irregular character ratios, or encoded parameters (such as powershell.exe -e, -enc, or -encodedcommand).

---

Operationalizing Threat Hunt Findings

A hunt is not complete when an analyst identifies an anomaly. To provide durable enterprise value, findings must be operationalized:

1. Case Creation & Containment: The analyst creates an Elastic Security Case, links all relevant Timeline evidence, records adversary IOCs, and initiates host containment via Elastic Defend.
2. Rule Codification: The specific behavioral pattern identified during the hunt is codified into an automated Detection Engine rule. If the hunt successfully identified LOLBin abuse via certutil.exe, a custom EQL rule is deployed to automatically alert the SOC on future occurrences.
3. Exception Refinement: Any legitimate business use cases discovered during the hunt baseline are formally added to the new rule as Rule Exceptions, preventing false positive fatigue for operational triage teams.