9.3 Event Correlation Rules with Event Query Language (EQL)

The Necessity of Stateful Correlation in Modern Threat Detection

Single-event detection rules (such as Custom Query and Threshold rules) evaluate documents in isolation. However, sophisticated adversaries rarely execute an entire attack in a single command. Real-world cyber attacks unfold as a progressive sequence of actions across the cyber kill chain: an initial exploit or phishing email triggers process execution, which drops a payload to disk, modifies a persistence registry key, and ultimately establishes an encrypted outbound network beacon to a Command and Control (C2) server.

Evaluating any one of these events in isolation frequently leads to either false positives or false negatives. For example, running powershell.exe is common administrative behavior. Creating a file in C:\Users\Public\ may be benign. Establishing an outbound HTTPS connection is normal web traffic. But when the same PowerShell process drops an executable file and immediately opens an external network socket, the sequence represents a high-confidence attack. Event Query Language (EQL) is Elastic's purpose-built declarative query language designed specifically to correlate chronological event sequences across diverse ECS telemetry categories.

---

EQL Syntax Fundamentals: Event Types and Conditions

EQL is native to Elasticsearch and operates directly on ECS-compliant documents. Unlike KQL or Lucene, which operate on broad document fields, EQL queries begin by specifying the event category followed by a conditional expression enclosed in a where block:

category where condition

+-------------------------------------------------------------+
| EQL Event Anatomy                                           |
|                                                             |
|   process where event.type == "start" and                   |
|     process.name == "cmd.exe" and                           |
|     process.parent.name == "winword.exe"                    |
|   \_____/       \______________________________________/    |
|      |                             |                        |
|  Event Category              Boolean Conditions             |
+-------------------------------------------------------------+

1. ECS Event Categories

EQL maps directly to the ECS event.category field. Common EQL event categories include:

• process: Process creation, start, and termination events (event.category: "process").
• file: File creation, modification, and deletion events (event.category: "file").
• network: Inbound and outbound network connections, DNS queries, and flow records (event.category: "network").
• registry: Windows registry key and value modifications (event.category: "registry").
• authentication: User logons, Kerberos ticket grants, and logoffs (event.category: "authentication").

2. Operators and Functions

EQL supports rich comparison operators and string evaluation functions:

• Comparison Operators: equality (==), inequality (!=), and relational numeric comparisons (less than, less than or equal to, greater than, greater than or equal to).
• Logical Operators: and, or, not
• Set Membership: process.name in ("powershell.exe", "pwsh.exe", "cmd.exe")
• Wildcard and Case-Insensitive Matching:
  • == is an exact, case-sensitive comparison; : is a case-insensitive comparison that also accepts * and ? wildcards: process.name : "POWERSHELL.EXE"
  • like matches wildcards case-sensitively and like~ case-insensitively: process.command_line like~ "*downloadstring*"
  • in checks a list case-sensitively and in~ case-insensitively: process.name in~ ("powershell.exe", "pwsh.exe")
• Strings: Always use double quotes. EQL does not accept single-quoted strings.
• String Functions: concat(), stringContains(), length(), startsWith(), endsWith()

---

Sequence Queries and Temporal Constraints: sequence with maxspan

The true power of EQL lies in sequence queries. A sequence specifies two or more chronological steps that must occur in an exact temporal order.

sequence with maxspan=5m
  [process where event.type == "start" and process.name == "mshta.exe"]
  [network where event.type == "start" and destination.port in (80, 443)]

1. The maxspan Parameter

The maxspan clause defines the maximum allowable time window between the first matching event and the final matching event in the sequence:

• Format: maxspan=5m (5 minutes), maxspan=1h (1 hour), maxspan=30s (30 seconds).
• If Step 1 occurs at 10:00:00 and Step 2 occurs at 10:06:00 with maxspan=5m, no alert is generated because the elapsed time (6 minutes) exceeds the constraint.

2. Strict Chronological Ordering

EQL enforces temporal ordering based on @timestamp, with an optional tiebreaker field (such as event.sequence) set in the rule's EQL settings for events with identical timestamps. In the sequence above, the network event must occur after the process start event. If the network event occurred at 10:00:00 and mshta.exe started at 10:01:00, the sequence does not match.

---

Correlation Keys: The by Keyword and Entity Binding

In an enterprise generating millions of events across thousands of endpoints, simply verifying that Step 1 and Step 2 occurred within 5 minutes across the organization is insufficient. An mshta.exe process executing on Workstation A and a network connection occurring on Server B within 5 minutes are completely unrelated.

EQL solves this using correlation keys defined via the by keyword. Correlation keys bind sequence steps to shared entity identifiers:

sequence with maxspan=5m
  [process where event.type == "start" and process.name == "mshta.exe"] by host.id, process.entity_id
  [network where event.type == "start" and destination.port in (80, 443)] by host.id, process.entity_id

Timeline:
  T0: Host A (PID 1024 / mshta.exe) Starts
       | [Correlation Key: host.id = A, process.entity_id = 1024]
       v
  T1: Host B (PID 9988 / curl.exe) Connects Outbound
       | [Correlation Key: host.id = B, process.entity_id = 9988] -> NO MATCH (Wrong Host/PID)
       v
  T2: Host A (PID 2048 / svchost.exe) Connects Outbound
       | [Correlation Key: host.id = A, process.entity_id = 2048] -> NO MATCH (Wrong PID)
       v
  T3: Host A (PID 1024 / mshta.exe) Connects Outbound to Port 443
       | [Correlation Key: host.id = A, process.entity_id = 1024] -> EXACT MATCH!
       v
  [ ALERT SIGNAL GENERATED: Full Attack Path Correlated ]

Why process.entity_id is Mandatory

In Windows and Linux operating systems, OS Process IDs (PIDs) are recycled over time. Furthermore, two different machines frequently run processes with the exact same numerical PID. Elastic Defend generates a globally unique identifier for every running process stored in process.entity_id.

• Binding by by host.id, process.entity_id guarantees that the network connection was initiated by the exact same running process instance that spawned mshta.exe on that specific host.

Cross-Field Relationship Binding

Correlation keys are not restricted to identical field names. EQL allows binding parent and child relationships across sequence steps:

sequence with maxspan=1m
  [process where event.type == "start" and process.name == "cmd.exe"] by host.id, process.entity_id
  [process where event.type == "start" and process.name == "whoami.exe"] by host.id, process.parent.entity_id

Here, the sequence binds the process.entity_id of Step 1 to the process.parent.entity_id of Step 2, mathematically verifying that cmd.exe directly spawned whoami.exe.

---

Real-World Threat Scenarios with EQL

Scenario 1: Living off the Land Binaries (LOLBins) & In-Memory Execution

Adversaries frequently use trusted system binaries to download and execute malicious code, bypassing traditional application allow-listing.

sequence with maxspan=5m
  [process where event.type == "start" and 
   process.name in~ ("powershell.exe", "pwsh.exe", "cmd.exe") and
   process.args in~ ("-enc", "-encodedcommand", "-e")] by host.id, process.entity_id
  [file where event.action in ("creation", "modification") and 
   file.extension in ("exe", "dll", "vbs", "ps1")] by host.id, process.entity_id
  [network where event.type == "start" and 
   not cidrmatch(destination.ip, "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16")] by host.id, process.entity_id

• Analysis: Detects an encoded PowerShell or command script that drops an executable or script to disk and establishes an outbound non-RFC1918 internet connection, all within 5 minutes and executed by the same process.

Scenario 2: LSASS Memory Dumping for Credential Theft

Adversaries target the Local Security Authority Subsystem Service (lsass.exe) to extract plaintext passwords, NTLM hashes, and Kerberos tickets.

sequence with maxspan=3m
  [process where event.type == "start" and 
   process.name in~ ("rundll32.exe", "procdump.exe", "procdump64.exe")] by host.id, process.entity_id
  [file where event.action in ("creation", "modification") and 
   file.extension in ("dmp", "dump") and 
   file.path : ("*\\AppData\\Local\\Temp\\*", "*\\Users\\Public\\*")] by host.id, process.entity_id

• Analysis: Correlates the execution of common memory dumping utilities (rundll32 calling comsvcs.dll or Sysinternals procdump) with the creation of a .dmp file in temporary directories.

Scenario 3: Persistence Mechanism Creation via Scheduled Tasks

Adversaries establish persistence by scheduling recurring tasks after staging malicious scripts.

sequence with maxspan=10m
  [file where event.action == "creation" and 
   file.path : "C:\\Users\\*\\AppData\\Roaming\\*.bat"] by host.id
  [process where event.type == "start" and 
   process.name == "schtasks.exe" and 
   process.args : "/create"] by host.id

• Analysis: Correlates the staging of a batch file in user AppData with an immediate schtasks /create invocation on the same host.

---

Advanced Sequence Modifiers: The until Cancellation Clause

In complex operational environments, a sequence of events might look suspicious only if an expected authorizing or terminating event fails to occur. The until keyword defines a cancellation condition that terminates sequence matching.

sequence with maxspan=15m
  [process where event.type == "start" and process.name == "vpn_client.exe"] by host.id, user.name
  [network where event.type == "start" and destination.port == 22] by host.id, user.name
until
  [authentication where event.action == "mfa_success"] by host.id, user.name

How until Operates:

• If vpn_client.exe starts, the Detection Engine enters a pending sequence state for that (host.id, user.name) tuple.
• If an mfa_success authentication event occurs before the network connection to port 22, the pending sequence state is immediately discarded and reset.
• If the network connection occurs without any intervening MFA validation within 15 minutes, the sequence completes and generates an alert.
• Operational Benefit: Drastically reduces false positives by factoring in expected administrative validation steps.

---

EQL Missing Events: Detecting Non-Occurrences

Traditional detection rules identify when a bad event happens. EQL missing event clauses detect when an expected event does not happen inside a time-bounded sequence. Prefix a clause with !:

sequence by host.name, user.name with maxspan=5s
  [ authentication where event.code : "4624" ]
  ![ authentication where event.code : "4647" ]

This example from Elastic's EQL reference finds logons that are not followed by a logoff within 5 seconds. The rules are:

• with maxspan is mandatory whenever a sequence contains a missing-event clause.
• Missing-event clauses can appear at the beginning, in the middle, or at the end of a sequence, and a sequence can have several, but it needs at least one positive clause.
• Event correlation rules support this syntax, so you can alert on absences directly.

SecOps Use Cases for Missing-Event Sequences:

1. Endpoint Tampering: A security service stops and is not restarted within a few minutes:

   sequence by host.id with maxspan=5m
     [process where event.type == "end" and process.name : "elastic-agent.exe"]
     ![process where event.type == "start" and process.name : "elastic-agent.exe"]
   

2. Backup Pipeline Failure: A snapshot job starts, but no completion event follows within the expected window.

Missing-event sequences give proactive alerting on stealthy defense evasion, such as adversaries terminating logging daemons or suppressing reporting agents.

---

EQL Syntax and Construction Reference

The following reference table summarizes the essential syntax elements and keywords of Event Query Language in Elastic Security: