9.1 Detection Engine Architecture & Prebuilt Rule Management

The Central Role of the Detection Engine in Elastic Security

In modern enterprise SecOps environments, ingesting millions of telemetry events every minute into Elasticsearch is only the foundational step of a security program. Without automated, continuous evaluation of that incoming stream against known adversary tactics, techniques, and procedures (TTPs), threat detection remains purely reactive. The Elastic Security Detection Engine serves as the real-time analytical core of the Elastic SIEM architecture. It continuously evaluates incoming telemetry against hundreds of scheduled queries, correlation patterns, and behavioral heuristics, transforming terabytes of raw telemetry into prioritized, actionable security alerts.

Rather than executing ad-hoc searches inside an analyst's interactive session, the Detection Engine operates autonomously within the Elastic Stack. Mastering its distributed execution model, underlying storage structures, prebuilt rule catalogs, and upgrade mechanics is essential for any certified SIEM analyst responsible for enterprise threat detection.

---

Distributed Execution Architecture: Kibana Task Manager

The Detection Engine is implemented as a set of background microservices running within Kibana. It delegates rule scheduling and execution coordination to the Kibana Task Manager.

+-------------------------------------------------------------+
|                     Kibana Task Manager                     |
|  - Polls .kibana_task_manager index for due tasks           |
|  - Claims task (type per rule, e.g. alerting:siem.queryRule)|
|  - Coordinates distributed workers across Kibana nodes      |
+-------------------------------------------------------------+
                               |
                               v
+-------------------------------------------------------------+
|                  Elasticsearch Cluster Core                 |
|  - Executes Query DSL / EQL against target data streams     |
|  - Searches: logs-endpoint.*, logs-system.*, logs-network.* |
|  - Evaluates search window: [now - (interval + look-back)]  |
+-------------------------------------------------------------+
                               |
                               v
+-------------------------------------------------------------+
|                     Signal Generation                       |
|  - Evaluates matching hits against rule logic & exceptions   |
|  - Skips duplicates using deterministic alert IDs           |
|  - Writes alerts to .alerts-security.alerts-{space_id}      |
+-------------------------------------------------------------+
                               |
                               v
+-------------------------------------------------------------+
|                 Downstream Actions & SOC UI                 |
|  - Updates Elastic Security Alerts UI table & flyouts       |
|  - Dispatches action connectors (Slack, Jira, ServiceNow)   |
+-------------------------------------------------------------+

1. The Task Lifecycle and Lease Mechanism

Each enabled detection rule registers a recurring task with the Kibana Task Manager, recorded in the internal .kibana_task_manager index under a task type for its rule type (for example alerting:siem.queryRule or alerting:siem.eqlRule):

• Task Polling: Kibana worker processes periodically poll the task manager index for tasks whose scheduled execution timestamp has arrived.
• Task Claiming and Leasing: When a Kibana node claims a rule task, it writes an ownership lease with an expiration timeout into the task document. This prevents multiple Kibana instances in a load-balanced cluster from executing the exact same rule simultaneously.
• Execution Isolation: If an individual rule query times out or fails due to an index error, the task manager records the error state in the rule execution logs without disrupting other active detection tasks running across the cluster.

2. Query Execution against Elasticsearch Shards

Once a rule task is claimed, the Detection Engine translates the rule's criteria (whether written in KQL, Lucene, EQL, or ES|QL) into an optimized Elasticsearch search request. This query targets the specific data views or index patterns assigned to the rule (for example, logs-endpoint.events.*, logs-system.*, or winlogbeat-*).

The query executes directly against the Elasticsearch data nodes hosting the target index shards. Elasticsearch handles the heavy lifting of data filtering, aggregations, and document scoring, returning only matching hits or aggregated metric buckets back to the Detection Engine.

3. Alert Destination: Space-Partitioned Alert Indices

When query results satisfy the rule criteria (and are not filtered out by rule exceptions), the Detection Engine constructs an alert document (historically termed a "signal"). Alerts are written directly into a dedicated, space-partitioned hidden index in Elasticsearch:

.alerts-security.alerts-{space_id}

For example, in the default Kibana Space, alerts are indexed into .alerts-security.alerts-default. In a dedicated security space named soc-finance, alerts are stored in .alerts-security.alerts-soc-finance.

Architectural Benefits of Partitioned Alert Indices:

• Multi-Tenant Security and RBAC: Storing alerts in space-specific indices allows Elasticsearch Role-Based Access Control (RBAC) and Document-Level Security (DLS) to strictly enforce tenant isolation. Analysts assigned to a retail space cannot query or view alerts belonging to a corporate finance space.
• Index Lifecycle Management (ILM): Alert indices are governed by automated ILM policies. By default, alert indices use the .alerts-ilm-policy policy, which rolls over at 50 GB or 30 days and has no delete phase. Alerts are therefore kept until you change the policy or delete them.
• ECS Compliance: Every alert document written to .alerts-security.alerts-* embeds the complete original telemetry fields normalized to the Elastic Common Schema (ECS), alongside dedicated alert metadata fields under the kibana.alert.* and signal.* namespaces.

---

Prebuilt Elastic Security Detection Rules

Elastic Security provides an extensive library of more than 1,000 prebuilt detection rules developed, tested, and maintained by the Elastic Security Intelligence & Analytics team and the open-source security community. These rules encode detection logic for active adversary campaigns, malware strains, living-off-the-land techniques (LOLBins), ransomware behaviors, and cloud infrastructure misconfigurations.

1. Installing and Updating Prebuilt Rules

Prebuilt rules are not installed or enabled automatically. Go to Rules → Detection rules (SIEM) and click Add Elastic rules. The badge shows how many are available. Then choose Install all, install a single rule, or install a selection, filtering by tags such as OS: Windows. Installed rules stay disabled until you enable them individually or with Bulk actions → Enable. The Detection Engine API can also install them (PUT /api/detection_engine/rules/prepackaged).

Elastic ships new and updated prebuilt rules in the security_detection_engine Fleet package. When updated versions exist for rules you have installed, a Rule Updates tab appears on the Rules page.

2. MITRE ATT&CK Framework Mapping

Every prebuilt rule is mapped directly to the MITRE ATT&CK matrix, providing immediate operational context regarding where an observed activity falls within the cyber kill chain:

• Tactics: The adversary's high-level tactical objective (e.g., Initial Access, Execution, Persistence, Privilege Escalation, Defense Evasion, Credential Access, Discovery, Lateral Movement, Collection, Command and Control, Exfiltration, Impact).
• Techniques: The specific technical mechanism used to achieve the tactical goal (e.g., T1059 - Command and Scripting Interpreter).
• Sub-Techniques: Granular implementation variants (e.g., T1059.001 - PowerShell, T1059.003 - Windows Command Shell).

In the Elastic Security UI, these mappings populate the interactive MITRE ATT&CK Coverage Matrix, allowing SecOps leads to visualize detection coverage across the entire threat landscape and identify blind spots in enterprise telemetry.

3. Metadata Categorization: Tags, Severity, and Risk Scoring

Prebuilt rules include standardized metadata attributes that govern prioritization and alert triage:

• Rule Tags: Structured labels identifying target operating systems (OS: Windows, OS: Linux, OS: macOS), threat categories (Threat: Cobalt Strike, Threat: Ransomware), data sources (Data Source: Elastic Defend, Data Source: System Logs, Data Source: AWS CloudTrail), and compliance frameworks.
• Severity Levels: Categorical indicator of threat potential:
  • Low: Informational or minor policy violations; low likelihood of immediate harm.
  • Medium: Suspicious activity warranting investigation; common administrative tools used in unusual contexts.
  • High: Strong indicator of malicious behavior; confirmed exploitation tools, privilege escalation, or unauthorized credential access.
  • Critical: Severe and urgent threat; active ransomware deployment, destructive wiping, or confirmed remote code execution on domain infrastructure.
• Risk Score (0 to 100): A value set on the rule itself, which can be overridden from a source-event field. Elastic's guideline ranges are 0–21 low, 22–47 medium, 48–73 high and 74–100 critical, and the rule form suggests 21, 47, 73 or 99 when you pick a severity. The alert's kibana.alert.risk_score later feeds entity risk scoring.

4. License Tier Dependencies

Elastic Security's documentation lists only a few detection features outside the free Basic tier. Basic includes every rule type except machine learning: custom query, threshold, event correlation (EQL), indicator match, new terms and ES|QL. It also includes exceptions and value lists.

• Platinum or higher: machine learning jobs and rules, alert suppression, alert notifications to external systems through rule actions, Cases connectors to third-party ticketing systems, entity risk scoring and host isolation.
• Enterprise: Elastic AI Assistant, Attack Discovery and Session View.

---

Prebuilt Rule Updates, Duplication and Exceptions

Prebuilt rules follow different editing rules from custom rules, and the difference is a frequent exam trap.

What You Can and Cannot Change

In Elastic Security 8.15 you can't modify most settings of an Elastic prebuilt rule. You can only edit its rule actions and add exceptions, and you can enable or disable it. To change anything else, such as the query, index patterns, schedule or severity, you duplicate the rule (Bulk actions → Duplicate, optionally including its exceptions) and edit the copy. The duplicate is a separate custom rule and no longer receives Elastic's updates for the original.

[ Elastic publishes new rule versions (security_detection_engine package) ]
                 |
                 v
[ Rules page shows a "Rule Updates" tab for installed rules ]
                 |
                 v
[ Open a rule: Updates tab = field-by-field diff; JSON view = full diff ]
   (Current rule vs Elastic update; deletions red, additions green)
                 |
                 v
[ Update rule | Update selected | Update all ]
   Exceptions and actions attached to the rule remain in place

Reviewing and Installing Updates

Open Rules → Detection rules (SIEM) → Rule Updates. Selecting a rule name opens a flyout. Its Updates tab compares the Current rule with the Elastic update field by field, and the JSON view tab compares the whole rule, with deleted characters in red and added characters in green. You then update one rule, a selection, or all of them. UI updates are supported for the current Elastic Security version and the three previous minor releases. After that you must upgrade the stack to keep receiving them.

Operational Best Practice: Exceptions Before Duplication

To suppress organization-specific false positives, add a rule exception to the prebuilt rule, for example excluding an approved vulnerability scanner or service account. Exceptions stay attached when Elastic updates the rule, so you keep both your tuning and Elastic's improved logic. Duplicate a rule only when the detection logic itself must change, and accept that you now maintain that copy yourself.

---

Detection Engine Execution Health and Rule Monitoring

Maintaining a reliable detection pipeline requires continuous monitoring of rule execution health. The Rules page's Monitoring tab (with its Gap column) and the prebuilt Detection rule monitoring dashboard show execution status, durations, schedule delays and gaps, so you can find operational failures before they cause missed attacks.

Common Rule Execution Failure Modes

1. Execution Gaps: An execution gap occurs when a rule misses one or more scheduled execution cycles. This typically results from Kibana Task Manager queue saturation, where too many rules run concurrently, or worker threads are exhausted.
2. Search Timeouts: If a rule targets broad index wildcards (e.g., * instead of logs-*) or contains an unoptimized query executing against billions of unindexed text fields, the Elasticsearch search query may run too long and fail, leaving the run in an error or warning state.
3. Circuit Breaker & Memory Exceptions: High-cardinality aggregations inside threshold rules can exceed Elasticsearch field data circuit breakers, returning HTTP 500 errors to the Detection Engine.
4. Missing Field / Mapping Errors: If a rule relies on an ECS field that does not exist in the target index mapping or has an incompatible field type (e.g., searching an unindexed keyword field as text), query compilation fails.

Analyst teams should monitor the .kibana-event-log-* and Kibana server logs to alert on any detection rules entering persistent error or execution-gap states.

---

Prebuilt Rule Metadata Attributes Reference

The following reference table summarizes the primary metadata attributes of Elastic prebuilt detection rules, their formats, and their operational implications in a SOC: