3.1 ECS Structure, Core Principles & Essential Field Sets

The Ingestion Normalization Challenge in SecOps

A modern Security Operations Center (SOC) ingests telemetry from dozens of disparate enterprise technologies: Microsoft Windows Event Logs, Linux auditd daemons, network perimeter firewalls (Palo Alto, Fortinet, Cisco ASA), intrusion detection sensors (Zeek, Suricata), cloud audit services (AWS CloudTrail, Microsoft Entra ID, Google Cloud Audit Logs), and endpoint detection agents.

Without a universal schema, each vendor represents identical security concepts under conflicting field names:

• Source IP Address: Recorded variously as src, src_ip, c_ip, SourceIp, client_address, or remote_addr.
• User Identity: Stored as user, user_id, suser, TargetUserName, AccountName, or caller_identity.
• Process Executable: Logged as Image, ProcessName, comm, proc_name, or exe.

This syntactic fragmentation paralyzes detection engineering. If an analyst builds a detection rule for brute-force credential stuffing or process injection, they would need to craft, test, and maintain separate query variants for every log source and vendor format. Adding a new identity provider or endpoint sensor would require rewriting hundreds of detection rules.

The Elastic Common Schema (ECS) resolves this challenge. ECS is an open-source, extensible specification that standardizes field names, data types, and semantic categorization across all events stored in Elasticsearch. By transforming telemetry into ECS-compliant documents upon ingestion, the SOC establishes:

1. Vendor-Agnostic Detection Engineering: Prebuilt Elastic Security rules and custom detections evaluate standardized fields (e.g., event.category: "authentication" AND event.outcome: "failure"). The rule fires whether the failure occurred on an AWS IAM console, a Cisco AnyConnect VPN, a Linux bastion server, or an Active Directory Domain Controller.
2. Correlated Threat Hunting: Incident responders can trace an adversary across network, host, and cloud boundaries using unified queries in Kibana Discover, Timelines, and Cases.
3. Reusable Visualizations and Analytics: Kibana Lens dashboards, Machine Learning anomaly detection baselines, and ES|QL hunting queries execute across all telemetry streams without vendor-specific transformations.

---

ECS Design Principles: Dot-Notation and Nesting

Elasticsearch stores documents natively as JSON objects. ECS leverages this architecture through a standardized dot-notation naming hierarchy. For instance, host.name, host.ip, host.os.type, and host.geo.country_name.

In Elasticsearch index mappings, a field defined with dots is structurally represented as a nested hierarchy:

{
  "host": {
    "name": "srv-dc01.corp.internal",
    "ip": ["10.100.4.10", "192.168.1.25"],
    "os": {
      "type": "windows",
      "family": "windows",
      "name": "Windows Server 2022"
    }
  }
}

Core Schema Conventions

• Strict Lowercase and Underscores: Field names must be entirely lowercase. Words within a single field token are separated by underscores (e.g., process.command_line, dns.question.registered_domain), never camelCase or hyphens. This eliminates case-sensitivity mapping collisions in Elasticsearch.
• Singular vs. Plural: Field set names and individual fields are singular (e.g., user, host, process, file), except when a field inherently stores an array of plural items, such as tags, labels, dns.answers, or related.ip.
• No Redundant Prefixes: Field names within a set do not repeat the field set name. For example, use host.name and host.id, rather than host.host_name or host.host_id.
• Root vs. Field Set Boundaries: Root-level fields are limited to foundational metadata, primarily @timestamp, message, tags, and labels. All domain-specific attributes belong to dedicated ECS field sets.

---

Essential SecOps Field Sets

Security analysts must master the primary ECS field sets that capture entities, actions, and network layers during an attack.

1. Host Telemetry (host.*)

Describes the physical, virtual, or cloud computing system where the event occurred or was observed:

• host.name: The fully qualified hostname or computer name (e.g., "srv-dc01.corp.internal").
• host.id: A persistent unique host identifier, such as the system machine-guid or cloud instance ID.
• host.ip: An array of all IP addresses configured on the host's network interfaces.
• host.mac: Array of hardware MAC addresses.
• host.os.type: Standardized operating system family: "windows", "linux", "macos", "ios", "android".
• host.os.name: Full distribution or release name (e.g., "Ubuntu Server", "Windows 11 Pro").
• host.architecture: CPU architecture (e.g., "x86_64", "arm64").

2. Identity and Users (user.* and user.target.*)

Represents user identities and security principals involved in the activity:

• user.name: The username executing the action or logging in (e.g., "jdoe", "svc_sql").
• user.domain: The Active Directory domain, Kerberos realm, or local workgroup (e.g., "CORP").
• user.id: The unique system or directory identifier, such as a Windows SID ("S-1-5-21-...") or Linux UID ("1001").
• user.email: Primary email address associated with the identity.
• user.roles: Array of role names, security groups, or entitlements granted to the user.
• user.target.*: Populated when an operation acts upon a separate account. When an administrator resets a password or creates an account, user.* represents the administrator, while user.target.* represents the affected user.

3. Network and Transport (network.*, source.*, destination.*)

Captures network communications, segregating protocol metadata from connection endpoints:

• network.transport: Layer 4 transport protocol ("tcp", "udp", "icmp").
• network.protocol: Application-layer protocol ("dns", "http", "tls", "ssh", "smb").
• network.direction: Traffic direction relative to the observer ("ingress", "egress", "inbound", "outbound", "internal", "external").
• network.community_id: Standardized Community ID hash string calculated from the IP 5-tuple, enabling direct correlation across Zeek, Suricata, and firewall logs.
• network.bytes / network.packets: Aggregate byte and packet counts transferred across the connection.
• source.ip and source.port: Originating IP address and source port.
• destination.ip and destination.port: Receiving IP address and service port.
• source.geo.* / destination.geo.*: Geolocation attributes enriched from IP lookups (country_name, city_name, location).
• source.as.* / destination.as.*: Autonomous System details (number, organization.name).

4. Process Execution (process.*)

Critical for detecting malware execution, living-off-the-land binaries (LOLBins), and script interpreter abuse:

• process.name: The executable filename without path (e.g., "powershell.exe", "cmd.exe", "curl").
• process.executable: The full absolute filesystem path (e.g., "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe").
• process.command_line: The full command string, including arguments, flags, and encoded scripts.
• process.pid: The operating system Process Identifier.
• process.parent.pid: The PID of the spawning parent process.
• process.parent.name / process.parent.executable: Name and path of the parent process (vital for identifying anomalous parent-child chains such as excel.exe spawning powershell.exe).
• process.hash.sha256 / process.hash.md5: Cryptographic checksums of the executed binary.
• process.entity_id: A unique process identifier supplied by the data source (Elastic Defend generates one; Sysmon supplies a process GUID) that stays unique even when the operating system reuses PIDs.

5. File System Activity (file.*)

• file.path: Full path to the file on disk (e.g., "/etc/shadow", "C:\\Windows\\System32\\drivers\\etc\\hosts").
• file.name: Filename component (e.g., "payload.exe").
• file.extension: File extension without leading dot (e.g., "exe", "ps1", "dll").
• file.size: Size of the file in bytes.
• file.hash.sha256: Cryptographic SHA-256 hash of the target file.
• file.owner: User account owning the filesystem object.

6. Domain Name System (dns.*)

• dns.question.name: The queried fully qualified domain name (FQDN), e.g., "c2.attacker-infrastructure.com".
• dns.question.type: Resource record type ("A", "AAAA", "TXT", "MX", "CNAME").
• dns.question.registered_domain: The registrable domain or eTLD+1 (e.g., "attacker-infrastructure.com").
• dns.answers.data: Array of returned DNS records (IP addresses or text strings).
• dns.response_code: Standard DNS response code ("NOERROR", "NXDOMAIN", "SERVFAIL").

---

The ECS Event Categorization Taxonomy

The most critical component of ECS for detection engineering is the four-tier categorization framework:

1. event.kind: Architectural classification of the event. Permitted values are strictly controlled. In current ECS 8.x releases they are "alert", "asset", "enrichment", "event", "metric", "state", "pipeline_error", and "signal". Standard audit and system logs use "event", whereas detection alerts or IDS triggers use "alert".
2. event.category: High-level domain of activity. Common values include: authentication, configuration, database, driver, file, host, iam, intrusion_detection, malware, network, package, process, registry, session, threat, vulnerability, and web. A document can contain multiple categories in an array.
3. event.type: Specific action, state change, or sub-classification. Common values include: access, admin, allowed, change, connection, creation, deletion, denied, end, error, group, info, installation, protocol, start, user.
4. event.outcome: Result of the attempted action. Must be exactly one of: "success", "failure", or "unknown".

Event Categorization Matrix for Common SecOps Scenarios

Note the firewall row carefully. ECS judges event.outcome from the perspective of the event producer. Its own reference example classifies a firewall that successfully blocks a connection as event.type: ["connection", "denied"] with event.outcome: "success", because the firewall did what it set out to do. To find blocked traffic, search event.category: network and event.type: denied rather than event.outcome: failure. ECS also says outcome is generally left empty for event.type: info events.

By leveraging this standardized categorization matrix, security teams can construct vendor-independent correlation rules. An analyst investigating credential attacks can write a single detection rule targeting event.category: "authentication" AND event.outcome: "failure" that immediately alerts on suspicious activity across Linux, Windows, Okta, and cloud identity providers.