9.2 Creating and Structuring Datasets & Hierarchies

9.2 Creating and Structuring Datasets & Hierarchies

Quick Summary: Building a data model involves creating a structured hierarchy of datasets starting with one or more root datasets. Splunk provides three root dataset types: Event, Search, and Transaction. Root Event datasets form the foundation of most operational models and establish baseline search constraints. Child datasets branch from parent datasets, automatically inheriting all parent constraints and attributes while applying additional constraints to narrow data scope. Attributes (fields) are defined using four methods: Auto-extracted, Eval expressions, Lookups, and Regular Expressions.

---

1. The Three Root Dataset Types in Detail

When creating a dataset at the top level of a data model, Splunk prompts the administrator or Power User to select one of three Root Dataset Types. Choosing the appropriate root dataset type is a critical architectural decision that dictates how data is filtered, which SPL commands are permitted, and whether the dataset can be accelerated.

+------------------------------------------------------------------------------------------------+
|                                    THREE ROOT DATASET TYPES                                    |
|                                                                                                |
|  1. ROOT EVENT DATASET                                                                         |
|     [ Constraint Search: index=web sourcetype=access_combined status=* ]                       |
|     • Direct filter over raw indexed events.                                                   |
|     • NO transforming commands allowed.                                                        |
|     • FULLY ACCELERABLE via TSIDX summary acceleration.                                        |
|     • Ideal for: High-volume raw log streams (Web, Firewall, Windows, Authentication).         |
|                                                                                                |
|  2. ROOT SEARCH DATASET                                                                        |
|     [ SPL Search Pipeline: index=sales | stats sum(amount) AS revenue BY store_id ]             |
|     • Full arbitrary SPL search query permitted (including stats, eval, lookup, join).         |
|     • Transforming commands ARE allowed.                                                       |
|     • CANNOT BE ACCELERATED with Data Model Acceleration.                                      |
|     • Ideal for: Pre-aggregated reporting data and complex multi-command pipelines.            |
|                                                                                                |
|  3. ROOT TRANSACTION DATASET                                                                   |
|     [ Transaction Definition: Group by session_id, maxspan=30m, maxpause=5m ]                  |
|     • Groups raw events into multi-event transactions based on common fields & time spans.     |
|     • Computes duration and eventcount automatically.                                          |
|     • CANNOT BE ACCELERATED with Data Model Acceleration.                                      |
|     • Ideal for: Multi-step checkout flows, user login-to-logout session tracking.             |
+------------------------------------------------------------------------------------------------+

Comprehensive Comparison of Root Dataset Types

[!IMPORTANT] Core Certification Rule: Only Root Event datasets (and their child datasets) are eligible for Data Model Acceleration. If an exam question asks which dataset type can be accelerated to build high-performance .tsidx files, the answer is always Event datasets.

---

2. Root Event Dataset Constraints & Best Practices

A Root Event Dataset defines the foundational boundary for an entire branch of a data model. The boundary is established using a Constraint Search String.

Writing Effective Constraints

Constraints use standard Splunk search filter syntax (the initial boolean search segment before any pipe | symbol). Constraints should follow strict scoping principles:

1. Always Specify an Index: Explicitly target specific indexes (e.g., index=web or index=firewall) to prevent Splunk from searching the default index or scanning every accessible index.
2. Specify Sourcetypes or Sources: Use sourcetype=access_combined or sourcetype=cisco:asa to restrict the parser to known event formats.
3. Include Baseline Field Wildcards: If downstream child datasets or attributes require specific fields, ensure those fields exist by including them in the constraint (e.g., status=* or action=*).

+------------------------------------------------------------------------------------------------+
|                                 VALID VS. INVALID ROOT CONSTRAINTS                             |
|                                                                                                |
|  VALID ROOT EVENT CONSTRAINTS:                                                                 |
|  ✔ index=web sourcetype=access_combined                                                        |
|  ✔ index=security (sourcetype=WinEventLog:Security OR sourcetype=linux_secure)                 |
|  ✔ index=firewall action=* (vendor_action=allow OR vendor_action=deny)                         |
|  ✔ index=paloalto type=traffic transport=tcp                                                   |
|                                                                                                |
|  INVALID ROOT EVENT CONSTRAINTS:                                                               |
|  ✘ index=web | eval mb = bytes/1024/1024       (Pipe commands NOT permitted in constraints)    |
|  ✘ index=web | stats count BY clientip         (Transforming commands NOT permitted)           |
|  ✘ index=* *                                   (Too broad; degrades search performance)       |
+------------------------------------------------------------------------------------------------+

---

3. Building Hierarchical Child Datasets & Constraint Cascading

Child datasets allow data model architects to create specialized sub-categories of data branching from parent datasets. This architectural pattern mirrors object-oriented inheritance and relational sub-typing.

+------------------------------------------------------------------------------------------------+
|                               DATASET INHERITANCE HIERARCHY TREE                               |
|                                                                                                |
|  ROOT EVENT DATASET: Web_Access                                                                |
|  Constraint: index=web sourcetype=access_*                                                     |
|  Attributes: _time, host, source, sourcetype, clientip, method, uri_path, status, bytes        |
|  │                                                                                             |
|  ├── CHILD DATASET 1: Successful_Requests                                                      |
|  │   Constraint: status >= 200 AND status < 400                                                |
|  │   Inherits: All 9 Root Attributes                                                           |
|  │   Added Attribute: download_speed_kbps                                                      |
|  │   │                                                                                         |
|  │   └── GRANDCHILD DATASET 1.1: Media_Downloads                                              |
|  │       Constraint: uri_path="*.mp4" OR uri_path="*.zip"                                      |
|  │       Inherits: All 9 Root Attributes + download_speed_kbps (10 total)                       |
|  │       Added Attribute: media_type                                                           |
|  │                                                                                             |
|  └── CHILD DATASET 2: Web_Errors                                                               |
|      Constraint: status >= 400                                                                 |
|      Inherits: All 9 Root Attributes                                                           |
|      Added Attribute: error_severity                                                           |
|      │                                                                                         |
|      ├── GRANDCHILD DATASET 2.1: Client_Errors_4xx                                             |
|      │   Constraint: status >= 400 AND status < 500                                            |
|      │   Inherits: All 9 Root Attributes + error_severity (10 total)                           |
|      │                                                                                         |
|      └── GRANDCHILD DATASET 2.2: Server_Errors_5xx                                             |
|          Constraint: status >= 500 AND status < 600                                            |
|          Inherits: All 9 Root Attributes + error_severity (10 total)                           |
|          Added Attribute: server_cluster_id                                                    |
+------------------------------------------------------------------------------------------------+

The Mechanics of Cascading Constraints

When a user or Pivot report queries a child dataset, Splunk evaluates the effective query by combining all parent and ancestor constraints using Boolean AND logic:

$\text{Effective Child Constraint} = \text{Root Constraint} \land \text{Child 1 Constraint} \land \text{Child 2 Constraint} \dots$

For example, querying the grandchild dataset Server_Errors_5xx executes the following combined constraint:

(index=web sourcetype=access_*) AND (status >= 400) AND (status >= 500 AND status < 600)

Because of this strict hierarchical filtering, child datasets can only represent a subset of their parent's events. A child dataset cannot contain events that were filtered out by its parent.

---

4. Attribute Types & Field Definition Methods

Attributes are the named fields within a dataset that are exposed for reporting, filtering, and metric aggregation. Splunk provides four distinct methods for adding attributes to a dataset:

+------------------------------------------------------------------------------------------------+
|                                FOUR ATTRIBUTE EXTRACTION METHODS                               |
|                                                                                                |
|  1. AUTO-EXTRACTED ATTRIBUTES                                                                  |
|     Maps standard fields already extracted at search-time via props.conf, index-time headers,   |
|     or default Splunk metadata fields (_time, host, source, sourcetype, punct).               |
|                                                                                                |
|  2. EVAL EXPRESSION ATTRIBUTES                                                                 |
|     Calculates dynamic attributes at search time using standard eval functions and logic.      |
|     Example: eval response_time_sec = round(response_time_ms / 1000, 2)                         |
|     Example: eval status_type = if(status < 400, "Success", "Failure")                         |
|                                                                                                |
|  3. LOOKUP ATTRIBUTES                                                                          |
|     Enriches dataset events by matching key fields against static CSVs or KV Store lookups.    |
|     Example: Match clientip against geo_ip.csv to output country, city, and latitude.          |
|                                                                                                |
|  4. REGULAR EXPRESSION (REGEX) ATTRIBUTES                                                      |
|     Extracts new fields directly from _raw on-the-fly using named capture groups: (?P<field>...)|
|     Example: Regex session_id=(?P<session_token>[A-Fa-f0-9]{32}) on _raw                      |
+------------------------------------------------------------------------------------------------+

Comprehensive Attribute Configuration Matrix

Attribute Data Types

Each defined attribute must be assigned one of five standard data types in Splunk:

1. String: Alphanumeric text values (e.g., user, uri_path, method).
2. Number: Numeric integers or floating-point decimals (e.g., bytes, response_time, status).
3. Boolean: True/False binary values.
4. IPv4: Standard dotted-quad IP addresses (e.g., 192.168.1.50), enabling subnet and CIDR comparisons.
5. Timestamp: Epoch or formatted date-time values used for time-series aggregation.

Attribute Flags: Optional, Required, and Hidden

When configuring attributes, architects can apply specific behavioral flags:

• Optional (Default): Events are included in the dataset whether the attribute is present or null.
• Required: Events must contain a non-null value for this attribute to be included in the dataset. If the field is missing, the event is filtered out.
• Hidden: The attribute is evaluated and available for internal calculations, constraints, or lookups, but is hidden from the Pivot interface so non-technical users are not confused by internal technical fields.

---

5. Attribute Inheritance & Override Rules

Data models enforce strict inheritance mechanics across their dataset trees:

+------------------------------------------------------------------------------------------------+
|                                ATTRIBUTE INHERITANCE & OVERRIDES                               |
|                                                                                                |
|  Root Dataset: Network_Base                                                                    |
|  [ Attributes: src_ip, dest_ip, port, transport, action="unknown" ]                            |
|        │                                                                                       |
|        ▼ Inherits all 5 parent attributes                                                      |
|  Child Dataset: Firewall_Traffic                                                               |
|  [ Inherited: src_ip, dest_ip, port, transport ]                                               |
|  [ OVERRIDDEN ATTRIBUTE: action = if(vendor_code==1, "allow", "deny") ]                        |
|  [ ADDED ATTRIBUTE: rule_id (Auto-extracted) ]                                                 |
|        │                                                                                       |
|        ▼ Inherits all parent + child attributes (6 total)                                      |
|  Grandchild Dataset: Threat_Drops                                                              |
|  [ Inherited: src_ip, dest_ip, port, transport, action, rule_id ]                              |
|  [ ADDED ATTRIBUTE: threat_signature (Lookup from threat_intel.csv) ]                          |
+------------------------------------------------------------------------------------------------+

1. Automatic Downward Inheritance: When an attribute is added to a parent dataset, every child, grandchild, and descendant dataset automatically inherits that attribute.
2. Child-Specific Attribute Additions: Attributes added directly to a child dataset exist exclusively within that child branch. Parent datasets and sibling child branches do not have access to child-specific attributes.
3. Attribute Overriding / Shadowing: If a child dataset defines an attribute with the exact same name as an inherited parent attribute, the child's definition overrides the parent's definition for that branch. This is commonly used to replace a generic root eval calculation with a specialized child formula.

---

6. Step-by-Step Dataset Hierarchy Walkthrough in Splunk Web

Scenario: Building an E-Commerce Web Intelligence Data Model

Step 1: Create the Data Model Container

1. Navigate to Settings > Data models.
2. Click New Data Model.
3. Enter Title: ECommerce_Web_Model (ID: ECommerce_Web_Model), App: Search & Reporting.
4. Click Save.

Step 2: Add the Root Event Dataset

1. Click Add Dataset > Root Event.
2. Dataset Name: Web_Requests (Dataset ID: Web_Requests).
3. Enter Constraints: index=ecommerce sourcetype=access_combined status=*.
4. Click Save.

Step 3: Add Attributes to the Root Dataset

1. Click Add Attribute > Auto-Extracted.
2. Select: clientip, method, status, bytes, uri_path.
3. Click Add Attribute > Eval Expression.
   • Field Name: response_category
   • Expression: case(status<300, "Success", status<400, "Redirection", status<500, "Client Error", status>=500, "Server Error")
   • Type: String.
4. Click Add Attribute > Lookup.
   • Lookup Table: geo_ip_lookup
   • Input Field: clientip = ip
   • Output Fields: country, city.

Step 4: Add Child Datasets with Cascading Constraints

1. Select the Web_Requests dataset.
2. Click Add Dataset > Child.
3. Dataset Name: Checkout_Errors.
4. Enter Child Constraints: uri_path="/checkout/*" status>=400.
5. Notice that Checkout_Errors automatically displays all parent attributes (clientip, method, status, response_category, country, city).
6. Click Add Attribute > Regular Expression to extract payment gateway error codes from _raw:
   • Regular Expression: error_code=(?P<gw_error_code>[A-Z0-9_]+)
   • Type: String.

Step 5: Validate Dataset in Pivot

1. Click the Pivot button in the upper right corner.
2. Select Checkout_Errors.
3. Split rows by country and gw_error_code, column values by Count of Checkout_Errors.
4. Verify that results render accurately in seconds.