0.2 Splunk Knowledge Objects & Search Pipeline Architecture

0.2 Splunk Knowledge Objects & Search Pipeline Architecture

Power User Role Boundaries & Knowledge Object Ecosystem

In enterprise Splunk deployments, the Power User role occupies a pivotal tier within the role-based access control (RBAC) hierarchy. While standard Users can execute ad-hoc searches, build personal dashboards, and view shared reports, they lack permissions to share knowledge objects globally or build acceleration data models. Conversely, Administrators govern host-level infrastructure, manage index clustering topologies, configure forwarders, and modify underlying operating system files.

+-----------------------------------------------------------------------------------+
|                        SPLUNK RBAC PERMISSION HIERARCHY                           |
+-----------------------------------------------------------------------------------+
| ADMIN ROLE:                                                                       |
| • Host-level config files (`/etc/system/local/`, CLI access, server restart)       |
| • Index & bucket lifecycle management, cluster topology, licensing, forwarders   |
| • User & role management, authentication (LDAP/SAML), system-wide capabilities     |
+-----------------------------------------------------------------------------------+
                                         ▲
                                         │ (Elevated Knowledge Permissions)
+-----------------------------------------------------------------------------------+
| POWER USER ROLE (Certification Target):                                           |
| • Create, edit, and share Knowledge Objects (Field Extractions, Aliases, Eval)    |
| • Manage Knowledge Object permissions: Private ➔ App-Specific ➔ Global Scope      |
| • Build and structure Data Models & Datasets; configure Data Model Acceleration   |
| • Create and maintain Event Types, Tags, Search Macros, and Workflow Actions       |
| • Implement Common Information Model (CIM) field normalization                    |
+-----------------------------------------------------------------------------------+
                                         ▲
                                         │ (Basic Search & Reporting)
+-----------------------------------------------------------------------------------+
| USER ROLE:                                                                        |
| • Run ad-hoc SPL queries against authorized indexes                               |
| • Create private searches, reports, alerts, and dashboard panels                  |
| • Cannot share objects globally or manage data model definitions                  |
+-----------------------------------------------------------------------------------+

---

The Search-Time Field Discovery Execution Pipeline

When a user executes an SPL search, the Splunk Search Head processes events through a deterministic, multi-stage pipeline. Each stage generates fields or metadata that become available to subsequent downstream stages. Understanding this precise sequence is one of the most heavily tested concepts on the exam.

Detailed Pipeline Stages

1. Stage 0: Default and Indexed Fields
   • Splunk parses raw events and retrieves fields stored directly in index bucket TSIDX files during index time: _time, _raw, host, source, sourcetype, index, and punct.
2. Stage 1: Field Extractions (Inline & Transforms)
   • Search-time extractions execute using regular expressions defined in props.conf (EXTRACT-<class>) and referenced transforms (REPORT-<class> in transforms.conf). Delimiter-based extractions also execute at this stage.
3. Stage 2: Field Aliases
   • Splunk maps existing extracted or default field names to alternative alias names (FIELDALIAS-<class> in props.conf). Aliasing does not replace original field names; both original and aliased field names coexist simultaneously.
4. Stage 3: Calculated Fields
   • Splunk executes eval expressions defined in props.conf (EVAL-<fieldname> = <eval-expression>). Calculated fields can evaluate default fields, extracted fields, and field aliases.
5. Stage 4: Lookups
   • Splunk matches event fields against external CSV files, KV Store collections, or external scripts (LOOKUP-<class> in props.conf matching definitions in transforms.conf). Lookups can use input fields produced by Stages 0 through 3 (including calculated fields) and output new enrichment fields.
6. Stage 5: Event Types
   • Splunk evaluates search strings defined in eventtypes.conf against enriched events. Event type search criteria can match default fields, extracted fields, field aliases, calculated fields, and lookup output fields.
7. Stage 6: Tags
   • Splunk attaches labels defined in tags.conf to specific field-value pairs or event types. Because tags depend on both event types and all accumulated field values, they are assigned at the very end of the search-time discovery pipeline.

---

Knowledge Object Dependency Matrix

The strict sequential nature of the pipeline dictates which knowledge objects can reference or depend on other knowledge objects:

Key Exam Architectural Takeaways

• Can a Calculated Field reference a Field Alias? YES. Field Aliases (Stage 2) execute before Calculated Fields (Stage 3).
• Can a Lookup match on a Calculated Field? YES. Calculated Fields (Stage 3) execute before Lookups (Stage 4).
• Can an Event Type search query use a Lookup output field? YES. Lookups (Stage 4) populate fields before Event Types (Stage 5) evaluate.
• Can a Field Alias rename a Calculated Field? NO. Field Aliases (Stage 2) run before Calculated Fields (Stage 3). If you need an alias for a calculated result, perform the rename within the calculated eval expression itself or via SPL.
• Can an Event Type search contain a subsearch, pipe (|), or another event type? NO. Event type definitions in Splunk cannot include pipes or subsearches.

---

Knowledge Object Scoping & Permission Architecture

Every knowledge object created in Splunk exists within a defined sharing and permission boundary:

Permission Governance Principles

1. Read vs. Write Permissions: Granting Read permission allows users to execute searches utilizing the knowledge object. Granting Write permission allows users to edit, clone, change sharing scopes, or delete the knowledge object.
2. Promoting Scopes: A Power User can promote an object from Private to This App Only, or from This App Only to Global via the Splunk Web UI under Settings > Knowledge Objects > Permissions.
3. Sharing to Global: When an object is shared globally, its underlying configuration remains stored inside the originating application's local/ directory, while an entry is added to metadata/local.meta marking export = system.

---

Configuration File Precedence & Hierarchy

Splunk configures knowledge objects through .conf files. When an identical stanza or attribute exists across multiple directories, Splunk resolves conflicts using strict precedence rules.

Precedence Resolution Hierarchy (Search Time / User Context)

When a user runs a search, Splunk evaluates configuration files in the following top-down order (highest priority to lowest priority):

1. Current User Directory:             $SPLUNK_HOME/etc/users/<username>/<app_name>/
2. App Local Directory:                $SPLUNK_HOME/etc/apps/<app_name>/local/
3. App Default Directory:              $SPLUNK_HOME/etc/apps/<app_name>/default/
4. System Local Directory:             $SPLUNK_HOME/etc/system/local/
5. System Default Directory:           $SPLUNK_HOME/etc/system/default/

Precedence Principles

• local always overrides default: Any configuration written in a local/ directory overrides identical stanzas and settings in the corresponding default/ directory.
• Never Edit default/ Directories: Default directories contain vendor-shipped configurations that are overwritten during Splunk upgrades. All customizations must be saved in local/ or user directories.
• App Tie-Breaking Depends on Context (a classic exam trap): Splunk uses two opposite orderings. In the app/user context — which is how props.conf is evaluated at search time — the currently running app always wins, regardless of name, and the remaining apps are ranked in reverse-lexicographical order (app B beats app A); other apps are only consulted for settings exported globally. In the global context — how props.conf is evaluated at index time, and how files like inputs.conf are always evaluated — plain lexicographical order applies (app A beats app B, and any app starting with an uppercase letter beats any app starting with a lowercase letter).

---

Search Pipeline Mechanics: Streaming vs. Transforming Commands

Understanding how Splunk executes SPL across distributed Search Heads and Indexers is essential for search optimization and Power User performance tuning.

+-----------------------------------------------------------------------------------+
|                        SPL COMMAND CLASSIFICATION MATRIX                          |
+-----------------------------------------------------------------------------------+
| DISTRIBUTABLE STREAMING COMMANDS (Runs on Indexers in parallel)                   |
| Commands: `search`, `where`, `eval`, `fields`, `rename`, `replace`, `rex`         |
| Behavior: Operates record-by-record on raw event streams at the Indexer level.    |
| Benefit:  Filters out unwanted data before sending results across the network.    |
+-----------------------------------------------------------------------------------+
                                         │
                                         ▼ (Filtered intermediate data stream)
+-----------------------------------------------------------------------------------+
| CENTRALIZED STREAMING COMMANDS (Runs on Search Head record-by-record)             |
| Commands: `head`, `streamstats`, `autoregress`, `transaction` (in most cases)     |
| Behavior: Applies a transformation to each event but only on the Search Head.     |
+-----------------------------------------------------------------------------------+
                                         │
                                         ▼ (Ordered stream)
+-----------------------------------------------------------------------------------+
| TRANSFORMING COMMANDS (Map on Indexers ➔ Reduce on Search Head)                  |
| Commands: `stats`, `chart`, `timechart`, `top`, `rare`, `contingency`             |
| Behavior: Converts event streams into aggregated statistical data tables.         |
| Architecture: Pre-aggregates on Indexers (Map), finalizes on Search Head (Reduce).|
+-----------------------------------------------------------------------------------+
                                         │
                                         ▼ (Statistical Table)
+-----------------------------------------------------------------------------------+
| DATASET PROCESSING COMMANDS (needs the entire dataset before it can run)          |
| Commands: `sort`, `eventstats`, `tail`, `reverse`, `map`, `fillnull` (no fields)  |
| Behavior: Requires the complete result set in memory before producing output.     |
| NOTE: `dedup` is DISTRIBUTABLE STREAMING by default -- it only becomes a dataset  |
| processing command with `sortby` or `keepevents=true`. `transaction` is a         |
| CENTRALIZED STREAMING command in most cases, dataset processing with some args.   |
+-----------------------------------------------------------------------------------+

Search Optimization Best Practices for Power Users

1. Filter Early: Always place specific index=, sourcetype=, and keyword filters at the beginning of the search string.
2. Pushdown to Indexers: Utilize distributable streaming commands (eval, where, fields) before transforming commands so Indexers handle compute in parallel.
3. Prune Fields Early: Use | fields + <needed_fields> immediately after initial filtering to minimize memory usage and network transfer between Indexers and Search Heads.
4. Prefer stats over transaction: Whenever possible, use stats (a transforming command that distributes map work to Indexers) rather than transaction, which Splunk classifies as a centralized streaming command — all the work happens on the Search Head and consumes substantial RAM.

Splunk documents six command types: distributable streaming, centralized streaming, transforming, generating, orchestrating, and dataset processing. A single command can belong to more than one type depending on its arguments, which is exactly what exam distractors exploit.