8.3 Creating Search Workflow Actions & Dynamic Secondary Searches

8.3 Creating Search Workflow Actions & Dynamic Secondary Searches

Quick Summary: Search workflow actions execute secondary, parameterized Splunk searches directly from the Event Actions or Field Actions menus. By interpolating event tokens (e.g., $src_ip$, $user$, $_time$) into pre-authored SPL templates, search actions enable instant cross-index correlation, deep-dive forensic pivots, and automated root-cause analysis. Key configuration parameters include search string templates (search.search_string), time range inheritance and offsets (search.earliest, search.latest, search.preserve_timerange), application context (search.app), and target view routing (search.view).

---

1. The Power of Search Workflow Actions

In enterprise operations, discovering an anomaly in one log source is almost always the beginning of an investigation, not the end. For example:

• An alert in index=web shows an HTTP 500 error associated with clientip=198.51.100.42 at 14:15:00. The analyst needs to immediately inspect index=firewall for all connections from that IP, and index=os for database CPU spikes at that exact time.
• A security alert flags a failed login for user=svc_finance in index=wineventlog. The SOC analyst needs to inspect all authentication events for that user across the entire network in the surrounding 30-minute window.

Without search workflow actions, analysts must manually open a new search window, retype the target index names, reconstruct complex SPL boolean logic, copy-paste IP addresses or usernames, and manually set the time picker.

Search workflow actions automate this entire pivoting lifecycle into a single click.

+------------------------------------------------------------------------------------------------+
|                               SEARCH WORKFLOW ACTION PIVOT FLOW                                |
|                                                                                                |
|  1. Initiating Search Result (Web Access Logs):                                                |
|     2026-08-24 14:15:00 | host=web-srv01 | clientip=198.51.100.42 | status=500                 |
|                                                                                                |
|  2. User clicks Event Action: "Investigate Firewall Traffic for 198.51.100.42 (+/- 15m)"       |
|                                                                                                |
|  3. Splunk Interpolates Configured SPL Template:                                               |
|     Template: index=firewall (src_ip=$clientip$ OR dest_ip=$clientip$) | stats count by action |
|     Resolved: index=firewall (src_ip=198.51.100.42 OR dest_ip=198.51.100.42) | stats count... |
|                                                                                                |
|  4. Splunk Resolves Time Range:                                                                |
|     Event Time: 14:15:00  --> Earliest: 14:00:00 (-15m)  --> Latest: 14:30:00 (+15m)          |
|                                                                                                |
|  5. Splunk Opens Search App in New Tab and Executes the Resolved Search Immediately!          |
+------------------------------------------------------------------------------------------------+

---

2. Configuration Parameters for Search Workflow Actions

Search workflow actions are governed by dedicated attributes in workflow_actions.conf or the corresponding form fields in Splunk Web under Settings > Fields > Workflow actions:

---

3. Time Range Inheritance & Relative Offset Mechanics

One of the most critical aspects of Search Workflow Actions is controlling the time range of the secondary search. Splunk provides two primary models: Time Range Preservation and Relative Time Offsets.

+------------------------------------------------------------------------------------------------+
|                                 TIME RANGE SELECTION STRATEGIES                                |
|                                                                                                |
|  OPTION A: PRESERVE TIME RANGE (search.preserve_timerange = true)                              |
|  • Secondary search inherits the broad time window of the initiating search (e.g., "Last 7d").|
|  • Use case: Broad summary reports and aggregate pattern analysis.                            |
|                                                                                                |
|  OPTION B: RELATIVE TIME OFFSET (search.preserve_timerange = false)                            |
|  • Secondary search anchors around the exact event timestamp ($_time$) with specified bounds.   |
|  • search.earliest = -15m@m  (15 minutes before event)                                         |
|  • search.latest   = +15m@m  (15 minutes after event)                                          |
|  • Use case: High-precision forensic triage and local event correlation.                       |
+------------------------------------------------------------------------------------------------+

1. Preserving Global Time Range (search.preserve_timerange = true)

If an analyst ran a search over "Previous 24 Hours" or "Last 7 Days", setting search.preserve_timerange = true causes the secondary search to execute across that identical 24-hour or 7-day window. This is ideal when the analyst wants to know if a specific error code or user appeared anywhere else during the overall time period under investigation.

2. Forensic Window Offsets (search.preserve_timerange = false)

When triaging a specific incident (e.g., a malware beacon or server crash), searching a broad 7-day window generates overwhelming noise. By setting search.preserve_timerange = false and specifying relative offsets in search.earliest and search.latest:

• search.earliest = -30m
• search.latest = +30m

Splunk calculates the time bounds relative to the exact _time of the clicked event. If the event occurred at 14:15:00, the secondary search is strictly bound between 13:45:00 and 14:45:00, providing instant, targeted forensic context.

---

4. Application & View Targeting

Search workflow actions allow Power Users to direct the secondary search into specialized Splunk apps and custom views:

                           ┌────────────────────────────────────────┐
                           │   Search Workflow Action Routing       │
                           └───────────────────┬────────────────────┘
                                               │
                      ┌────────────────────────┴────────────────────────┐
                      ▼                                                 ▼
         ┌─────────────────────────┐                       ┌─────────────────────────┐
         │  search.app = <app>     │                       │  search.view = <view>   │
         │ (Application Container) │                       │  (Target View / UI)     │
         ├─────────────────────────┤                       ├─────────────────────────┤
         │ • search (Default)      │                       │ • search (Default view) │
         │ • SplunkEnterpriseSec   │                       │ • flashtimeline (Classic)│
         │ • ITSI / AppAnalytics   │                       │ • custom_dashboard      │
         └─────────────────────────┘                       └─────────────────────────┘

Why App & View Targeting Matters:

1. Knowledge Object Isolation: If the secondary search utilizes search macros, calculated fields, or event types defined only within a specific application (e.g., SplunkEnterpriseSecuritySuite), setting search.app = SplunkEnterpriseSecuritySuite ensures the SPL query has access to those private/app-scoped knowledge objects.
2. Custom Investigation Dashboards: By routing to a custom dashboard view (search.view = investigator_dashboard), the search query can populate a specialized multi-panel visualization layout rather than the generic Search & Reporting interface.

---

5. SOC & NOC Real-World Implementation Scenarios

Scenario 1: Cross-Index User Audit across Windows & VPN Logs

An enterprise SOC wants an Event Action on Windows Security logs that immediately searches across both VPN and Domain Controller logs for all activity involving that user within 1 hour before and after the event.

Configuration in workflow_actions.conf:

# $SPLUNK_HOME/etc/apps/search/local/workflow_actions.conf

[correlate_user_activity]
type = search
label = Audit all activity for user $user$ (+/- 60m)
display_location = event_menu
sourcetype = WinEventLog:Security
search.app = search
search.view = search
search.search_string = (index=wineventlog OR index=vpn) user="$user$" | stats count, values(action), values(src_ip) by index, sourcetype
search.preserve_timerange = false
search.earliest = -60m
search.latest = +60m
search.target = blank
search.run_on_open = true

Scenario 2: Network Connection Triage from Web Access Logs

When inspecting web access logs (access_combined), an analyst wants a Field Action on clientip to correlate firewall permit/drop actions.

Configuration in workflow_actions.conf:

[firewall_triage_by_ip]
type = search
label = Trace Firewall Traffic for IP $@field_value$
display_location = field_menu
fields = clientip, src_ip, dest_ip
search.search_string = index=firewall (src_ip="$@field_value$" OR dest_ip="$@field_value$") | stats count by action, dest_port, proto
search.preserve_timerange = false
search.earliest = -15m
search.latest = +15m
search.target = blank

---

6. Comprehensive Workflow Actions Configuration Reference

Here is a complete comparison of all configuration parameters across the three workflow action types in workflow_actions.conf:

---

7. Power User Troubleshooting & Best Practices

1. Always Quote Substituted String Tokens in SPL: When authoring search.search_string, always wrap field tokens in double quotes if they might contain spaces or special characters: index=web user="$user$". If user contains john doe, unquoted user=$user$ resolves to user=john doe, which is parsed as user=john AND doe.
2. Verify Field Token Existence: Remember that missing tokens cause automatic suppression. If your search query references $src_ip$ and $dest_ip$, but an event only has $src_ip$, the action will not appear. Use separate workflow actions or generalized tokens ($@field_value$) when appropriate.
3. Test with search.run_on_open = false during Development: When authoring complex secondary searches, set search.run_on_open = false. This populates the search bar in the new tab without executing immediately, allowing you to inspect the resolved SPL query and time range before running.