12.2 Alert Management, Threat Hunting, Triage, Reporting

Alert Management Is a Workflow

Once correlation creates alerts, the SOC needs a queue, not a directory of log files. Alert management means assigning owners, honoring severity, updating status, grouping related notables, and disposing each item so a Critical New case does not sit untouched for half a shift.

SAL1 training content calls out alert management, threat hunting, triage, security reporting, and using alert properties such as severity and status. The SOC Simulator scores classification, escalation, and case reports in later workflow chapters. This section is the SIEM-console skill that feeds those reports: you prove or disprove a notable with searches, then write what you did.

Properties you actually click:

• Severity — informational, low, medium, high, critical (names vary). It answers “how bad if true?” Tune the detection if everything is Critical; do not silently ignore High because you are tired.
• Status — New → In progress → Pending (waiting on a system owner) → Closed, plus Suppressed for known noisy patterns. Suppressed is not a synonym for “I disliked this alert.”
• Owner — if two analysts open the same notable, you duplicate work and split the timeline.
• Urgency vs severity — some products multiply severity by asset criticality. A Low on a domain controller can outrank a High on a lab VM. If your console has a criticality field, use it.
• Related notables — same src, same user, same host. Pivot before close.

Close with a verdict in the note even when the product’s status vocabulary is just “Closed”: true positive (malicious or attempted), false positive (the rule misfired), or benign true positive (the pattern happened but was authorized). Later SAL1 workflow material goes deeper on those labels; here, know that status without a verdict is incomplete reporting.

Hunting vs Alert-Driven Triage

Alert-driven triage is reactive. The SIEM, EDR, or email gateway already put a row in the queue. The rule supplied the hypothesis. You are time-boxed: validate, enrich, close, or escalate.

Threat hunting is proactive. You start with a question the current rules may not cover: “Which workstations launched powershell.exe with a network connection to a destination we have never seen?” You search, stack, and only then open a ticket if the answer looks evil. Hunting uses the same SIEM indexes. It is not a different product and not “more advanced Splunk.” It is a different trigger.

Junior trap: hunting while High/Critical New items miss acknowledgment targets. Finish the assigned queue unless a lead gave you a hunt window. Opposite trap: a SOC that only ever clicks vendor notables never finds the attacker who stayed under every threshold. Both modes belong in a week’s work; they do not replace each other.

Splunk SPL a Junior Analyst Actually Types

Search Processing Language (SPL) is how you query Splunk. You do not need to be a Splunk architect for SAL1. You need a small set of commands that turn thousands of events into an answer. Anatomy:

index=<where> <filters> | <transforming commands>

The time picker is part of the search. Twenty-four hours on a verbose index can time out. Fifteen minutes can miss the first failures of a brute-force that started sixteen minutes ago. When an alert says “22 events from 14:02–14:08,” set the picker a little before the first event and a little after the last so you see setup and follow-on success.

Everything before the first pipe is an implicit search: Boolean AND / OR / NOT, quoted phrases, field=value. Example:

index=wineventlog sourcetype="WinEventLog:Security" EventCode=4625 TargetUserName=svc_backup

Avoid a leading wildcard on a huge index (*password*); it is slow. Use NOT EventCode=4672 to subtract privilege-on-logon noise when that EventCode drowns the view. If a field was never parsed, you can still string-match _raw, but that is a fallback, not a deployment strategy.

stats — counts and distinct counts

stats aggregates. This is how 8,000 failures become a table of offenders:

index=wineventlog EventCode=4625
| stats count by IpAddress, TargetUserName
| where count > 10
| sort -count

Password spray vs brute force in one extra aggregation:

index=wineventlog EventCode=4625
| stats count, dc(TargetUserName) as unique_users by IpAddress
| where unique_users > 15 OR count > 20

High count and unique_users=1 is hammering one account. High unique_users with modest per-user counts is a spray. That distinction belongs in the case report.

dedup — one row per entity, not a volume metric

dedup keeps the first event per listed fields (after sort, “first” is often latest _time). Useful for “latest successful 4624 per user” on a handover, or collapsing identical proxy rows for a screenshot.

index=wineventlog EventCode=4624
| sort -_time
| dedup TargetUserName
| table _time, TargetUserName, IpAddress, LogonType

Trap: dedup before stats count destroys the very volume that proved brute force. Dedup is for presentation or unique-entity lists. Measure volume with stats.

timechart — shape over time

index=wineventlog EventCode=4625 IpAddress=203.0.113.88
| timechart span=5m count by TargetUserName

A flat drip all day is often a misconfigured service. A spike at 14:00 is a burst. Beaconing hunts use timechart span=1h count looking for suspiciously regular columns. If the chart is empty, check the time picker and the index name before you close as false positive.

table — what you paste into a ticket

... | table _time, host, TargetUserName, IpAddress, LogonType, FailureReason

Analysts paste table output into case notes. Do not dump _raw XML for a manager. Do not screenshot 5,000 rows; screenshot the stats summary and attach a few representative events.

lookup — extra columns from a CSV or KV store

... | lookup asset_lookup dest as dest OUTPUT owner, criticality, business_unit
| lookup ti_ip_lookup IpAddress as src OUTPUT threat_category, confidence

Lookups join live events to asset owners, VIP lists, or threat-intel indicators. “Another 4625” becomes “4625 from a listed C2 IP against a domain-admin workstation.” SAL1 later treats metrics and lookups as a SOC-operations topic; the SPL verb that applies those tables during a search is lookup. If the lookup file is stale, enrichment is stale — say so in the report.

You will also meet head, sort, eval, rename, rex (extract from _raw when parsing failed), and transaction. Learn them when a search requires them. The six ideas above — search filters, stats, dedup, timechart, table, lookup — cover most simulator and first-job questions.

Other SIEMs spell the same jobs differently (KQL summarize in Sentinel, Elasticsearch aggregations, QRadar AQL). For SAL1, type SPL. For an interview, say “I grouped failures by source and distinct users” and then name the local syntax.

Worked Queue Triage (Alert-Driven)

Given. Splunk alert, severity High, status New, title “Brute force: svc_backup from 203.0.113.88,” 22 events, 14:02–14:08.

1. Take ownership; set status to In progress so a teammate does not duplicate you.
2. Time range 13:50–14:20 (pad the advertised window).
3. Validate the pattern, including LogonType and failure codes:

index=wineventlog EventCode=4625 TargetUserName=svc_backup IpAddress="203.0.113.88"
| stats count, min(_time) as first, max(_time) as last by LogonType, SubStatus

1. Ask whether they got in:

index=wineventlog EventCode=4624 TargetUserName=svc_backup IpAddress="203.0.113.88"
| table _time, LogonType, LogonProcessName, AuthenticationPackageName

1. lookup the IP on the threat list and the host on the asset list. Check proxy/firewall for the same IP if those indexes exist.
2. If zero 4624s — still an attempt. Close or escalate per runbook (many SOCs escalate High service-account brute force even without success). If a 4624 exists — this is no longer “just failures”; escalate immediately, hunt that session’s processes, and do not sit on status New.
3. Write the report: indexes, time range, SPL, counts, first/last, success/fail, lookup hits, related notables, next action. Set status Closed or leave In progress if you handed it to IR.

Worked Hunt (No Alert Yet)

Question. After 21:00, which hosts show powershell.exe making outbound connections to destinations not on a common-allow lookup? Sysmon Event ID 3 (network connect) in Splunk is a typical place to ask this when EDR coverage has gaps.

index=sysmon EventCode=3 Image="*\\powershell.exe"
| lookup popular_dest_lookup dest OUTPUT is_common
| search is_common!=1
| stats dc(dest) as dests, values(dest) as dest_list by host, User
| where dests > 5

You are not closing a notable. You are producing candidates. If WS-042 shows 40 destinations, then you create a ticket — and from that moment you are back in alert-driven discipline: time-boxed, documented, severity assigned. An empty hunt is still worth a one-paragraph note so the next shift does not rerun the identical question without knowing it was already asked.

Security Reporting from the SIEM

Reporting is how the work leaves your head:

• Case notes — timeline, indicators (IPs, users, hashes), searches run, verdict. The SAL1 simulator scores case reports; practice writing them from table and stats, not from memory.
• Shift handover — still-open High/Critical, pending owners, hunts in flight, lookups that failed.
• Periodic SOC view — volume by severity, true-positive rate, noisiest rules. Mean time to acknowledge and respond come from timestamps on status changes; those metrics are a later chapter, but the SIEM is where the clocks start.
• Tuning request — “Rule 1024 fires on the scanner every Tuesday 02:00; add this src to the allow lookup.” That is a report, not a complaint in chat.

A note that only says “looks like brute force” is not a report. Include data source, time range, filter or SPL, counts, and what would change your mind (for example, “I would escalate if a 4624 from this IP appears”). That sentence is how another analyst reruns your work without guessing.