9.2 Describing and Creating Alerts

Alerts at User level

An alert is a saved search designed for monitoring. Splunk runs the search on a schedule or continuously in real time, evaluates trigger conditions against the results, and — only when those conditions match — executes alert actions. That conditional step is the defining difference from a scheduled report.

Domain topics 8.3 and 8.4 expect you to describe alert types, create an alert from a search, set triggers and throttling, and attach basic actions. Keep Enterprise Security correlation and notable-event frameworks out of scope.

---

1. Alert vs scheduled report

Memory hook: Reports deliver on a clock. Alerts deliver on a condition (checked on a clock or in real time).

Both objects can email results. The exam cares whether that email is unconditional (report schedule) or conditional (alert trigger).

---

2. Scheduled alerts vs real-time alerts

Alert type is about search timing, not about whether email exists.

Scheduled alerts

• Search runs on a preset schedule or cron expression (similar mental model to scheduled reports).
• After each run, Splunk evaluates trigger conditions (for example number of results, number of hosts, or a custom condition).
• If conditions match, the alert can trigger once for that result set or once per matching result.
• Usually lighter on resources than continuous real-time searches — prefer scheduled alerts when minute-level detection is not required.

Real-time alerts

• Search runs continuously.
• Per-result mode: each matching event can trigger the alert (also called per-result alerting).
• Rolling time window mode: evaluate counts inside a moving window (for example more than 10 results in 5 minutes).
• More resource-intensive; Splunk guidance often says prefer scheduled alerts when possible.

User exam focus: Know that both types exist, that real-time is continuous and costlier, and that scheduled alerts combine a schedule with conditions (unlike scheduled reports).

---

3. Trigger conditions and trigger frequency

Trigger configuration answers two questions:

1. What pattern in the results should count as a fire?
2. How many times should actions run when that pattern appears?

Common condition ideas (scheduled / rolling-window style)

• Number of Results is greater than / less than / equal to a threshold
• Field-oriented counts such as number of hosts
• Custom conditions that further filter the result set

Example: search for status=500 over the last 15 minutes on a 15-minute schedule; trigger when Number of Results is greater than 50.

Once vs for each result

If 80 error events match and you choose for each result, you can generate up to 80 action executions before throttling. That is why throttling matters.

---

4. Throttling (suppression)

Throttling suppresses additional alert actions for a configured period after a trigger so the same condition does not spam inboxes.

Key points:

• Trigger conditions decide whether the alert is eligible to fire.
• Throttling decides how soon it may fire again after a successful trigger.
• Configure a suppression time period (must be greater than zero when enabled).
• Some designs also suppress per field values (for example per host) so one noisy host does not silence others — know the concept; deep per-field design is secondary at User level.

Scenario: Real-time per-result alert on every failed login without throttling can email continuously during an attack. Adding a 10-minute throttle collapses the storm into manageable notifications.

---

5. Create an alert from a search (User workflow)

1. Run and refine the monitoring search in Search & Reporting (tight time range, precise filters).
2. Choose Save As → Alert.
3. Name the alert; set permissions appropriate to your sharing needs (private vs app).
4. Choose alert type: scheduled or real-time.
5. For scheduled: set schedule and evaluate trigger conditions; choose once vs per result.
6. For real-time: choose per-result or rolling-window triggering.
7. Optionally enable throttle and set the suppression window.
8. Attach actions — memorize these User-level actions:
   • Email — notify recipients with results / links
   • List in Triggered Alerts (Add to Triggered Alerts) — show the fire on the Activity page
9. Save and verify with a controlled test when possible.

Severity levels (Info, Low, Medium, High, Critical) help sort fires later on the Triggered Alerts page; they do not replace trigger logic.

---

6. Exam traps

• Choosing Alert when the requirement is "email the dashboard stats every Monday regardless of values" → that is a scheduled report.
• Choosing scheduled report when the requirement is "email only if count > 100" → that is an alert.
• Enabling real-time per-result alerting for a high-volume sourcetype with no throttle → resource and email flood risk.
• Forgetting that List in Triggered Alerts is an action you enable — not every alert automatically appears unless configured to list there (per Splunk's monitor/triggered-alerts action model).

Practice one lab path end-to-end: search → Save As Alert → scheduled + number of results → throttle → email + list. That single path covers most of topics 8.3 and 8.4.