6.3 Monitors, Alerts, and Notifications
Key Takeaways
- Metric monitors evaluate metric values against alert and warning thresholds; Host monitors track whether hosts stop reporting via datadog.agent.up.
- Grouping a monitor by a tag key (avg by host, group by device) creates separate alert evaluations per tag value — essential for per-disk or per-host alerting.
- Monitor tags are metadata on the monitor object itself, separate from metric tags used in the query scope or group-by logic.
- No-data notification settings alert when expected telemetry stops — catching Agent failures and broken pipelines, not just bad metric values.
- Alert aggregation controls whether you receive one summarized notification or separate alerts for each group that breaches a threshold.
Why Monitors Matter on the Exam
Dashboards help you see problems; monitors tell you about problems when nobody is watching. The Visualization & Utilization domain includes monitors and alerts because effective observability requires both investigation views and automated response. The Fundamentals exam tests monitor types, grouping behavior, tag distinctions, and notification settings — not advanced composite or anomaly detection logic.
Monitor Types You Must Know
| Monitor type | Evaluates | Typical use case |
|---|---|---|
| Metric | Metric values against thresholds | CPU above 90%, latency above 500 ms, disk below 10% free |
| Host | Whether hosts report data (datadog.agent.up) | Alert when a host stops sending telemetry |
| Service Check | Agent integration check status | Integration health, custom check pass/fail |
| Event | Incoming events matching a query | Deployment failures, security signals |
| Log | Log count or pattern thresholds | Error spike in application logs |
For threshold questions — "compare a metric against a fixed limit" — the answer is metric monitor. For "alert when hosts stop reporting" — the answer is host monitor, typically evaluating the datadog.agent.up service check.
Thresholds, Warning vs Alert, and Evaluation Window
A metric monitor defines alert and optional warning thresholds. Datadog evaluates the metric over a configurable lookback window (for example, "avg over the last 5 minutes") and transitions the monitor through states: OK, Warn, Alert, and No Data.
The evaluation logic matters on the exam. A monitor set to "avg of system.cpu.user over the last 5 minutes > 90" fires only if the condition holds across the full window — a brief spike may not trigger if the average stays below threshold. Read the scenario carefully for whether it describes sustained saturation (avg) or instantaneous peaks (max).
Multi-Alert Grouping: avg by and group by
One of the most-tested concepts is grouping. When you create a metric monitor and set avg by host (or group by device, service, availability_zone), Datadog evaluates the threshold separately for each tag value. Each group gets its own alert state.
This is critical for disk space monitoring. If you monitor system.disk.in_use without grouping, a fleet average might look healthy while one disk on one host is full. Group by host and device creates separate evaluations for each disk on each system — the exam explicitly tests this scenario.
Conversely, when a team wants one summarized alert instead of fifty separate notifications for fifty hosts, they adjust the alert aggregation strategy. Too much aggregation hides which group failed; too little creates notification noise. The exam may ask which setting controls summarized vs per-group notifications — that is alert aggregation, not log rollover or DogStatsD buffering.
Monitor Tags vs Metric Tags
This distinction trips up many candidates:
- Metric tags appear in the monitor query — they scope which data is evaluated (
env:prod) or define group-by dimensions (by {host}). - Monitor tags are metadata on the monitor object itself — used to search, filter, and organize monitors in the UI (for example
team:platform,severity:high).
Monitor tags do not determine the Agent hostname, cannot be set only through DogStatsD, and are not interchangeable with metric tags in the query. If a question asks what monitor tags are for, the answer is ownership, search, and organization — not query scoping.
No-Data Handling
No-data notification tells Datadog how to behave when expected telemetry disappears. This is distinct from threshold alerts: a host can stop reporting (no data) without ever crossing a CPU threshold.
Enable no-data alerts when silence itself is the signal — Agent stopped, integration broken, pipeline stalled. The exam contrasts this with threshold-only monitors that stay OK forever when data stops flowing. No-data handling does not make every monitor composite, convert no-data into success, or remove the need for thresholds — it complements them.
Practical pairing: a host monitor on datadog.agent.up for infrastructure visibility plus metric monitors with no-data enabled for critical application metrics.
Notifications and Message Variables
When a monitor fires, Datadog sends notifications to configured channels — email, Slack, PagerDuty, webhooks, and others. Notification messages support template variables that render context from the triggering group: host name, metric value, threshold, and tags.
Conditional variables render a value only when a specific tag is present on the triggering group — useful for messages that should show service or env when available but stay clean when those tags are missing. This is an exam-level detail: conditional variables are about notification content, not hostname resolution or rollup settings.
Monitor Lifecycle and States
Monitors move through states that appear in the Monitors UI and on status pages:
- OK — condition not met
- Warn — warning threshold crossed
- Alert — alert threshold crossed
- No Data — expected metrics missing
- Ignored — manually or automatically suppressed
Understanding states helps you interpret exam scenarios about on-call response. A monitor in No Data with no-data notifications disabled may silently indicate a collection problem — the kind of gap Fundamentals questions you to recognize.
Common Exam Traps
- Host monitors alert on reporting gaps, not Agent installation, YAML editing, or custom metric storage.
- Group by host/device creates separate evaluations, not a single aggregate alert.
- Monitor tags are for monitor management, not query scoping.
- No-data alerts detect telemetry stoppage, not bad values.
- Metric type changes on the Metrics Summary page affect monitors globally, not just one view.
A Datadog Host monitor is best suited for which problem?
When creating a metric monitor for low disk space, why might you group by both host and device?
Which statement about monitor tags is correct?