6.3 Monitors, Alerts, and Notifications
Key Takeaways
Metric monitors evaluate metric values against alert and warning thresholds; Host monitors track whether hosts stop reporting via datadog.agent.up.
Grouping a monitor by a tag key (avg by host, group by device) creates separate alert evaluations per tag value — essential for per-disk or per-host alerting.
Monitor tags are metadata on the monitor object itself, separate from metric tags used in the query scope or group-by logic.
No-data notification settings alert when expected telemetry stops — catching Agent failures and broken pipelines, not just bad metric values.
Alert aggregation controls whether you receive one summarized notification or separate alerts for each group that breaches a threshold.
Why Monitors Matter on the Exam
Dashboards help you see problems; monitors tell you about problems when nobody is watching. The Visualization & Utilization domain includes monitors and alerts because effective observability requires both investigation views and automated response. The Fundamentals exam tests monitor types, grouping behavior, tag distinctions, and notification settings — not advanced composite or anomaly detection logic.
Monitor Types You Must Know
| Monitor type | Evaluates | Typical use case |
|---|---|---|
| Metric | Metric values against thresholds | CPU above 90%, latency above 500 ms, disk below 10% free |
| Host | Whether hosts report data (datadog.agent.up) | Alert when a host stops sending telemetry |
| Service Check | Agent integration check status | Integration health, custom check pass/fail |
| Event | Incoming events matching a query | Deployment failures, security signals |
| Log | Log count or pattern thresholds | Error spike in application logs |
For threshold questions — "compare a metric against a fixed limit" — the answer is metric monitor. For "alert when hosts stop reporting" — the answer is host monitor, typically evaluating the datadog.agent.up service check.
Thresholds, Warning vs Alert, and Evaluation Window
A metric monitor defines alert and optional warning thresholds. Datadog evaluates the metric over a configurable lookback window (for example, "avg over the last 5 minutes") and transitions the monitor through states: OK, Warn, Alert, and No Data.
The evaluation logic matters on the exam. A monitor set to "avg of system.cpu.user over the last 5 minutes > 90" fires only if the condition holds across the full window — a brief spike may not trigger if the average stays below threshold. Read the scenario carefully for whether it describes sustained saturation (avg) or instantaneous peaks (max).
Multi-Alert Grouping: avg by and group by
One of the most-tested concepts is grouping. When you create a metric monitor and set avg by host (or group by device, service, availability_zone), Datadog evaluates the threshold separately for each tag value. Each group gets its own alert state.
This is critical for disk space monitoring. If you monitor system.disk.in_use without grouping, a fleet average might look healthy while one disk on one host is full. Group by host and device creates separate evaluations for each disk on each system — the exam explicitly tests this scenario.
Conversely, when a team wants one summarized alert instead of fifty separate notifications for fifty hosts, they adjust the alert aggregation strategy. Too much aggregation hides which group failed; too little creates notification noise. The exam may ask which setting controls summarized vs per-group notifications — that is alert aggregation, not log rollover or DogStatsD buffering.
Monitor Tags vs Metric Tags
This distinction trips up many candidates:
- Metric tags appear in the monitor query — they scope which data is evaluated (
env:prod) or define group-by dimensions (by {host}). - Monitor tags are metadata on the monitor object itself — used to search, filter, and organize monitors in the UI (for example
team:platform,severity:high).
Monitor tags do not determine the Agent hostname, cannot be set only through DogStatsD, and are not interchangeable with metric tags in the query. If a question asks what monitor tags are for, the answer is ownership, search, and organization — not query scoping.
No-Data Handling
No-data notification tells Datadog how to behave when expected telemetry disappears. This is distinct from threshold alerts: a host can stop reporting (no data) without ever crossing a CPU threshold.
Enable no-data alerts when silence itself is the signal — Agent stopped, integration broken, pipeline stalled. The exam contrasts this with threshold-only monitors that stay OK forever when data stops flowing. No-data handling does not make every monitor composite, convert no-data into success, or remove the need for thresholds — it complements them.
Practical pairing: a host monitor on datadog.agent.up for infrastructure visibility plus metric monitors with no-data enabled for critical application metrics.
Notifications and Message Variables
When a monitor fires, Datadog sends notifications to configured channels — email, Slack, PagerDuty, webhooks, and others. Notification messages support template variables that render context from the triggering group: host name, metric value, threshold, and tags.
Conditional variables render a value only when a specific tag is present on the triggering group — useful for messages that should show service or env when available but stay clean when those tags are missing. This is an exam-level detail: conditional variables are about notification content, not hostname resolution or rollup settings.
Monitor Lifecycle and States
Monitors move through states that appear in the Monitors UI and on status pages:
- OK — condition not met
- Warn — warning threshold crossed
- Alert — alert threshold crossed
- No Data — expected metrics missing
- Ignored — manually or automatically suppressed
Understanding states helps you interpret exam scenarios about on-call response. A monitor in No Data with no-data notifications disabled may silently indicate a collection problem — the kind of gap Fundamentals questions you to recognize.
Common Exam Traps
- Host monitors alert on reporting gaps, not Agent installation, YAML editing, or custom metric storage.
- Group by host/device creates separate evaluations, not a single aggregate alert.
- Monitor tags are for monitor management, not query scoping.
- No-data alerts detect telemetry stoppage, not bad values.
- Metric type changes on the Metrics Summary page affect monitors globally, not just one view.
A Datadog Host monitor is best suited for which problem?
Alerting when hosts stop reporting data
Installing the Agent on Windows
Storing custom metrics
Editing YAML files
When creating a metric monitor for low disk space, why might you group by both host and device?
To convert the metric into logs
To create separate alert evaluations for each disk on each host
To hide tags from dashboards
To force UDP delivery
Which statement about monitor tags is correct?
They are the same thing as metric tags in the monitor query
They determine the Agent hostname
They are separate from metric tags and are used to search and organize monitors
They can be set only through the DogStatsD protocol
Sections you finish are checked off in the contents.