15.2 Alarms & Notifications: Trigger Rules, Absence Alarms, Split Messages, Suppression & Delivery
Key Takeaways
OCI Monitoring evaluates every alarm once per minute.
An alarm can have up to two trigger rules, for example a critical rule at 95% CPU and a warning rule at 90% CPU.
The trigger delay (pendingDuration) is how long the condition must persist before an alarm moves to the FIRING state.
An absence alarm uses a default absence detection period of two hours, configurable from one minute to three days.
An alarm can send at most 60 messages per evaluation to a Notifications topic and up to 100,000 to a Streaming stream.
What an Alarm Contains
An alarm is an MQL alarm query plus the notification destination to use when it fires. Its main properties:
| Property | Purpose | Example |
|---|---|---|
| Metric namespace and query | The MQL expression, including the trigger rule | CpuUtilization[1m].mean() > 90 |
| Trigger rule | The condition (threshold or absence), its severity, trigger delay, and body | Severity CRITICAL, delay PT5M |
| Severity | Critical, Error, Warning, or Info | Critical for outages, Warning for early signs |
| Body | Human-readable guidance, which can include runbook links and dynamic variables such as {{query}} | "High CPU. Follow runbook step 3." |
| Destination | A Notifications topic or a Streaming stream | ops-critical topic |
| Message grouping | Group all metric streams into one message, or split per metric stream | Split for per-server alerts |
| Message format | Raw, pretty JSON, or formatted (ONS optimized) | Formatted for email |
| Repeat notification | Re-send while the alarm stays firing | Every 1 hour |
An alarm can have up to two trigger rules, so one alarm can send a Warning at 90% and a Critical at 95%.
How Evaluation Works
Monitoring evaluates each alarm once per minute. For alarm queries, the only valid resolution is 1m, while the interval in the query (for example [5m]) sets the window that each data point aggregates.
The trigger delay (pendingDuration) sets how long the condition must hold before the alarm fires. Consider CpuUtilization[1m].percentile(0.9) > 85 with a trigger delay of 3 minutes: the alarm moves to FIRING only after three consecutive one-minute evaluations are true. It returns to OK when the breaching condition has been clear for the most recent minute. A trigger delay filters out short spikes, which is the main lever for reducing noise.
Updates to an alarm can take up to five minutes to take effect everywhere.
Threshold and Absence Alarms
- A threshold alarm fires when a metric value crosses a limit, for example memory above 85%.
- An absence alarm uses
absent()and fires when a metric stops arriving, which often means an agent, instance, or application has stopped reporting. The absence detection period defaults to two hours and can be set from one minute to three days.
When a metric stream that caused the alarm to fire stops reporting, Monitoring starts an internal reset period of 10 minutes. If the stream stays absent for that whole period, the alarm stops evaluating it and can send a RESET message. A RESET means you should check the health of the resource, because it may have stopped emitting data rather than recovered.
Alarm States and Message Types
| Message type | When it is sent |
|---|---|
OK_TO_FIRING | The alarm moves from OK to FIRING |
FIRING_TO_OK | The condition clears |
REPEAT | The alarm stays FIRING and repeat notifications are configured |
RESET | The alarm returns to OK after one or more internal resets |
Grouped Versus Split Notifications
- Grouped: one message describes all metric streams that meet the condition. This is good for a fleet-wide view, such as "4 instances above 90% CPU".
- Split per metric stream: Monitoring evaluates each metric stream separately and sends a message for each one that changes state. This is good for routing a ticket per server. The
dedupekeyfield helps receivers group messages for the same alarm and stream.
Split messages are subject to destination limits per evaluation: up to 60 messages for a Notifications topic and 100,000 for a Streaming stream. Extra messages are dropped. An alarm tracks up to 200,000 metric streams. For large fleets that need per-resource messages, use a stream as the destination, or use grouped messages.
Suppressions
A suppression stops an alarm from publishing messages during a time range, such as a maintenance window. Suppressions can apply to the whole alarm or to specific dimensions, for example only the instance being patched, and you can suppress several alarms at once. The alarm definition stays unchanged; only its messages stop until the suppression ends, which is safer than disabling or deleting alarms and forgetting to restore them.
Notifications Topics and Delivery
The usual destination is an OCI Notifications topic. Its subscriptions fan the alarm out to email, Slack, PagerDuty, SMS, HTTPS endpoints, or Functions (Section 7.2). Notifications delivers at least once per subscription and retries failed deliveries, so receivers should be idempotent. A function subscription can run automated remediation, such as scaling an instance pool or collecting diagnostics.
When a single topic or stream serves many alarms, it can be overused and delay notifications. Oracle recommends separate topics or streams for high-volume alarms, enough stream partitions for the expected load, and higher tenancy limits when needed.
Designing Useful Alarms
- Alert on symptoms users feel, such as API Gateway 5xx errors or latency, and use resource metrics for diagnosis.
- Use trigger delays to ignore brief spikes, and two trigger rules for early warning and real problems.
- Add absence alarms for critical metrics so silent failures are noticed.
- Write bodies that tell the on-call engineer what to do, with runbook links.
- Route by severity: Critical to paging, Warning to chat or email.
- Apply suppressions during planned maintenance instead of disabling alarms.
An alarm on API latency fires several times a day because of 30-second spikes that recover on their own. The team only wants to be paged when latency stays high for five minutes. What should be changed?
Change the alarm query resolution to 5 minutes
Enable repeat notifications at a 5-minute interval
Set the trigger delay (pendingDuration) to 5 minutes
Change the alarm destination from a topic to a stream
A split-notification alarm watches 500 web servers and sends messages to a Notifications topic. During an outage, 300 servers cross the threshold in the same minute, but the team receives only some of the messages. Why?
The alarm is evaluated only once every five minutes, so it missed most transitions
Notifications discards alarm messages from alarms that have no trigger delay
Split notifications are limited to 100 metric streams per alarm, so the rest are ignored
A topic destination accepts at most 60 messages per evaluation, so the rest are dropped
A batch job writes a custom metric every 10 minutes. If the job stops, the team wants to know within 30 minutes. Which alarm design fits?
An absence alarm using absent() on the heartbeat metric
A threshold alarm with JobHeartbeat[1m].count() < 1
A threshold alarm on the heartbeat with repeat notifications every 30 minutes
A suppression on the heartbeat alarm that ends after 30 minutes
Sections you finish are checked off in the contents.