13.2 Cloud Monitoring: Alerts, Custom Metrics & Dashboards
Key Takeaways
- An alerting policy consists of one or more conditions plus one or more notification channels; Cloud Monitoring auto-creates a timestamped incident when a condition is met and auto-closes it when the condition clears.
- The five alerting condition families are metric threshold, metric absence, forecasted metric-value, PromQL, and log-based — each maps to a different exam scenario phrase.
- Log-based metrics derive counter or distribution data straight from Cloud Logging entries with no application code changes; user-defined custom metrics require writing to the Cloud Monitoring API and use the custom.googleapis.com/ prefix.
- Uptime checks are black-box external reachability probes run from multiple global locations and are distinct from internal metric-threshold alerting.
- Dashboards come prebuilt automatically per resource type or can be custom-built via Console, API, or Terraform, queried with MQL or PromQL.
Why This Topic Matters on the Exam
The official exam guide lists "creating Cloud Monitoring alerts" and "creating/ingesting custom metrics" as explicit, testable bullets under Domain 4. In practice, ACE questions rarely ask you to define "monitoring" in the abstract — they describe an operational requirement ("notify the on-call team only when latency crosses a threshold for five straight minutes") and ask which Cloud Monitoring construct to configure. Knowing the pieces of an alerting policy and the two ways to get non-default data into Cloud Monitoring is the difference between guessing and answering confidently.
Cloud Monitoring Building Blocks
Cloud Monitoring (the modern name for what used to be called Stackdriver Monitoring) collects metrics, uptime data, and events across Google Cloud, AWS, on-premises systems, and third-party applications, then lets you visualize, alert on, and analyze them.
A metrics scope (formerly called a "Workspace") is what ties one host project to one or more monitored projects, letting a single Cloud Monitoring view span multiple GCP projects — common in organizations that separate projects by environment (dev/staging/prod) but want unified dashboards and alerting.
Alerting Policies
An alerting policy is the core object that watches for a condition and fires a notification. It has three parts:
- Condition(s) — what to watch and when it counts as "bad."
- Notification channel(s) — where the alert goes.
- Documentation — optional Markdown notes attached to the alert so the on-call responder has context (runbook link, owning team, etc.).
| Condition type | Triggers when | Typical use |
|---|---|---|
| Metric threshold | A metric value crosses above/below a value for a sustained duration | "CPU utilization > 90% for 5 minutes" |
| Metric absence | A monitored time series stops reporting data for a defined window (up to 24 hours) | Detect a crashed or stalled process that stops emitting metrics entirely |
| Forecasted metric-value | Extrapolation predicts a threshold breach 1-7 days ahead | Proactively catch a disk that will fill up before it actually does |
| PromQL | A Prometheus Query Language expression evaluates true | Complex, multi-metric conditions, especially for GKE workloads emitting Prometheus-format metrics |
| Log-based | A Cloud Logging entry matches a filter | Alert the moment a specific error string appears in logs, with no code changes |
Aggregation matters: each condition specifies an alignment period (the time bucket, e.g., 1 minute) and a reducer (mean, max, sum, count, percentile) applied within that bucket, plus an optional group by on resource labels (e.g., alert per-region rather than on the aggregate across all regions).
Notification channels are project-level resources — email, SMS, Pub/Sub, Slack, PagerDuty, or a generic webhook — created once and then attached to any number of policies. Incidents are the timestamped records Cloud Monitoring creates each time a policy's condition is met; Monitoring auto-closes an incident once the condition clears.
Policies can be built in the Console, via the Cloud Monitoring API, gcloud, or Terraform (google_monitoring_alert_policy) — the exam expects you to recognize that alerting is fully automatable via API/Terraform, not just console-only.
Uptime checks are a distinct, simpler mechanism: black-box HTTP(S)/TCP probes sent from multiple global locations to test whether a public endpoint is reachable at all. An uptime check's failure can itself feed an alerting policy — this is "external reachability," as opposed to a metric-threshold condition on internal telemetry.
Custom Metrics: Two Paths
Cloud Monitoring ships with thousands of built-in ("system") metrics for GCP resources, but the exam guide specifically calls out creating/ingesting custom metrics — meaning data Google does not collect for you automatically. There are two supported paths:
- Log-based metrics — Cloud Logging can automatically derive a counter metric (how many matching entries occurred) or a distribution metric (a numeric value extracted from each entry, e.g., response size) from any log filter, with no application code changes. Example: a counter metric counting
severity=ERRORlog entries containing"PaymentDeclined", which you then alert on directly. - User-defined custom metrics — application code (or the Ops Agent, or an OpenTelemetry exporter) writes data points directly to the Cloud Monitoring API (
projects.timeSeries.create). Custom metric types use thecustom.googleapis.com/orexternal.googleapis.com/prometheus/prefix so Cloud Monitoring can distinguish them from built-in system metrics.
Exam trap: a scenario describing "we want a metric based on how often a specific error message appears in our logs, and we don't want to modify application code" is asking for a log-based metric, not a custom API-ingested metric — the "no code change" phrase is the tell.
Dashboards
Dashboards visualize metrics as widgets (line charts, scorecards, alerting charts, log panels). Google auto-generates prebuilt dashboards per resource type (Compute Engine, GKE, Cloud SQL, and more) the moment you use that resource — no configuration needed. You can also build custom dashboards through the Console, the Monitoring Dashboards API, or Terraform (google_monitoring_dashboard), querying data with MQL (Monitoring Query Language) or PromQL widgets.
Applying It: A Worked Scenario
"You need to page the on-call engineer via PagerDuty only when the 95th-percentile request latency for a Cloud Run service exceeds 500ms for five consecutive minutes."
To configure this: create an alerting policy with a metric threshold condition on the Cloud Run request-latency metric, set the aggregation to a 95th-percentile reducer over a 1-minute alignment period, require the condition to hold for a 5-minute duration, and attach a PagerDuty notification channel. No custom metric is needed here — Cloud Run's built-in latency metric already exists; you are only configuring the alerting policy around it.
Section Takeaways
- An alerting policy = condition(s) + notification channel(s), evaluated continuously, producing auto-closing incidents.
- Metric-threshold, metric-absence, forecasted, PromQL, and log-based are the five condition families the exam expects you to distinguish.
- Log-based metrics require zero application code changes; user-defined custom metrics require writing to the Monitoring API (or via Ops Agent/OpenTelemetry).
- Uptime checks test external reachability and can themselves trigger alerts, distinct from internal metric-threshold monitoring.
- Dashboards can be prebuilt (automatic) or custom (Console/API/Terraform), queried with MQL or PromQL.
A team wants an alert to fire whenever a specific error string appears in their application logs, and they do not want to modify any application code to make this happen. Which Cloud Monitoring feature should they use?
An alerting policy needs to notify the team only when a time series that normally reports every minute stops reporting data entirely for 20 minutes, which would indicate a crashed process. Which condition type fits?