16.3 Application Metrics, Health Checks & Alerting with Cloud Monitoring

Application Metrics, Health Checks & Alerting with Cloud Monitoring

Maintaining the health, availability, and performance of cloud-native microservices requires comprehensive metric instrumentation and proactive alerting. Cloud Monitoring (formerly Stackdriver Monitoring) is Google Cloud's centralized telemetry platform, collecting system and application metrics, generating real-time dashboards, and evaluating alerting policies.

For the Google Cloud Professional Cloud Developer exam, you must master the Cloud Monitoring multidimensional data model, understand metric kinds and value types, design custom application metrics while avoiding cardinality traps, analyze latency distributions using heatmaps, configure time series alignment and reduction operations, and implement global uptime checks.

---

The Cloud Monitoring Data Model: Metric Kinds and Value Types

In Cloud Monitoring, telemetry is organized into Time Series. A time series is a regular or irregular sequence of timestamped data points uniquely identified by a Metric Descriptor, a Monitored Resource Descriptor, and associated label key-value pairs:

Time Series = MetricDescriptor + MonitoredResource + MetricLabels + Points[(t0, v0), (t1, v1), ...]

Metric Kinds (metricKind)

The metric kind defines how data points in the time series relate to the passage of time:

Exam Tip: Google Cloud natively prioritizes CUMULATIVE for counters (such as request counts). Because network packets can be delayed or arrive out of order, cumulative counters prevent data loss: if an intermediate scraping interval is missed, the next cumulative point captures the complete uninterrupted count since boot.

Value Types (valueType)

Every metric specifies the data format of its recorded values:

• BOOL: Boolean state (true / false), e.g., service binary health check.
• INT64: Signed 64-bit integer, e.g., exact transaction counts or queue depth.
• DOUBLE: Double-precision 64-bit floating point, e.g., CPU ratio (0.0 to 1.0).
• DISTRIBUTION: A complex statistical histogram containing histogram buckets, count, mean, and sum of squared deviations. Essential for recording request latency histograms, allowing post-hoc calculation of arbitrary percentiles (p50, p90, p95, p99).

System Metrics vs. Custom Application Metrics

• System Metrics: Ingested automatically by Google Cloud infrastructure without requiring agent installation or developer instrumentation (e.g., run.googleapis.com/container/cpu/utilizations, compute.googleapis.com/instance/disk/read_bytes_count). Many built-in Google Cloud metrics are non-chargeable; agent, user-defined, and Prometheus metrics follow their own pricing rules.
• Custom Application Metrics: User-defined metrics published under the custom.googleapis.com/ namespace. Applications publish custom metrics via:
  1. Google Cloud Managed Service for Prometheus (GMP): Standard /metrics Prometheus scraping using GKE PodMonitoring custom resources.
  2. OpenTelemetry Cloud Monitoring Exporter: Streaming application metrics directly via the OTel Metrics SDK.
  3. Cloud Monitoring API: Explicitly creating metric descriptors and publishing via projects.timeSeries.create.

The High-Cardinality Metric Anti-Pattern

A critical exam concept is avoiding high cardinality in custom metric labels. In Cloud Monitoring, a separate, persistent time series is allocated for every unique combination of label values.

## HIGH CARDINALITY DISASTER (DO NOT DO THIS):
custom.googleapis.com/orders/processed
Labels:
  - environment: "production"     # Cardinality: 2 (prod, staging)
  - payment_method: "credit_card"  # Cardinality: 4 (card, paypal, apple_pay, crypto)
  - user_id: "USER-8492019"       # Cardinality: 10,000,000 unique users!

Total Time Series Created = 2 * 4 * 10,000,000 = 80,000,000 Time Series!

Consequences of High Cardinality

1. Quota Exhaustion: Projects enforce strict quotas on the number of active time series (limits depend on metric and monitored-resource dimensions; check the current quota table). Exceeding this quota results in HTTP 429 quota exhaustion errors, dropping subsequent metric writes.
2. Exorbitant Monitoring Costs: Billing depends on the metric ingestion path and current pricing tiers; custom-byte and Prometheus-sample pricing are different. Creating millions of time series generates thousands of dollars in unintended metric ingestion charges.
3. Query Degradation: Dashboards and alerting policies time out attempting to aggregate across millions of individual series.

Developer Best Practice: Metric labels must have low, bounded cardinality (e.g., HTTP status code class 2xx/4xx/5xx, cloud region, deployment environment). High-cardinality attributes like user_id, order_id, ip_address, or request_url belong exclusively in Cloud Logging structured payload fields or Cloud Trace span attributes, never in Cloud Monitoring metric labels.

---

Dashboards and Visualization: Heatmaps for Latency Percentiles

The Fallacy of the Average (Mean)

A common production pitfall is monitoring API latency using arithmetic mean (sum(duration) / count). If 99 requests take 10ms and 1 request takes 2,000ms, the average latency is ~30ms, completely hiding the fact that 1% of users are experiencing unacceptable timeouts. Production latency must be tracked using percentiles: p50 (median), p95, and p99.

Heatmaps for DISTRIBUTION Metrics

• Line charts of p99 latency can still obscure multimodal latency distributions. A Heatmap chart visualizes the full underlying DISTRIBUTION metric.
• The X-axis represents time, the Y-axis represents latency bucket boundaries (logarithmic or linear), and the color saturation/darkness of each pixel indicates the volume of requests falling into that specific bucket.
• Diagnosing Issues via Heatmaps:
  • A solid dark band across the bottom indicates consistent low-latency requests.
  • A faint secondary band running higher up indicates a dual-path execution problem (e.g., database cache hits at 10ms vs. unindexed queries at 800ms).
  • Vertical streaks indicate momentary system-wide stalls, such as stop-the-world JVM garbage collection cycles or pod evictions.

---

Time Series Alignment and Reduction Pipeline

Raw telemetry points arrive at irregular intervals from hundreds of distributed container instances. Before Cloud Monitoring can plot a metric on a chart or evaluate an alerting condition, the time series must pass through a two-stage aggregation pipeline:

[Raw Asynchronous Points from Multiple Container Instances]
                            │
                            ▼
[Stage 1: Temporal Alignment (Per-Series Normalization)]
  • Partitions time into uniform intervals (Alignment Period, e.g., 1m, 5m)
  • Applies an Aligner function to condense points within each interval:
    - ALIGN_RATE: Converts CUMULATIVE counters into per-second rates (req/sec)
    - ALIGN_DELTA: Computes net change across the interval
    - ALIGN_MEAN: Computes average value for GAUGE metrics (CPU, Memory)
    - ALIGN_PERCENTILE_99: Extracts 99th percentile from DISTRIBUTION
                            │
                            ▼
[Stage 2: Spatial Reduction (Cross-Series Aggregation)]
  • Combines multiple normalized time series across instances into fewer series
  • Applies a Reducer function:
    - REDUCE_SUM: Adds values across instances (Total cluster requests/sec)
    - REDUCE_MEAN: Computes cluster-wide average utilization
    - REDUCE_MAX: Tracks peak utilization across any single node
  • Group-By: Groups resulting series by specific labels (e.g., response_code)
                            │
                            ▼
[Evaluation: Alerting Policy Threshold / Dashboard Line Chart]

Alerting Policies: Thresholds, Metric Absence, MQL and PromQL

An Alerting Policy defines the conditions under which Cloud Monitoring generates incidents and dispatches notifications.

Condition Types

1. Metric Threshold Condition: Triggers when an aligned and reduced metric value crosses a threshold (e.g., HTTP 5xx error rate > 5%). To prevent false alarms from momentary 10-second traffic bursts, policies enforce an evaluation duration window (e.g., "condition must be met for 5 consecutive minutes").
2. Metric Absence Condition: Triggers when expected telemetry stops arriving entirely for a specified duration. Crucial for monitoring asynchronous background workers, cron jobs, or IoT devices: if a message processing worker fails to report a heartbeat metric for 10 minutes, an absence alert fires immediately.

PromQL and Existing MQL Assets

Use the query builder or PromQL for new Monitoring charts and alerts. MQL is no longer recommended; since July 22, 2025, new MQL charts and alert policies cannot be created through the console, although existing assets and API workflows remain supported.

For example, a PromQL condition for a bounded-cardinality application counter exported through Managed Service for Prometheus can compare the error rate to the total rate:

sum(rate(checkout_requests_total{status_class="5xx"}[5m]))
/
sum(rate(checkout_requests_total[5m])) > 0.05

Configure the condition's retest/evaluation duration separately (for example five minutes), select the intended service/environment, and define behavior for no traffic and missing data. The five-minute rate window is not itself a five-minute sustained-breach requirement. For fleet latency, combine compatible histogram buckets before estimating a percentile; averaging instance p99 values does not produce the fleet p99.

---

Notification Channels and Global Uptime Checks

Notification Channels and Automated Remediation

When an alerting policy condition triggers an incident, Cloud Monitoring routes notifications to configured channels:

• Standard Channels: Email, SMS, Slack, PagerDuty, Webhooks.
• Automated Remediation via Pub/Sub: Alerts can publish directly to a Cloud Pub/Sub topic. This enables event-driven remediation: an alert message triggers a Cloud Function or Cloud Run service to restart a failing process, purge an overloaded Memorystore cache, or spin up emergency compute capacity without requiring human operator intervention.

Global Uptime Checks

Internal health checks (such as Kubernetes liveness/readiness probes) only confirm that a container process is running inside its local VPC. They cannot verify whether external customers can reach your service through public DNS, Edge Points of Presence (PoPs), and Cloud Load Balancers.

Global Uptime Checks resolve this by executing external synthetic HTTP/HTTPS/TCP probes from multiple geographic regions simultaneously (e.g., US East, US West, Europe West, Asia Pacific):

• Probing Frequency: Choose a supported check interval, such as 1, 5, 10, or 15 minutes, and verify the current API options.
• Validation Criteria: Checks assert expected HTTP status codes (e.g., 200 OK), response body string/regex matching, and SSL/TLS certificate expiration (dispatching alerts e.g., 30 days before a certificate expires).
• Resilience: Configure the alert aggregation/threshold to require failures from the chosen number of checker locations; this is not an unconditional default, preventing false alarms caused by localized transit ISP anomalies.

Official references checked October 18, 2026: Google Cloud 1.