12.2 Observability: Cloud Monitoring, Cloud Logging, and Alerting

12.2 Observability: Cloud Monitoring, Cloud Logging, and Alerting

Quick Answer: End-to-end observability across Google Cloud data systems relies on the Cloud Operations Suite (formerly Stackdriver). For streaming pipelines, Cloud Monitoring tracks Cloud Pub/Sub subscription/num_undelivered_messages (backlog volume) and subscription/oldest_unacked_message_age (subscriber health and poison pills), alongside Cloud Dataflow job/system_lag (watermark delay). For analytical and storage layers, monitoring tracks BigQuery slots/allocated_slots (slot starvation) and Bigtable server/latencies (p99 latency spikes indicating tablet hotspotting). Structured JSON logging via Cloud Logging enables rapid querying with Logging Query Language (LQL). To optimize costs, Log Router Sinks export logs to Cloud Storage (7-year compliance archives), BigQuery Log Analytics (SQL-based security analytics), or Pub/Sub (automated remediation), while Log-Based Metrics convert custom application error patterns into actionable time-series alerts.

---

The Observability Triad in Decoupled Cloud Data Platforms

Modern enterprise cloud architectures decouple ingestion (Cloud Pub/Sub), stream processing (Cloud Dataflow), orchestration (Cloud Composer), storage (Cloud Bigtable, Cloud Storage), and data warehousing (BigQuery). While this decoupling provides limitless elasticity and independent scaling, it introduces operational opacity: a delayed dashboard in Looker could originate from subscriber starvation in Pub/Sub, key skew in a Dataflow shuffle stage, slot quota exhaustion in BigQuery, or row key hotspotting in Bigtable.

Achieving true observability across these distributed components requires instrumenting the Observability Triad:

1. Metrics (Cloud Monitoring): Numerical time-series representing platform health, resource saturation, and throughput aggregated over temporal intervals.
2. Logs (Cloud Logging): Structured contextual records capturing discrete events, stack traces, schema errors, and operational state transitions.
3. Traces and Audits (Cloud Trace & Cloud Audit Logs): Request propagation paths across service boundaries, tracking latency attribution and data access provenance.

+-------------------------------------------------------------------------+
|                    DATA SERVICES TELEMETRY SUITE                        |
|                                                                         |
|  [Cloud Pub/Sub]  --> subscription/num_undelivered_messages             |
|                       subscription/oldest_unacked_message_age           |
|                                                                         |
|  [Cloud Dataflow] --> job/system_lag (watermark delay)                  |
|                       job/estimated_backlog_processing_time             |
|                       job/elements_produced_count                       |
|                                                                         |
|  [BigQuery]       --> slots/allocated_slots vs total_available_slots    |
|                       query/execution/time                              |
|                                                                         |
|  [Cloud Bigtable] --> server/latencies (p95, p99 read/write)            |
|                       server/cpu_load (cluster utilization)             |
+-------------------------------------------------------------------------+

---

Core Cloud Monitoring Metrics for Google Cloud Data Services

Cloud Monitoring collects platform-level time-series metrics from Google Cloud services automatically at 1-minute intervals (with certain high-resolution metrics available at 10-second intervals).

1. Cloud Pub/Sub Telemetry

Pub/Sub serves as the elastic ingestion buffer for streaming architectures. Two primary metrics reveal subscriber health:

• subscription/num_undelivered_messages: The total number of messages that have been published to a topic but not yet acknowledged by subscribers. A sustained upward trend indicates subscriber starvation, consumer worker crashes, or an ingress rate exceeding consumer processing capacity.
• subscription/oldest_unacked_message_age: The elapsed time (in seconds) since the oldest unacknowledged message was published. This is the definitive indicator of subscriber freshness. If num_undelivered_messages is relatively low but oldest_unacked_message_age climbs continuously, a "poison-pill" message is repeatedly causing consumer worker exceptions, timing out, and being redelivered.
• subscription/ack_message_operation_count: The volume of acknowledgment requests returned by subscriber clients. Comparing this against topic/send_request_count confirms whether consumption is keeping pace with production.

2. Cloud Dataflow Telemetry

Dataflow stream and batch processing pipelines expose internal execution metrics from the Apache Beam runner:

• job/system_lag: The current maximum difference (in seconds) between the pipeline's event-time watermark and the current wall-clock processing time. In healthy streaming pipelines, system_lag remains flat and low (typically 5–30 seconds). A linearly increasing system_lag indicates that workers cannot advance the watermark due to stuck processing, worker memory thrashing, or upstream partition starvation.
• job/estimated_backlog_processing_time: The estimated time (in seconds) required for the current worker VM pool to process outstanding backlog data. Dataflow's Autoscaler monitors this metric directly to determine when to provision additional worker VMs.
• job/elements_produced_count: The cumulative count of PCollection elements produced by a specific pipeline transform. Plotting this across consecutive steps identifies bottlenecks: if Step A produces 100,000 elements/sec but Step B emits only 100 elements/sec, Step B is performing expensive transformations, un-indexed lookups, or suffering from severe data skew.

3. BigQuery Telemetry

BigQuery decouples execution slots (Borg containers) from Capacitor columnar storage. Monitoring focuses on slot saturation and execution queuing:

• slots/allocated_slots vs. slots/total_available_slots: Measures slot consumption within an organization's capacity reservation or dynamic autoscaling reservation pool. When allocated_slots approaches total_available_slots (100% saturation), new incoming queries are placed into a pending execution queue, increasing end-to-end latency.
• query/execution/time: The distribution of query elapsed execution times. Spikes in execution time without corresponding increases in scanned data bytes indicate slot throttling, partition pruning failures, or un-optimized join topologies.
• job/num_in_flight_queries: The instantaneous count of concurrently running queries. Exceeding project-level concurrency limits pushes jobs into queued states.

4. Cloud Bigtable Telemetry

Bigtable delivers single-digit millisecond latency at petabyte scale. Monitoring must catch hot tablets and resource saturation early:

• server/latencies: Tracks read and write request latencies broken down by percentiles (p50, p95, p99). SREs focus on the p99 latency: a jump in p99 write latency while p50 remains flat indicates tablet hotspotting (traffic concentrating on a single storage node due to poor row key design).
• server/cpu_load: The CPU utilization of the Bigtable cluster. Recommended operational thresholds:
  • Multi-cluster routing (High Availability): Target sustained CPU $\le 70%$. If one zone or cluster suffers an outage, the remaining cluster must have sufficient headroom to absorb 100% of the diverted traffic without failing.
  • Single-cluster instances: Target sustained CPU $\le 80%$.
• server/modified_rows_rate: Tracks write ingestion throughput in rows per second.

---

Core Telemetry Reference for GCP Data Workloads

---

Structured Logging and Log Router Sinks

Platform metrics indicate that an issue exists; structured logs explain why it occurred. In production data engineering, emitting raw unstructured text logs is an anti-pattern because automated log parsing engines cannot query unindexed text at scale.

1. Structured JSON Logging

Dataflow pipelines, Cloud Functions, and Composer tasks should emit structured JSON to stdout or stderr. The Google Cloud Logging agent automatically parses structured JSON into first-class searchable fields under the jsonPayload object:

{
  "severity": "ERROR",
  "message": "Pipeline transformation failure: schema mismatch on financial payload",
  "timestamp": "2026-09-14T16:00:00.123456Z",
  "logging.googleapis.com/labels": {
    "environment": "production",
    "pipeline_name": "payment-settlement-stream"
  },
  "jsonPayload": {
    "pipeline_id": "df-settle-prod-9942",
    "step_name": "ValidateAccountBalance",
    "error_code": "ERR_NEGATIVE_BALANCE",
    "record_id": "rec_8831920",
    "account_id": "acc_4021",
    "stack_trace": "com.company.data.InvalidBalanceException: Account balance cannot be negative..."
  }
}

2. Logging Query Language (LQL)

Engineers query logs across thousands of worker VMs using the Logging Query Language in the Cloud Logging Logs Explorer:

-- Filter for Dataflow worker errors on a specific streaming pipeline
resource.type="dataflow_step"
resource.labels.job_id="2026-09-14_08_00_00-1234567890"
severity>=ERROR
jsonPayload.error_code="ERR_NEGATIVE_BALANCE"

3. Log Router Sinks and BigQuery Log Analytics

Retaining all operational logs inside Cloud Logging is cost-prohibitive for high-throughput streaming systems that emit millions of log lines per second. Log Router Sinks intercept incoming logs and route them to external destinations based on an inclusion filter, completely bypassing Cloud Logging storage fees if excluded from the _Default bucket:

• BigQuery Sink / Log Analytics: Routes audit trails, error records, and pipeline telemetry directly into BigQuery. When linked with BigQuery Log Analytics, engineers can query petabytes of real-time logs using standard SQL, joining log events directly with analytical data tables.
• Cloud Storage Sink: Exports raw logs in compressed JSON batches to Cloud Storage (using Nearline, Coldline, or Archive storage classes) for long-term regulatory compliance (e.g., 7-year audit retention).
• Cloud Pub/Sub Sink: Routes real-time error logs to a Pub/Sub topic to trigger automated self-healing workflows (such as invoking Cloud Functions to restart a dead consumer or post rich alerts to incident management webhooks).

                             Log Router Architecture
                             
    [Dataflow Workers] --+
    [Pub/Sub Telemetry] -+---> [Log Router] 
    [BigQuery Audits]  --+          |
                                    +---> [BigQuery / Log Analytics] (SQL Analysis)
                                    +---> [Cloud Storage] (Compliance Retention)
                                    +---> [Pub/Sub Topic] (Auto-Remediation)
                                    +---> [_Default Log Bucket] (Logs Explorer)

---

Log-Based Metrics

Often, specialized application-level failures (e.g., specific validation rejection codes or deserialization errors) are recorded in logs but are not tracked by Google Cloud's default system metrics. Log-based metrics convert log data into time-series metrics in Cloud Monitoring:

1. Counter Metrics: Increments a numerical counter each time a log entry matching an inclusion filter arrives. For example, counting occurrences of jsonPayload.error_code="ERR_SCHEMA_VIOLATION".
2. Distribution Metrics: Extracts a continuous numerical value from a structured JSON log field (e.g., jsonPayload.parsing_latency_ms or jsonPayload.batch_byte_size) and tracks statistical distributions (p50, p95, p99, mean, variance) over time.

Once created, log-based metrics can be visualized on Cloud Monitoring dashboards and used directly in Alerting Policies.

---

Designing Resilient Alerting Policies

Alert fatigue is a primary operational hazard in SRE. When engineers receive hundreds of non-actionable emails or pages every week, critical outages are ignored. Production alerting policies must adhere to strict principles:

• Alert on Symptoms, Not Causes: Page on symptoms that directly violate customer SLOs (e.g., high data freshness latency or rising dead-letter rates) rather than temporary internal causes (e.g., single-node CPU spikes).
• Metric Absence Alerts: In streaming data, if an upstream source crashes completely, it stops emitting logs and metrics. A standard threshold alert (e.g., error_count > 5) will never fire because zero logs are emitted. Data engineers must configure Absence of Data Alerts (metric_absence) that trigger if zero records arrive over an expected 10-minute heartbeat window.
• Notification Routing & Escalation: Low-priority anomalies route to Slack or ticketing systems; urgent SLO burn rate violations route directly to on-call engineers via PagerDuty.