10.2 Observability: Cloud Monitoring, Logging & Alert Policies

Observability: Cloud Monitoring, Logging & Alert Policies

Architectural Objective: Observability is the prerequisite for operational excellence, high availability, and rapid incident triage. A Google Professional Cloud Architect must design comprehensive observability pipelines using Google Cloud Monitoring and Cloud Logging: selecting between agentless metrics and the Google Cloud Ops Agent, implementing custom and log-based metrics, architecting Log Router sinks and exclusion filters, enabling BigQuery-powered Log Analytics, and building proactive, anomaly-aware alerting policies.

---

Cloud Monitoring Architecture & Telemetry Ingestion

Google Cloud Monitoring (formerly Stackdriver) is a fully managed, hyper-scalable telemetry and observability service that ingests metrics, events, and metadata across Google Cloud, hybrid on-premises environments, and multicloud infrastructure.

+-----------------------------------------------------------------------------------+
|                        CLOUD MONITORING METRICS ARCHITECTURE                      |
+-----------------------------------------------------------------------------------+
| 1. AGENTLESS METRICS     | Automatically collected by GCP hypervisors & APIs.     |
| (Platform Layer)         | CPU utilization, network I/O, disk throughput, LB QPS. |
+--------------------------+---------------------------------------------------------+
| 2. GOOGLE CLOUD OPS AGENT| Installed inside VM Guest OS (Linux / Windows).         |
| (Host & Application)     | Guest RAM usage, disk swap, processes, Apache, MySQL.  |
+--------------------------+---------------------------------------------------------+
| 3. CUSTOM & OTel METRICS | Application-level metrics pushed via Monitoring API,    |
| (Software Layer)         | OpenTelemetry Collector, or Managed Service for Prom.   |
+-----------------------------------------------------------------------------------+

Time Series Data Model

Cloud Monitoring stores all metric data points as Time Series. Each time series consists of:

• Monitored Resource: The specific cloud asset emitting telemetry (e.g., gce_instance, k8s_pod, cloudsql_database).
• Metric Descriptor: The metric type name (e.g., compute.googleapis.com/instance/cpu/utilization), metric kind (GAUGE, DELTA, CUMULATIVE), and value type (BOOL, INT64, DOUBLE, DISTRIBUTION).
• Labels: Key-value metadata pairs providing dimensionality (e.g., instance_id, zone, response_code).
• Data Points: Timestamped numeric values.

Agentless Ingestion vs. Google Cloud Ops Agent

[!WARNING] Exam Trap: Compute Engine VM instances do not report Guest OS Memory (RAM) utilization or filesystem disk space to Cloud Monitoring by default. If an exam scenario requires autoscaling Managed Instance Groups (MIGs) or triggering alerts based on memory utilization, you must install the Google Cloud Ops Agent.

Custom Metrics & Log-Based Metrics

When standard platform and Ops Agent metrics are insufficient to track business domain logic or application state, architects employ Custom Metrics and Log-Based Metrics.

+-----------------------------------------------------------------------------------+
|                         LOG-BASED METRICS ARCHITECTURE                            |
+-----------------------------------------------------------------------------------+
| Application writes JSON log: {"status": 500, "latency_ms": 342, "user": "u89"}      |
|                                          │                                        |
|                                          v                                        |
| [ Cloud Logging Ingestion Pipeline ] ──> [ Log-Based Metric Evaluator ]           |
|                                                      │                            |
|                      ┌───────────────────────────────┴─────────────────────────┐  |
|                      ▼                                                         ▼  |
|          [ COUNTER METRIC ]                                       [ DISTRIBUTION ]|
|  Counts occurrences matching filter:                     Extracts numerical value |
|  `jsonPayload.status = 500`                              `jsonPayload.latency_ms` |
|  Produces metric: `custom.googleapis.com/errors_500`     Produces p50/p95/p99 dist|
+-----------------------------------------------------------------------------------+

1. Custom Metrics

• Ingestion: Emitted directly from application code using the Cloud Monitoring API (MetricServiceClient), OpenTelemetry SDKs, or the Google Cloud Managed Service for Prometheus (GMP).
• Cardinality Governance: High cardinality (e.g., placing unique User IDs or UUID transaction tokens into metric labels) creates millions of independent time series, causing massive ingestion costs and degraded dashboard query performance. User IDs must be placed in structured logs, never metric labels.

2. Log-Based Metrics: Counter vs. Distribution

Log-based metrics bridge Cloud Logging and Cloud Monitoring, extracting real-time time series from log entries as they pass through the ingestion pipeline:

• Counter Metrics: Increments a count each time a log entry matches a designated Cloud Logging query filter (e.g., resource.type="k8s_container" AND severity>=ERROR). Useful for counting application errors, login attempts, or checkout events.
• Distribution Metrics: Extracts a numeric field from structured JSON log payloads (e.g., extracting jsonPayload.processing_time_ms) and records the values in a statistical distribution histogram. Enables calculating p50, p90, p95, and p99 percentiles over time for alerting and SLO tracking.

---

Synthetic Monitoring: Uptime Checks & Broken Link Probers

While metrics reflect internal system state, Synthetic Monitoring simulates external user traffic to evaluate external reachability, SSL certificate validity, and endpoint responsiveness.

+-----------------------------------------------------------------------------------+
|                         SYNTHETIC MONITORING ARCHITECTURE                         |
+-----------------------------------------------------------------------------------+
| Probers located across 6 Global Regions (USA, Europe, Asia-Pacific, etc.)         |
|                 │                                                                 |
|                 +───> [ Global HTTPS Uptime Check ] ──> Public API Endpoint       |
|                 │     (Validates HTTP 200, Latency < 500ms, SSL Expiry > 14 days) |
|                 │                                                                 |
|                 +───> [ Private Uptime Check ] ───────> Internal VPC Endpoint     |
|                       (Via VPC Network / PSC)           (Internal Microservice)   |
+-----------------------------------------------------------------------------------+

• Global Uptime Checks: Google-managed probing servers located in multiple geographic regions periodically send HTTP, HTTPS, or TCP requests to public IP endpoints, URLs, App Engine apps, or Load Balancers. An alert triggers if multiple distinct geographic probing locations report failure simultaneously.
• Private Uptime Checks: Validates internal, non-public endpoints within a customer VPC network (such as internal load balancers or private GKE clusters) routed securely without traversing the public internet.
• Custom Synthetic Probers: Cloud Functions-based synthetic probers (using Puppeteer or Mocha) that execute multi-step synthetic user flows (e.g., logging in, adding an item to cart, and submitting a test order).

---

Cloud Monitoring Alerting Policies: Conditions & Notification

Alerting policies notify operations teams when systems exhibit abnormal behavior, risk breaching SLOs, or experience resource exhaustion.

+-----------------------------------------------------------------------------------+
|                         ALERTING POLICY ANATOMY                                   |
+-----------------------------------------------------------------------------------+
| 1. CONDITION        | What triggers the alert? (Threshold, Absence, Forecast,     |
|                     | Anomaly, PromQL, MQL Query).                                |
+---------------------+-------------------------------------------------------------+
| 2. AGGREGATION &    | How data points are grouped and aligned over time           |
|    ALIGNMENT        | (Alignment Period: 1m, 5m; Reducer: MEAN, SUM, PERCENTILE). |
+---------------------+-------------------------------------------------------------+
| 3. TRIGGER RULE     | Percentage of instances or number of time series violating |
|                     | condition (e.g., "Any time series violates for 5 minutes"). |
+---------------------+-------------------------------------------------------------+
| 4. NOTIFICATIONS    | PagerDuty, Slack, Email, Webhook (Cloud Run/Functions),     |
|                     | Cloud Pub/Sub Topic (Automated Event-Driven Remediation).   |
+---------------------+-------------------------------------------------------------+
| 5. DOCUMENTATION    | Markdown runbook instructions & troubleshooting links.      |
+-----------------------------------------------------------------------------------+

Advanced Alert Condition Types

Alert Snoozing & Mute Rules

To prevent alert storms during scheduled maintenance windows, database patching, or network upgrades, architects configure Mute Rules (Alert Snoozing). Mute rules suppress notification delivery for matching alert policies during a specified maintenance window while continuing to record incidents in Cloud Monitoring for historical auditing.

---

Cloud Logging Architecture: Ingestion, Storage & Sinks

Google Cloud Logging is a centralized, real-time log management service capable of ingesting exabytes of structured JSON, text, and binary logs across an enterprise footprint.

+-----------------------------------------------------------------------------------+
|                        CLOUD LOGGING INGESTION & ROUTER FLOW                      |
+-----------------------------------------------------------------------------------+
| [ Log Producers: GKE, Compute Engine, Cloud Run, Audit Logs, Custom Apps ]        |
|                                          │                                        |
|                                          v                                        |
|                             [ CLOUD LOG ROUTER ]                                  |
|                                          │                                        |
|             ┌────────────────────────────┼────────────────────────────┐           |
|             ▼                            ▼                            ▼           |
|   [ EXCLUSION FILTERS ]         [ DEFAULT SINK ]             [ EXPORT SINKS ]     |
|   Drops high-volume debug       Routes to `_Default`         Routes filtered logs |
|   logs before ingestion.        Log Bucket (30-day).         to enterprise targets|
|   (Zero Ingestion Cost)                  │                            │           |
|                                          v                            │           |
|                                [ LOG ANALYTICS ]                      │           |
|                                (BigQuery SQL queries                  │           |
|                                 directly on bucket)                   │           |
|                                                                       │           |
|             ┌─────────────────────────────────────────────────────────┴─────────┐ |
|             ▼                            ▼                            ▼         ▼ |
|     [ Cloud Storage ]             [ BigQuery ]                 [ Pub/Sub ]  [Other]|
|     Long-term compliance          Real-time SQL analytics      SIEM export  (GCP   |
|     cold archive (years).         & machine learning.          (Splunk/Datadog)Proj|
+-----------------------------------------------------------------------------------+

1. Log Buckets & Default Retention

Cloud Logging stores ingested logs in specialized Log Buckets (distinct from Cloud Storage buckets):

• _Required Log Bucket: Automatically stores Admin Activity audit logs, System Event audit logs, and Access Transparency logs. Retained for 400 days at zero cost. Cannot be disabled, modified, or deleted.
• _Default Log Bucket: Automatically stores all other runtime logs (Data Access audit logs, GKE stdout/stderr, Compute Engine syslog, Cloud Run logs). Retained for 30 days by default (customizable from 1 to 3,650 days). Incur ingestion and storage fees.

2. Log Router: Exclusion Filters & Cost Optimization

The Log Router evaluates every log entry at the instant of ingestion against defined rules:

• Exclusion Filters: Drops unwanted, verbose logs (e.g., debug logs severity=DEBUG or high-frequency health checks httpRequest.requestUrl:"/healthz") before they enter log buckets. Excluded logs incur zero storage and zero ingestion costs.

3. Log Sinks & Export Destinations

Log Sinks stream filtered logs matching an inclusion filter to external destinations in real time:

Organization-Level Aggregated Sinks

In an enterprise with hundreds of Google Cloud projects under an Organization or Folder hierarchy, configuring log sinks per project creates massive administrative overhead. Architects create an Aggregated Log Sink at the Organization or Folder level with includeChildren = true. This automatically captures and centralizes all audit logs across current and future projects into a single security operations data lake.

---

Log Analytics Powered by BigQuery

Traditionally, running SQL queries on log data required exporting logs via a sink to BigQuery, paying for BigQuery streaming ingestion and duplicate storage. Log Analytics eliminates this redundancy by embedding a BigQuery SQL engine directly on top of Cloud Logging log buckets.

-- Direct SQL query executed inside Cloud Logging Log Analytics
SELECT
  timestamp,
  resource.type,
  json_payload.user_email,
  http_request.status,
  http_request.latency
FROM
  `my-project.global._Default._AllLogs`
WHERE
  http_request.status >= 500
  AND timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 24 HOUR)
ORDER BY
  timestamp DESC
LIMIT 100;

Architectural Advantages of Log Analytics

• Zero Ingestion Duplication: Query log data using ANSI BigQuery SQL directly where it resides inside the log bucket.
• BigQuery Studio & Looker Integration: Linked BigQuery datasets allow security analysts to query log buckets directly from the BigQuery console and visualize log trends in Looker Studio without ETL pipelines.
• Field-Level JSON Parsing: Queries nested JSON payloads (json_payload.request.headers.user_agent) using standard JSON SQL operators.

---

Concrete Architectural Scenario: Multi-Project Enterprise Observability

Scenario Profile

• Organization: Global retail enterprise operating 120 GCP projects partitioned across Development, Staging, and Production folders.
• Requirements: Centralized security audit repository; 7-year compliance archiving for financial transactions; real-time PagerDuty alerting for API errors; automated exclusion of high-volume non-critical health check logs to reduce monthly cloud spend.

[ Enterprise GCP Organization ]
  ├── Project: eCommerce-Prod-01 ──┐
  ├── Project: eCommerce-Prod-02 ──┼──> [ Aggregated Org Sink (includeChildren=true) ]
  └── Project: Payments-Prod-01  ──┘            │
                                                ├──> [ Pub/Sub ] ──> Corporate SIEM
                                                ├──> [ GCS Bucket ] (7-Yr Lock Archive)
                                                └──> [ Central Security Log Bucket ]
                                                          └── (Log Analytics Enabled)

Architecture Blueprint

1. Metrics Collection: Compute Engine VMs running the Google Cloud Ops Agent to stream memory, disk swap, and application logs into Cloud Monitoring.
2. Cost Optimization via Log Exclusions: Ingestion exclusion filter deployed on _Default bucket: jsonPayload.request_path="/healthz" OR severity=DEBUG, eliminating 45% of unnecessary log ingestion costs.
3. Centralized Log Aggregation: Organization-level Aggregated Log Sink (includeChildren=true) routing all cloudaudit.googleapis.com logs to a centralized Security-Audit-Project.
4. Compliance Archiving & SIEM Integration: Sinks configured to route financial audit logs to a Cloud Storage bucket with Bucket Lock (Object Retention) for 7 years, and streaming to Pub/Sub for ingestion by Chronicle SIEM.
5. Intelligent Alerting: Cloud Monitoring alerting policies with Metric Forecast conditions predicting disk saturation 48 hours in advance, and Multi-Window Multi-Burn-Rate policies paging on-call engineers via PagerDuty.

---

[!IMPORTANT] Exam Watch: On the PCA exam, if a question asks how to collect guest OS memory utilization or disk space from Compute Engine VMs, the answer is always the Google Cloud Ops Agent. If a question asks how to aggregate audit logs across an entire enterprise organization into a central project at minimal operational overhead, choose an Organization-level Aggregated Log Sink with includeChildren = true.