17.2 Log Routing, Export Destinations, and Exclusions

Log Router & Log Sinks Architecture

Every log entry emitted by an application or Google Cloud service enters the Log Router. The Log Router evaluates the entry against defined sink filters and exclusion rules before deciding where the log should be delivered.

                  +-------------------------------------+
                  |       Application Log Entry         |
                  |         (stdout / JSON)             |
                  +-------------------------------------+
                                     |
                                     v
                  +-------------------------------------+
                  |              Log Router             |
                  |  - Evaluates inclusion sink filters |
                  |  - Evaluates bucket exclusion rules |
                  +-------------------------------------+
                     /        |             |        \
                    /         |             |         \
        +----------+    +-----------+  +----------+  +-------------------+
        | Cloud    |    |  BigQuery |  |  Pub/Sub |  | Cloud Logging     |
        | Storage  |    |  (Log     |  |  (SIEM / |  | Buckets           |
        | (Archive)|    | Analytics)|  | Splunk)  |  | (_Default / _Req) |
        +----------+    +-----------+  +----------+  +-------------------+

Sink Destinations and Use Cases

Log Router sinks route entries to supported destinations, with destination-specific batching and delivery latency:

1. Cloud Storage (storage.googleapis.com/[BUCKET_NAME]):
   • Primary Use Case: Long-term archival and compliance retention (e.g., preserving audit logs for an explicitly applicable retention requirement).
   • Operational Profile: Batched JSON export; delivery is not immediate. Select the storage class from retention, access frequency and retrieval costs.
2. BigQuery (bigquery.googleapis.com/projects/[PROJECT_ID]/datasets/[DATASET_ID]):
   • Primary Use Case: SQL export analysis and ad-hoc querying, business intelligence dashboards, and cross-referencing system logs with business database tables.
   • Operational Profile: Streaming export into the configured table organization; date-sharded or partitioned behavior and clustering are not automatic universal defaults.
3. Pub/Sub (pubsub.googleapis.com/projects/[PROJECT_ID]/topics/[TOPIC_NAME]):
   • Primary Use Case: Real-time event streaming to third-party Security Information and Event Management (SIEM) systems (such as Splunk, Datadog, Chronicle, or Elastic) and event-driven automation via Cloud Functions.
   • Operational Profile: Streaming delivery with variable latency of individual log message payloads.
4. Another Google Cloud Project: Centralizing logs across hundreds of spoke projects into a dedicated centralized security logging project.
5. Cloud Logging Log Bucket: Dedicated log buckets with customized retention periods (up to 3,650 days) and regional residency constraints.

Creating Sinks via the gcloud CLI

When a sink is created, Cloud Logging generates a dedicated service account (writer identity) for that sink. You must grant this service account appropriate IAM write permissions on the destination resource:

## Create a sink routing critical errors to Pub/Sub
gcloud logging sinks create critical-errors-sink \
    pubsub.googleapis.com/projects/my-prod-project/topics/security-alerts \
    --log-filter="severity >= CRITICAL"

## Output displays: writerIdentity: serviceAccount:p12345678-9012@gcp-sa-logging.iam.gserviceaccount.com

## Grant the sink service account permission to publish to the target Pub/Sub topic
gcloud pubsub topics add-iam-policy-binding security-alerts \
    --member="serviceAccount:p12345678-9012@gcp-sa-logging.iam.gserviceaccount.com" \
    --role="roles/pubsub.publisher"

---

Log Exclusion Rules & Cost Management

Cloud Logging bills based on log ingestion volume beyond a monthly free tier (first 50 GiB per project). In high-throughput microservices, health-check endpoints, load balancer probes, and debug logging can generate terabytes of benign telemetry, causing runaway cloud bills.

Mechanics of Log Exclusions

• Exclusion Filters: Define criteria to prevent matched logs from being ingested into a log bucket (_Default or custom bucket).
• Percentile Exclusion (Sampling): Exclusion rules can drop a percentage of matching logs (e.g., exclude 99% of successful HTTP 200 health check logs while retaining 1% for statistical baseline tracking).
• Sink Decoupling: Sinks route logs independently of bucket exclusions. If you create an export sink to BigQuery for all application logs, and configure an exclusion filter on the _Default bucket for debug logs, the debug logs are still delivered to BigQuery while incurring zero Cloud Logging bucket storage costs.

Example: Excluding Health Check Telemetry

## Exclude 100% of successful HTTP 200 GET requests to /healthz
gcloud logging sinks update _Default \
    --add-exclusion="name=drop-health-probes,filter=resource.type=\"cloud_run_revision\" AND httpRequest.requestUrl:\"/healthz\" AND httpRequest.status=200,description=Drop benign health check logs"

---

Architectural Scenarios and Exam Traps

Scenario A: End-to-End Log and Trace Correlation

• Problem: An operations team uses Cloud Trace to detect latency spikes. When clicking on an anomalous span, the "Logs" tab in the Trace details pane displays "No logs found for this trace," even though the service logged extensive details.
• Root Cause: The application populated logging.googleapis.com/trace with only the raw 32-character hexadecimal trace ID (4bf92f3577b34da6a3ce929d0e0e4736).
• Resolution: Update the logging library to format the trace field with the full resource string: projects/[PROJECT_ID]/traces/[TRACE_ID]. Cloud Logging requires this full path to establish the relational link in Cloud Trace.

Scenario B: Multi-Region Compliance Audit Archive

• Requirement: Regulators mandate that all administrative access and data mutation logs be preserved in an immutable, tamper-proof state for 7 years at the lowest achievable storage cost. Real-time querying is not required.
• Resolution: Create a Log Router sink with the filter logName:"cloudaudit.googleapis.com". Route the sink to a Cloud Storage bucket configured with Archive Storage Class and a Bucket Lock (Object Retention) policy enforcing a 7-year retention period.

Scenario C: Preventing Health Check Ingestion Overruns

• Exam Trap: Developers often attempt to suppress health check logs by adding application-level logic to write health logs to a local disk file instead of stdout. Cloud Run's writable container filesystem is memory-backed; ordinary GKE container filesystems are not universally tmpfs. Local files can still exhaust memory or ephemeral storage and are not a durable logging system. On Cloud Run, unbounded writes consume RAM, eventually causing an Out-Of-Memory (OOMKilled) container crash. The correct Google Cloud architectural solution is writing to stdout and applying a Log Exclusion Filter at the Log Router.