17.1 Structured Logs, Trace Correlation, and Queries

Structured Application Logging & Log Sinks with Cloud Logging

Centralized observability in enterprise microservices requires structured telemetry that machines can parse, index, and correlate at high scale. The Google Cloud Professional Cloud Developer exam tests your understanding of structured logging patterns, special reserved JSON fields recognized by the Cloud Logging agent, the Logging Query Language (LQL), and Log Router architectures—including export sinks to BigQuery, Cloud Storage, and Pub/Sub, as well as cost-control exclusion filters.

---

Structured Logging vs. Unstructured Text Logs

When applications write unstructured strings to standard output (stdout) or standard error (stderr), Cloud Logging ingests each line as a plain-text textPayload. In contrast, when an application serializes its log entries as JSON objects to stdout, the underlying logging agent (such as the Cloud Logging agent, Fluent Bit in GKE, or the Cloud Run container supervisor) automatically deserializes the entry into a typed jsonPayload.

The Operational Hazards of Unstructured Logs

Unstructured logging poses severe operational liabilities in production:

• Opaque Payloads: Filtering by user ID, tenant ID, or HTTP response latency requires expensive, brittle regular expression searches across raw string messages.
• Loss of Metric Granularity: Log-based metrics can extract values from text with regular expressions, but structured numeric fields are easier to maintain and less fragile.
• Lack of Trace Association: Distributed trace spans cannot be linked automatically to log lines, forcing engineers to manually search across thousands of log streams during production incidents.

Structured vs. Unstructured Comparison

---

Cloud Logging Special Reserved JSON Fields

Google Cloud Logging recognizes several top-level reserved fields in your JSON payload. When present, the logging pipeline strips or promotes these fields into first-class metadata properties on the resulting LogEntry resource rather than nesting them generically within jsonPayload.

Core Reserved Fields

1. severity: Specifies the log severity level. Valid values are DEFAULT, DEBUG, INFO, NOTICE, WARNING, ERROR, CRITICAL, ALERT, and EMERGENCY. Set severity explicitly; defaults and stream-based handling depend on the collector/runtime.
2. message: Supplies the primary human-readable log message. In the Cloud Logging Logs Explorer and Cloud Error Reporting consoles, the string in message serves as the primary headline and stack trace payload.
3. time or timestamp: Overrides the ingestion timestamp with the exact moment the event occurred at the application layer. Accepted formats include an RFC 3339 string (e.g., "2026-10-18T15:04:05.123456Z") or a JSON object with seconds and nanos ({"seconds": 1760800000, "nanos": 0}). If omitted, the timestamp defaults to the time the log was ingested by the Google Cloud collector.
4. logging.googleapis.com/trace: Correlates the log entry with a distributed trace in Cloud Trace. Crucial Exam Rule: The trace field must be formatted as the complete resource path: projects/[PROJECT_ID]/traces/[TRACE_ID]. Passing only [TRACE_ID] will prevent Cloud Trace and Cloud Logging from linking the records.
5. logging.googleapis.com/spanId: Identifies the specific sub-operation within the trace. Must be a 16-character hexadecimal string representing a 64-bit integer (e.g., "000000000000004a").
6. logging.googleapis.com/trace_sampled: A boolean (true or false) indicating whether the trace was sampled for storage in Cloud Trace.
7. logging.googleapis.com/sourceLocation: A JSON object pinpointing the exact source code origin. Contains file (string), line (integer or string), and function (string).
8. logging.googleapis.com/labels: A map of string key-value pairs added to LogEntry.labels, available as structured labels; indexing and cross-project access require appropriate configuration.
9. httpRequest: A nested object representing an HTTP transaction, adhering to Google's standard schema: requestMethod, requestUrl, status, userAgent, remoteIp, referer, latency, cacheLookup, and responseSize.

Concrete Example: Emitting Structured JSON to Standard Output

The following Go microservice handler demonstrates how to format and emit structured JSON to stdout without external third-party SDK dependencies, using an active OpenTelemetry span context. The legacy X-Cloud-Trace-Context span number is decimal and must not be copied unchanged into the hexadecimal spanId field:

// Handler excerpt: tracing middleware must already extract the incoming context.
// Imports: encoding/json, fmt, net/http, os, time, go.opentelemetry.io/otel/trace.
func orderHandler(w http.ResponseWriter, r *http.Request) {
    sc := trace.SpanContextFromContext(r.Context())
    entry := map[string]any{
        "severity": "INFO", "message": "Order accepted",
        "time": time.Now().UTC().Format(time.RFC3339Nano),
    }
    if sc.IsValid() {
        entry["logging.googleapis.com/trace"] = fmt.Sprintf(
            "projects/%s/traces/%s", os.Getenv("GOOGLE_CLOUD_PROJECT"), sc.TraceID().String())
        entry["logging.googleapis.com/spanId"] = sc.SpanID().String()
        entry["logging.googleapis.com/trace_sampled"] = sc.IsSampled()
    }
    if err := json.NewEncoder(os.Stdout).Encode(entry); err != nil {
        http.Error(w, "log serialization failed", http.StatusInternalServerError)
        return
    }
    w.WriteHeader(http.StatusAccepted)
}

---

Querying Telemetry with Logging Query Language (LQL)

The Logs Explorer uses the Logging Query Language (LQL) to filter, inspect, and analyze logs across services and resource boundaries.

LQL Syntax Fundamentals

• Exact Equality: resource.type = "cloud_run_revision"
• Field In-Hierarchy: jsonPayload.tenantId = "acme-corp"
• Text Has / Substring Matching: jsonPayload.message : "timeout" (matches any token containing "timeout")
• Regular Expression Matching: jsonPayload.userId =~ "^usr_[0-9]{5}$"
• Numerical and Severity Comparisons: severity >= WARNING, httpRequest.latency > "2.5s", httpRequest.status >= 500
• Logical Operators: AND, OR, and NOT (or -). When multiple clauses appear on separate lines, AND is evaluated implicitly.

Complex LQL Query Example

To find all failed transactions on Cloud Run for a specific tenant within a precise time window:

resource.type = "cloud_run_revision"
resource.labels.service_name = "checkout-service"
severity >= ERROR
jsonPayload.tenantId = "acme-corp"
timestamp >= "2026-10-18T10:00:00Z" AND timestamp <= "2026-10-18T12:00:00Z"
NOT jsonPayload.message : "circuit_breaker_test"