12.1 Cloud Native Observability (Logging, Metrics & Tracing)

12.1 Cloud Native Observability (Logging, Metrics & Tracing)

Quick Answer: Cloud-native observability provides actionable visibility into system state through three core pillars: Metrics (numeric time-series telemetry scraped by Prometheus for alerting and scaling), Logs (event text streams emitted to stdout/stderr and collected by DaemonSet agents like Fluent Bit), and Traces (distributed request paths tracked across microservices via OpenTelemetry and Jaeger). Together, these signals enable rapid detection, isolation, and root-cause analysis.

In traditional monolithic environments, monitoring typically involved checking server uptime and CPU utilization. However, in dynamic, highly distributed cloud-native architectures—where hundreds of ephemeral containers continuously scale, migrate, and fail across a cluster—traditional monitoring is insufficient. Observability measures how well a system's internal state can be inferred solely from its external outputs.

---

The Three Pillars of Cloud Native Observability

Observability relies on three distinct types of telemetry signals, often referred to as the "Three Pillars of Observability":

┌─────────────────────────────────────────────────────────────────────────────┐
│                    THE THREE PILLARS OF OBSERVABILITY                       │
├──────────────────────────┬──────────────────────────┬───────────────────────┤
│         METRICS          │           LOGS           │        TRACES         │
├──────────────────────────┼──────────────────────────┼───────────────────────┤
│ • Numeric Time-Series    │ • Timestamped Events     │ • End-to-End Request  │
│ • Aggregated & Efficient │ • Rich Text & Context    │   Journey Across Apps │
│ • Ideal for Alerting &   │ • Ideal for Debugging    │ • Ideal for Latency   │
│   Real-time Dashboards   │   Specific Errors        │   & Bottleneck Analysis│
│ • Tools: Prometheus      │ • Tools: Fluent Bit, Loki│ • Tools: Jaeger, OTel │
└──────────────────────────┴──────────────────────────┴───────────────────────┘

Observability Telemetry Comparison Matrix

---

Prometheus Time-Series Monitoring Architecture

Prometheus is a CNCF graduated project and the industry standard for monitoring containerized Kubernetes workloads.

┌─────────────────┐       Scrapes HTTP      ┌─────────────────────┐
│  Node Exporter  │◄────────────────────────┤                     │
└─────────────────┘        /metrics         │                     │
                                            │                     │       Query PromQL
┌─────────────────┐       Scrapes HTTP      │  Prometheus Server  │◄───────────────────────┐
│kube-state-metrics◄────────────────────────┤  (TSDB Datastore)   │                        │
└─────────────────┘        /metrics         │                     ├────────────────┐       │
                                            │                     │                │       │
┌─────────────────┐       Scrapes HTTP      │                     │                ▼       │
│ App Pod (/metrics)◄───────────────────────┤                     │       ┌────────────────┴┐
└─────────────────┘        /metrics         └──────────┬──────────┘       │Grafana Dashboard│
                                                       │                  └─────────────────┘
                                                       │ Triggers Alerts
                                                       ▼
                                            ┌─────────────────────┐
                                            │    Alertmanager     │
                                            └─────────────────────┘

Key Architectural Components

1. Pull-Based Scraping Model: Unlike traditional agent-push frameworks, the Prometheus server periodically initiates HTTP GET requests to target /metrics endpoints published by applications and cluster services. Target discovery occurs dynamically via the Kubernetes API.
2. Time-Series Database (TSDB): Prometheus stores metric streams as time-series data consisting of a metric name, timestamp, numeric value, and key-value label pairs (e.g., http_requests_total{method="POST", status="500"}).
3. PromQL (Prometheus Query Language): A flexible functional query language used to compute real-time aggregations, rates of change, and quantiles (e.g., rate(http_requests_total[5m])).
4. Exporters: Auxiliary agents that translate existing non-Prometheus metrics into Prometheus format:
   • Node Exporter: Exposes host-level kernel and hardware metrics (CPU, memory, disk I/O).
   • kube-state-metrics: Listens to the Kubernetes API server and generates metrics about object state (e.g., Pod readiness, Deployment replica counts, resource quota usage).
5. Alertmanager: Handles alerts sent by Prometheus rules, deduplicating, grouping, and routing notifications to external channels like PagerDuty or Slack.
6. Grafana Integration: Grafana acts as the visualization layer, connecting to Prometheus to render rich dashboards.

---

Kubernetes Container Logging Architecture & Aggregation

Kubernetes does not provide a native centralized storage system for container logs. Instead, it defines a standard logging architecture based on Twelve-Factor application principles.

Node-Level Logging Pipeline

1. Standard Streams (stdout / stderr): Containerized applications write log messages directly to standard output (stdout) and standard error (stderr).
2. Container Runtime File Capture: The node container runtime (containerd or CRI-O) redirects these streams into log files on the host node filesystem, typically located at /var/log/pods/<namespace>_<pod-name>_<pod-uid>/<container-name>/0.log.
3. kubectl logs Command: Running kubectl logs <pod-name> queries the local node kubelet API, which streams text directly from these local node log files.

┌────────────────────────────────────────────────────────────────────────┐
│ WORKER NODE                                                            │
│                                                                        │
│ ┌───────────────────┐  Writes stdout   ┌─────────────────────────────┐ │
│ │ Containerized App │─────────────────►│ Container Runtime Log File  │ │
│ └───────────────────┘                  │ (/var/log/pods/...)         │ │
│                                        └──────────────┬──────────────┘ │
│                                                       │ Tails Log File │
│ ┌─────────────────────────────────────────────────────▼──────────────┐ │
│ │ Fluent Bit Logging Agent (DaemonSet Instance on Every Node)        │ │
│ └─────────────────────────────────────────────────────┬──────────────┘ │
└───────────────────────────────────────────────────────┼────────────────┘
                                                        │ Ships Logs
                                                        ▼
                                        ┌────────────────────────────────┐
                                        │  Centralized Storage Engine    │
                                        │  (Grafana Loki / Elasticsearch)│
                                        └────────────────────────────────┘

Centralized Log Aggregation Patterns

To persist logs when Pods terminate or nodes rotate, log aggregation agents operate using two main architectural patterns:

• Node-Level Agent (DaemonSet Pattern - Recommended): A lightweight log collector (such as Fluent Bit, Fluentd, or Vector) runs as a DaemonSet on every node. It tails all log files in /var/log/pods, enriches each log line with Kubernetes API metadata (namespace, pod name, container name, labels), and streams them to a centralized store like Grafana Loki or Elasticsearch.
• Sidecar Container Pattern: A secondary container runs alongside the app container in the same Pod to read logs from a shared volume or custom log file when the app cannot write to stdout/stderr.

---

Distributed Tracing & OpenTelemetry

While metrics indicate that a system is failing, and logs detail why an error occurred, Distributed Tracing explains where latency or failure occurred across a network of interconnected microservices.

Core Distributed Tracing Concepts

• Span: The fundamental building block of a trace. Represents a single named, timed contiguous unit of execution within a service (e.g., executing a database SQL query or calling an external HTTP API). A span contains start/end timestamps, tags, logs, and context metadata.
• Trace: A collection of structured spans forming a Directed Acyclic Graph (DAG) that represents the complete end-to-end execution path of a single client request across multiple microservices.
• Span Context Propagation: As a request travels between microservices over HTTP or gRPC, services pass tracing context metadata—primarily a unique TraceID and SpanID—via HTTP headers (standardized by the W3C Trace Context header traceparent).

[Client Request] ──► [API Gateway (Span A)] ──► [Auth Service (Span B)]
                                            └──► [Order Service (Span C)] ──► [Database Query (Span D)]

Total Trace ID: 4bf92f3577b34da6a3ce929d0e0e4736

OpenTelemetry (OTel) & Tracing Backends

• OpenTelemetry (OTel): A CNCF graduated project resulting from the merger of OpenTracing and OpenCensus. OTel provides a unified, vendor-agnostic set of APIs, SDKs, and the OTel Collector to generate, process, and export metrics, logs, and traces.
• Jaeger & Zipkin: Open-source distributed tracing storage engines and UIs (both CNCF projects) that receive trace spans from OTel agents and render interactive visual timeline graphs for latency analysis.