6.7 Container & Node Logging Architecture, Journalctl & Fluentd

6.7 Container & Node Logging Architecture, Journalctl & Fluentd

In containerized environments, applications are ephemeral. When a container crashes, restarts, or is rescheduled to another node, its local in-memory state and ephemeral files are lost. To maintain observability and troubleshoot complex failures, Kubernetes establishes a standardized logging architecture.

Understanding how application output travels from standard streams to on-disk files, how kubectl logs streams data, how host daemons log to systemd-journald, and how cluster-wide log forwarders operate is vital for the CKA exam and production engineering.

---

1. Container Engine Logging Pipeline & Storage Layout

Kubernetes assumes that containerized applications write their operational logs directly to standard output (stdout) and standard error (stderr).

+-----------------------------------------------------------------------------------------+
|                           CONTAINER LOG PIPELINE ARCHITECTURE                           |
|                                                                                         |
|  [APPLICATION IN CONTAINER]                                                             |
|         | (stdout / stderr)                                                             |
|         v                                                                               |
|  [CONTAINER RUNTIME (containerd / CRI-O)]                                               |
|  - Formats logs into CRI format (timestamp stream log-line)                             |
|  - Writes to physical disk on host node:                                                |
|    /var/log/pods/<namespace>_<pod-name>_<pod-uid>/<container-name>/<retry-count>.log     |
|         |                                                                               |
|         +---> Symlinked to: /var/log/containers/<pod-name>_<namespace>_<container>.log  |
|         |                                                                               |
|         +====================================+====================================+     |
|         |                                    |                                    |     |
|         v                                    v                                    v     |
|  [KUBECTL LOGS STREAM]              [NODE-LEVEL AGENT]                   [HOST DAEMONS] |
|  - Kubelet serves log stream        - Fluentd / Fluent Bit / Promtail    - kubelet      |
|    to kube-apiserver over           - DaemonSet mounts /var/log/pods     - containerd   |
|    HTTPS port 10250                 - Forwards to Elastic / Loki         - journalctl   |
+-----------------------------------------------------------------------------------------+

Physical On-Disk Log Structure:

1. Primary Log Path:

   /var/log/pods/<namespace>_<pod-name>_<pod-uuid>/<container-name>/0.log
   

2. Convenience Symlink:

   /var/log/containers/<pod-name>_<namespace>_<container-name>-<container-id>.log
   

3. CRI Log Line Format:

   2026-08-23T08:15:30.123456789Z stdout F [INFO] Server listening on port 8080
   2026-08-23T08:15:31.987654321Z stderr F [ERROR] Failed to connect to database
   

   • 2026-08-23T...: ISO 8601 UTC timestamp.
   • stdout / stderr: Origin stream.
   • F (Full) / P (Partial): Indicates whether the log line was complete or truncated across multiple chunks.

---

2. Kubelet & Containerd Log Rotation Mechanics

To prevent rogue applications from generating massive logs that exhaust host disk space (DiskPressure), both the kubelet and containerd implement automatic log rotation.

Kubelet Configuration Parameters (/var/lib/kubelet/config.yaml):

• containerLogMaxSize: Maximum file size before rotation (default: 10Mi).
• containerLogMaxFiles: Maximum number of rotated log archives retained per container (default: 5).

Kubelet asks the runtime to rotate logs according to configured size and file-count limits. Exact rotated filenames and whether archives are compressed are runtime-specific, so inspect /var/log/pods and the node runtime rather than assuming a 0.log.1.gz convention.

---

3. Mastering kubectl logs Syntax

kubectl logs queries the kubelet API on the target node via kube-apiserver:

# 1. Basic log retrieval for a single-container pod
kubectl logs web-pod

# 2. Specify container in a multi-container pod
kubectl logs multi-pod -c app-container

# 3. Stream logs across ALL containers in a multi-container pod simultaneously
kubectl logs multi-pod --all-containers=true

# 4. Stream logs from a previously crashed container instance
kubectl logs web-pod --previous

# 5. Follow live log output in real time (-f)
kubectl logs -f web-pod

# 6. View the last N lines (--tail)
kubectl logs --tail=50 web-pod

# 7. View logs generated in the last 15 minutes (--since)
kubectl logs --since=15m web-pod

# 8. Include RFC3339 timestamps in log output
kubectl logs --timestamps web-pod

# 9. Aggregate logs from all pods matching a label selector
kubectl logs -l app=payment-service --tail=20

---

4. Host Systemd & OS Log Diagnostics (journalctl)

Host-level systemd services (such as kubelet and containerd) do not write to /var/log/pods. Their logs are managed by systemd-journald.

# View kubelet logs starting from the end of the journal (-e) with line limit (-n)
journalctl -u kubelet -e --no-pager -n 100

# Follow live kubelet logs
journalctl -u kubelet -f

# View containerd runtime logs
journalctl -u containerd -e --no-pager -n 100

# Filter logs by time window
journalctl -u kubelet --since "2026-08-23 07:00:00" --until "2026-08-23 08:00:00"

# Check kernel messages for hardware errors or OOM events
dmesg -T | tail -n 50

---

5. Cluster-Level Logging Architectures

Kubernetes does not provide a native centralized storage solution for logs. Three primary architectural patterns exist for cluster-level logging:

+-----------------------------------------------------------------------------------------+
|                        CLUSTER LOGGING PATTERNS COMPARISON                              |
|                                                                                         |
|  [PATTERN 1: NODE LOGGING AGENT DAEMONSET (Standard / Recommended)]                     |
|  - Fluent Bit / Fluentd runs as DaemonSet on every node.                                |
|  - Mounts host `/var/log/pods` read-only.                                               |
|  - Parses CRI formatting, enriches with K8s metadata, forwards to Elastic / Loki.       |
|  - Pros: Minimal resource overhead; zero changes to application pods.                   |
|                                                                                         |
|  [PATTERN 2: SIDECAR LOGGING CONTAINER]                                                 |
|  - Sidecar container runs inside application pod.                                       |
|  - Streams application log files (e.g., `/var/log/nginx/access.log`) to sidecar stdout |
|    or ships directly to central backend.                                                |
|  - Pros: Useful for legacy applications that cannot log to stdout.                      |
|  - Cons: Doubled memory/CPU overhead per pod.                                           |
|                                                                                         |
|  [PATTERN 3: DIRECT APPLICATION LOGGING]                                                |
|  - Application directly connects to logging backend (e.g., Logstash / Datadog API).     |
|  - Cons: Tightly couples application code to logging infrastructure.                    |
+-----------------------------------------------------------------------------------------+

Troubleshooting Missing Logs in Centralized Pipelines:

1. DaemonSet Pod Failed: Verify logging DaemonSet health: kubectl get pods -n logging.
2. Host Volume Mount Broken: Ensure Fluent Bit manifest mounts /var/log/pods and /var/log/containers.
3. Application Writing to Internal File: If kubectl logs produces nothing, the app may be logging to a private on-disk file instead of stdout/stderr.