5.1 Ephemeral Volumes, EmptyDir, HostPath & Config Volumes

5.1 Ephemeral Volumes, EmptyDir, HostPath & Config Volumes

In Kubernetes, the lifecycle of a container's filesystem is inherently decoupled from the state of the applications running inside it. By default, container runtimes (such as containerd or CRI-O) construct container root filesystems using copy-on-write overlay filesystems (e.g., overlay2). Image layers are immutable and read-only; when a container process writes, modifies, or deletes a file, changes are written to a thin, ephemeral writable layer. If the container process crashes or restarts, the Container Runtime Interface (CRI) destroys that writable layer and instantiates a pristine container from the original image—permanently obliterating any uncommitted data.

To preserve state across container restarts, coordinate data sharing among co-located containers, and expose runtime metadata or cryptographic secrets, Kubernetes provides the Volume abstraction. Unlike pure container storage, Kubernetes volumes possess an explicit lifecycle tied to the enclosing Pod, surviving container crashes and restarts as long as the Pod itself remains scheduled on the host node.

---

1. Storage Paradigms: Container Layer vs. Pod Volumes

Understanding where data resides within a worker node is fundamental for diagnosing performance bottlenecks, disk pressure, and security boundaries.

+-----------------------------------------------------------------------------------------+
|                                 WORKER NODE (HOST FILESYSTEM)                           |
|                                                                                         |
|  +-----------------------------------------------------------------------------------+  |
|  |                                  POD BOUNDARY                                     |  |
|  |                                                                                   |  |
|  |  +-------------------------------------+   +-----------------------------------+  |  |
|  |  |         APP CONTAINER               |   |        SIDECAR CONTAINER          |  |  |
|  |  |  +-------------------------------+  |   |  +-----------------------------+  |  |  |
|  |  |  | Writable Layer (Lost on crash)|  |   |  | Writable Layer (Lost on     |  |  |  |
|  |  |  +-------------------------------+  |   |  | crash)                      |  |  |  |
|  |  |  | Read-Only Image Layers        |  |   |  +-----------------------------+  |  |  |
|  |  |  +---------------+---------------+  |   |  | Read-Only Image Layers      |  |  |  |
|  |  +------------------|------------------+   +----------------|------------------+  |  |
|  |                     | Mount: /app/data                      | Mount: /log-input   |  |
|  |                     +-------------------+-------------------+                     |  |
|  |                                         |                                         |  |
|  |                                         v                                         |  |
|  |                        +---------------------------------+                        |  |
|  |                        |       POD-LEVEL VOLUME          |                        |  |
|  |                        |  (Survives Container Restarts)  |                        |  |
|  |                        +---------------------------------+                        |  |
|  +-----------------------------------------|-----------------------------------------+  |
|                                            |                                            |
|                                            v                                            |
|  +-----------------------------------------------------------------------------------+  |
|  | BACKING STORAGE: Node Local Disk (/var/lib/kubelet) | Node RAM (tmpfs) | Host Path|  |
|  +-----------------------------------------------------------------------------------+  |
+-----------------------------------------------------------------------------------------+

Comparison of Ephemeral Storage Primitives

---

2. emptyDir Volumes: Disk vs. RAM Backing

An emptyDir volume is created the moment a Pod is assigned to a worker node. As the name implies, it is initially empty. All containers within the Pod can read and write the exact same files within the emptyDir, though each container may mount it at an identical or entirely different mountPath.

Filesystem Mechanics & Path on Host

When backed by disk, kubelet initializes a directory on the worker node's root filesystem located at:

/var/lib/kubelet/pods/<pod-uid>/volumes/kubernetes.io~empty-dir/<volume-name>/

When the Pod is deleted, evicted due to resource exhaustion, or rescheduled onto another node, the entire contents of the emptyDir are permanently erased by the kubelet.

Disk vs. Memory (tmpfs) Configuration

By default, emptyDir is stored on the medium backing the node's /var/lib/kubelet directory (standard SSD or HDD). However, setting medium: Memory instructs Kubernetes to mount a Linux tmpfs (RAM-backed virtual filesystem).

apiVersion: v1
kind: Pod
metadata:
  name: cache-processing-pipeline
  namespace: default
spec:
  containers:
  - name: producer-engine
    image: redis:7-alpine
    command: ["sh", "-c", "redis-server --dir /cache/ramdisk --save ''"]
    volumeMounts:
    - name: fast-cache
      mountPath: /cache/ramdisk
    resources:
      limits:
        memory: "512Mi"
        cpu: "500m"
      requests:
        memory: "256Mi"
        cpu: "250m"
  - name: consumer-analyzer
    image: alpine:latest
    command: ["sh", "-c", "while true; do ls -la /shared-cache; sleep 5; done"]
    volumeMounts:
    - name: fast-cache
      mountPath: /shared-cache
      readOnly: true
  volumes:
  - name: fast-cache
    emptyDir:
      medium: Memory
      sizeLimit: 256Mi

[!WARNING] tmpfs Memory Accounting and OOMKiller Risk: When using medium: Memory, the data written to the emptyDir resides directly in Linux kernel RAM and is accounted against the container's Memory Limit. The sizeLimit caps volume capacity, so writes beyond it can fail with a no-space error. Separately, tmpfs pages count toward container or Pod memory accounting; if aggregate charged memory exceeds a cgroup memory limit, the container can be OOM-killed. Exceeding the volume size limit alone does not guarantee exit code 137.

---

3. hostPath Volumes: Node Binding & Security Hardening

A hostPath volume mounts a file or directory from the host worker node's filesystem directly into your Pod's container filesystem. While emptyDir creates a sandboxed directory managed entirely by kubelet, hostPath exposes arbitrary host directories.

Supported hostPath Types

Kubernetes requires specifying a type to validate host filesystem state before allowing the Pod to bind:

Legitimate DaemonSet Pattern vs. Container Host Escape Vulnerabilities

apiVersion: v1
kind: Pod
metadata:
  name: node-log-collector
  namespace: kube-system
spec:
  containers:
  - name: fluentbit
    image: fluent/fluent-bit:3.0
    volumeMounts:
    - name: host-var-log
      mountPath: /var/log
      readOnly: true
    - name: host-containerd-sock
      mountPath: /run/containerd/containerd.sock
  volumes:
  - name: host-var-log
    hostPath:
      path: /var/log
      type: Directory
  - name: host-containerd-sock
    hostPath:
      path: /run/containerd/containerd.sock
      type: Socket

[!CAUTION] Host Escape and Privilege Escalation Vulnerabilities: If an unprivileged application container is permitted to mount sensitive host directories such as /, /etc, /var/run, or /root with write permissions, a compromised container process can modify host system binaries, alter /etc/shadow, install malicious root cron jobs, or manipulate kubelet static pod manifests in /etc/kubernetes/manifests/ to achieve total cluster takeover.

Under modern Kubernetes security standards, Pod Security Admission (PSA) at the restricted profile completely bans hostPath volumes. When required for system monitoring or CNI DaemonSets, always enforce readOnly: true in volumeMounts.

---

4. Configuration Volumes & Projected Volumes

Kubernetes provides dedicated volume mechanisms to inject operational configuration and cryptographic identity into containers without baking sensitive data into container images.

Projected Volumes (spec.volumes[*].projected)

Rather than defining separate volume mounts for ConfigMaps, Secrets, Downward API fields, and ServiceAccount tokens, a Projected Volume aggregates all of these disparate sources into a single, cohesive directory structure.

apiVersion: v1
kind: Pod
metadata:
  name: projected-application-pod
  namespace: production
  labels:
    environment: production
    tier: backend
spec:
  containers:
  - name: web-api
    image: nginx:1.27-alpine
    volumeMounts:
    - name: unified-config-vault
      mountPath: /etc/app-runtime
      readOnly: true
  volumes:
  - name: unified-config-vault
    projected:
      defaultMode: 0400
      sources:
      - configMap:
          name: application-config
          items:
          - key: server.conf
            path: configs/server.conf
      - secret:
          name: database-credentials
          items:
          - key: db-password
            path: secrets/db_password.txt
            mode: 0400
      - downwardAPI:
          items:
          - path: metadata/pod_name
            fieldRef:
              fieldPath: metadata.name
          - path: metadata/pod_namespace
            fieldRef:
              fieldPath: metadata.namespace
          - path: metadata/cpu_limit
            resourceFieldRef:
              containerName: web-api
              resource: limits.cpu
      - serviceAccountToken:
          audience: vault-auth-service
          expirationSeconds: 7200
          path: tokens/vault-token

---

5. Advanced Volume Mount Directives: subPath, subPathExpr & mountPropagation

1. subPath (Preventing Directory Obliteration)

When mounting a ConfigMap, Secret, or Volume into a pre-existing directory (e.g., /etc/nginx/conf.d), standard mounting overlays and hides all existing container files in that directory. Specifying subPath allows mounting a single individual file without masking the rest of the target directory:

volumeMounts:
- name: custom-vhost-config
  mountPath: /etc/nginx/conf.d/custom-vhost.conf
  subPath: custom-vhost.conf

2. subPathExpr (Dynamic Variable Expansion)

Constructs dynamic subdirectory paths using container environment variables, particularly useful for multi-pod logging segregation:

env:
- name: POD_NAME
  valueFrom:
    fieldRef:
      fieldPath: metadata.name
volumeMounts:
- name: shared-storage
  mountPath: /mnt/logs
  subPathExpr: logs/$(POD_NAME)

3. mountPropagation (Linux Mount Namespaces)

Governs whether mounts created inside a container are shared back with the host OS or other containers on the same volume:

• None (Default / Private): Container receives no mounts created by the host after container start, and host sees no mounts created by container.
• HostToContainer (rslave): Container sees new mounts subsequently created by the host on that volume.
• Bidirectional (rshared): Mounts created inside the container are immediately reflected on the host filesystem. Strictly requires securityContext.privileged: true (used exclusively by CSI storage plugins and CNI daemons).

---

6. Diagnostic & Inspection Commands

# 1. Inspect volume disk usage from within a running pod container
kubectl exec -it cache-processing-pipeline -c producer-engine -- df -h /cache/ramdisk

# 2. Check for volume mount failures and hostPath permission errors in real-time
kubectl describe pod node-log-collector | grep -A 8 -E "(Mounts|Volumes|Events:)"

# 3. Identify pods mounting host sockets or sensitive host directories
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" : "}{.spec.volumes[*].hostPath.path}{"\n"}{end}' | grep -v " : $"