5.6 StatefulSet VolumeClaimTemplates & Persistent Storage Workflows

5.6 StatefulSet VolumeClaimTemplates & Persistent Storage Workflows

Deploying distributed, stateful clustering systems—such as Apache Kafka, Elasticsearch, Cassandra, ZooKeeper, and replicated MySQL/PostgreSQL databases—presents challenges that standard Kubernetes Deployments and ReplicaSets cannot resolve. Stateless Deployments treat pods as anonymous, interchangeable entities: all replicas share the exact same PodSpec, scale up and down arbitrarily, and if configured with a PVC, attempt to mount the identical shared volume (requiring ReadWriteMany support).

Clustered stateful applications, however, demand three architectural invariants:

1. Unique, persistent network identity that remains invariant across pod restarts.
2. Dedicated, isolated persistent storage per replica that is never shared with peer replicas.
3. Deterministic, ordered provisioning and graceful decommissioning to maintain distributed quorum.

Kubernetes fulfills these requirements through the StatefulSet workload controller.

---

1. StatefulSet vs. Deployment Storage Architecture

=============================== DEPLOYMENT STORAGE ===============================
  Deployment (replicas: 3) ---> References single PVC: shared-pvc
  +--------------+  +--------------+  +--------------+
  | web-pod-7x8a |  | web-pod-9b2c |  | web-pod-1z4e |
  +-------+------+  +-------+------+  +-------+------+
          |                 |                 |
          +-----------------+-----------------+
                            |
                            v
                  +-------------------+
                  |    shared-pvc     |  (Requires RWX; all pods share same disk)
                  +-------------------+

============================== STATEFULSET STORAGE ===============================
  StatefulSet: kafka (replicas: 3) with volumeClaimTemplate: data
  +---------------+    +---------------+    +---------------+
  |    kafka-0    |    |    kafka-1    |    |    kafka-2    |
  +-------+-------+    +-------+-------+    +-------+-------+
          |                    |                    |
          v                    v                    v
  +---------------+    +---------------+    +---------------+
  |  data-kafka-0 |    |  data-kafka-1 |    |  data-kafka-2 |
  |   (PVC / PV)  |    |   (PVC / PV)  |    |   (PVC / PV)  |
  +---------------+    +---------------+    +---------------+
  (Dedicated 1-to-1 persistent disk per ordinal replica; preserves state across failover)

---

2. Dynamic PVC Generation Formula

When a StatefulSet with replicas: N is applied, the StatefulSet controller dynamically synthesizes and creates $N$ individual PersistentVolumeClaim resources. The PVC names are generated using a deterministic formula:

<volumeClaimTemplate-name>-<statefulset-name>-<ordinal-index>

Formula Evaluation Matrix:

---

3. Pod Rescheduling & Persistent Storage Binding

The fundamental design guarantee of StatefulSets is that storage is bound to the ordinal identity, not the physical worker node.

+-----------------------------------------------------------------------------------------+
|                         STATEFULSET FAILOVER & REATTACHMENT                             |
|                                                                                         |
|  [INITIAL HEALTHY STATE]                                                                |
|  Worker Node 1:  kafka-0  <=======> Mounts: data-kafka-0 (PV-0)                         |
|  Worker Node 2:  kafka-1  <=======> Mounts: data-kafka-1 (PV-1)                         |
|                                                                                         |
|  [CRASH EVENT: Worker Node 2 Hardware Failure]                                          |
|  1. Node 2 becomes NotReady -> Kubelet stops heartbeating.                              |
|  2. Node controller evicts kafka-1.                                                     |
|  3. StatefulSet controller detects missing replica with ordinal 1.                       |
|  4. Scheduler places new kafka-1 on Worker Node 3.                                      |
|                                                                                         |
|  [FAILOVER RECOVERY: Worker Node 3]                                                     |
|  5. Kubelet on Node 3 watches Pod kafka-1.                                              |
|  6. CSI attacher detaches PV-1 from Node 2 and attaches PV-1 to Node 3.                 |
|  7. kafka-1 starts with 100% of its previous persistent data intact!                     |
+-----------------------------------------------------------------------------------------+

---

4. persistentVolumeClaimRetentionPolicy (StatefulSet Scaling & Deletion)

Historically, deleting or scaling down a StatefulSet never deleted any associated PVCs. While this protected against data loss, it caused orphaned PVCs and persistent cloud billing. Modern Kubernetes (stable since v1.32 and available in v1.35) provides the persistentVolumeClaimRetentionPolicy API.

spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain    # Options: Retain, Delete
    whenScaled: Retain     # Options: Retain, Delete

Retention Policies Explained:

• whenDeleted: Dictates what occurs to associated PVCs when the parent StatefulSet resource is deleted (kubectl delete statefulset <name>):
  • Retain (Default): PVCs and underlying PVs are preserved in the namespace.
  • Delete: All PVCs created by volumeClaimTemplates are automatically deleted along with the StatefulSet.
• whenScaled: Dictates what occurs to PVCs when the StatefulSet replica count is reduced (e.g., scaling from 5 replicas down to 2):
  • Retain (Default): PVCs for scaled-down pods (data-app-4, data-app-3, data-app-2) remain intact. If scaled back up, the pods reattach to them.
  • Delete: The PVCs belonging to the terminated replicas are permanently deleted, freeing cloud storage assets.

---

5. Pod Management Policies & Headless Services

1. podManagementPolicy (OrderedReady vs. Parallel)

• OrderedReady (Default): Replicas are created sequentially in strict ascending order ($0 \to 1 \to 2$). Pod $N$ is not launched until Pod $N-1$ is in the Running and Ready state. Teardown occurs in reverse order ($2 \to 1 \to 0$). Essential for clustering protocols (e.g., bootstrapping Raft leaders).
• Parallel: Replicas are launched and terminated concurrently without waiting for peer readiness. Used when workloads need independent storage identities but rapid scaling.

2. Headless Service (clusterIP: None)

A StatefulSet requires a companion Headless Service to establish stable network identities. CoreDNS automatically generates A/AAAA DNS records for each individual pod:

<pod-name>.<service-name>.<namespace>.svc.cluster.local

Example: kafka-0.kafka-headless.production.svc.cluster.local.

---

6. Partitioned Rolling Updates & Canary Migrations

StatefulSets support staged rollouts using the partition directive under spec.updateStrategy.rollingUpdate.

spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 2

Partition Mechanics & Canary Progression

When partition: 2 is specified on a 4-replica StatefulSet (db-0, db-1, db-2, db-3):

• Only replicas with ordinal index $\ge 2$ (db-2 and db-3) are updated to the new PodSpec.
• Replicas with ordinal index $< 2$ (db-0 and db-1) remain untouched on the legacy version.
• Canary Database Migration Workflow:
  1. Set partition: 3 -> Only db-3 is updated. Validate telemetry, database schema migrations, and replication lag.
  2. If healthy, decrement partition: 1 -> Updates db-2 and db-1.
  3. Decrement partition: 0 -> Updates the final replica (db-0), completing full cluster upgrade.

---

7. Production-Grade StatefulSet Declarative Manifest

apiVersion: v1
kind: Service
metadata:
  name: postgres-ha
  namespace: production
  labels:
    app: postgres
spec:
  clusterIP: None
  ports:
  - port: 5432
    name: postgresql
  selector:
    app: postgres
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: pg-cluster
  namespace: production
spec:
  serviceName: postgres-ha
  replicas: 3
  podManagementPolicy: OrderedReady
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:16-alpine
        ports:
        - containerPort: 5432
          name: postgresql
        volumeMounts:
        - name: pgdata
          mountPath: /var/lib/postgresql/data
          subPath: data
  volumeClaimTemplates:
  - metadata:
      name: pgdata
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: high-performance-ebs
      resources:
        requests:
          storage: 100Gi

---

8. StatefulSet Disaster Recovery & Storage Maintenance

# 1. Scale down StatefulSet to isolate broken replica
kubectl scale statefulset pg-cluster -n production --replicas=2

# 2. Inspect orphaned PVCs left behind after scale-down
kubectl get pvc -n production -l app=postgres

# 3. Perform online volume expansion on StatefulSet PVCs
kubectl patch pvc pgdata-pg-cluster-0 -n production --type=merge -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'
kubectl patch pvc pgdata-pg-cluster-1 -n production --type=merge -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'
kubectl patch pvc pgdata-pg-cluster-2 -n production --type=merge -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'