5.2 PersistentVolumes (PV) & PersistentVolumeClaims (PVC) Lifecycle

5.2 PersistentVolumes (PV) & PersistentVolumeClaims (PVC) Lifecycle

Managing persistent state in a containerized, distributed orchestration platform requires abstracting the underlying storage implementation details from application developers. In traditional on-premise or cloud environments, configuring storage arrays (such as iSCSI LUNs, NFS shares, AWS EBS volumes, or Ceph RBD pools) requires infrastructure-level privileges and vendor-specific configuration. Exposing these low-level parameters directly inside application Pod manifests breaks workload portability and violates the principle of least privilege.

Kubernetes solves this decoupling challenge through two core API primitives: the PersistentVolume (PV) and the PersistentVolumeClaim (PVC). This architectural separation enforces a strict division of responsibilities: cluster administrators manage the provisioned storage capacity (PVs), while application developers consume storage via declarative requests (PVCs) without needing knowledge of the backing storage fabric.

---

1. The Storage Abstraction Architecture & Persona Separation

+-----------------------------------------------------------------------------------------+
|                                    CLUSTER SCOPE                                        |
|                                                                                         |
|  +-----------------------------+                 +-----------------------------------+  |
|  |    STORAGE ADMINISTRATOR    |                 |      PersistentVolume (PV)        |  |
|  |  - Provisions SAN/NFS/Cloud |  ------------>  |  - Capacity: 50Gi                 |  |
|  |  - Defines Access Modes     |                 |  - AccessMode: ReadWriteOnce      |  |
|  |  - Reclaim Policy: Retain   |                 |  - ReclaimPolicy: Retain          |  |
|  +-----------------------------+                 +-----------------+-----------------+  |
|                                                                    |                    |
|====================================================================|====================|
|                          NAMESPACE SCOPE: production               | (1-to-1 Binding)   |
|                                                                    v                    |
|  +-----------------------------+                 +-----------------+-----------------+  |
|  |    APPLICATION DEVELOPER    |                 |   PersistentVolumeClaim (PVC)     |  |
|  |  - Requests: 30Gi RWO       |  ------------>  |  - Request: 30Gi                  |  |
|  |  - References StorageClass  |                 |  - AccessMode: ReadWriteOnce      |  |
|  +-----------------------------+                 +-----------------+-----------------+  |
|                                                                    |                    |
|                                                                    | spec.volumes.pvc   |
|                                                                    v                    |
|                                                  +-----------------+-----------------+  |
|                                                  |          WORKLOAD POD             |  |
|                                                  |  Mounts to /var/lib/postgresql    |  |
|                                                  +-----------------------------------+  |
+-----------------------------------------------------------------------------------------+

Architectural Distinction: PV vs. PVC

---

2. Complete 4-Phase PV & PVC Lifecycle

The interaction between PersistentVolumes and PersistentVolumeClaims is orchestrated by the control plane's persistentvolume-controller through four sequential phases:

1. Provisioning $\longrightarrow$ 2. Binding $\longrightarrow$ 3. Using $\longrightarrow$ 4. Reclaiming

Phase 1: Provisioning (Static vs. Dynamic)

• Static Provisioning: A cluster administrator manually provisions physical storage devices (e.g., creating an NFS export or Ceph volume) and submits declarative PersistentVolume manifests to the Kubernetes API server.
• Dynamic Provisioning: When no pre-existing static PV matches a user's PVC, the cluster automatically triggers a Container Storage Interface (CSI) provisioner via a StorageClass to create the backend storage asset and instantiate a corresponding PV object on the fly.

Phase 2: Binding

The persistentvolume-controller continuously watches for unbound PVCs and executes an evaluation algorithm to find the optimal matching PV.

• When a match is found, the controller executes an exclusive two-way mutual binding:
  • In the PV: Sets spec.claimRef to the PVC's UID, name, and namespace.
  • In the PVC: Sets spec.volumeName to the PV name and transitions status to Bound.
• If no matching PV exists and dynamic provisioning is disabled, the PVC remains indefinitely in Pending status.

Phase 3: Using

The application developer references the PVC in a Pod manifest under spec.volumes[*].persistentVolumeClaim.claimName.

• The scheduler places the Pod on an eligible node.
• The local kubelet and CSI Node Plugin attach the storage device, format it (if filesystem mode), and bind-mount it into the container's designated mountPath.

Phase 4: Reclaiming

When the application is decommissioned and the user deletes the PVC, the bound PV enters the reclamation phase dictated by its persistentVolumeReclaimPolicy:

• Retain: PV status transitions to Released. The underlying data on disk is preserved. The PV cannot be claimed by any other PVC until an administrator manually clears spec.claimRef.
• Delete: The PV object is automatically deleted from Kubernetes, and the CSI driver calls the underlying storage provider API to destroy the physical storage volume.
• Recycle (Deprecated): Runs a basic scrubbing command (rm -rf /mount/*) and returns the PV to Available.

---

3. PV Operational Status State Transitions

   [ Admin Manifest / CSI Provisioning ]
                     |
                     v
            +------------------+
            |    AVAILABLE     | <-------------------------+
            +--------+---------+                           |
                     |                                     |
                     | PVC matches criteria                | Admin cleans data
                     v                                     | & patches claimRef
            +------------------+                           |
            |      BOUND       |                           |
            +--------+---------+                           |
                     |                                     |
                     | User deletes PVC                    |
                     v                                     |
            +------------------+   Manual Reclaim          |
            |     RELEASED     | --------------------------+
            +--------+---------+
                     |
                     | Automated deletion error (CSI)
                     v
            +------------------+
            |      FAILED      |
            +------------------+

---

4. The 5-Point PVC-to-PV Binding Evaluation Algorithm

For persistentvolume-controller to bind an incoming PVC to a static PV, all five of the following evaluation criteria must pass simultaneously:

1. Storage Capacity: The PV capacity must be greater than or equal to the PVC requested storage (pv.spec.capacity.storage >= pvc.spec.resources.requests.storage). (Note: If a PVC requests 10Gi and the smallest available PV is 100Gi, Kubernetes will bind it, resulting in 90Gi of unallocated capacity).
2. Access Modes: The PV must satisfy all access modes declared in pvc.spec.accessModes (e.g., ReadWriteOnce, ReadWriteMany).
3. Volume Mode: spec.volumeMode must match identically (Filesystem with Filesystem, or Block with Block).
4. StorageClass: pvc.spec.storageClassName must match pv.spec.storageClassName. If a PVC specifies storageClassName: "", it will only bind to PVs that explicitly declare storageClassName: "" or omit the field.
5. Label Selectors: If the PVC specifies spec.selector.matchLabels or matchExpressions, the PV's metadata.labels must satisfy the selector.

---

5. Declarative Manifests: Filesystem vs. Raw Block Storage

Static PersistentVolume (Filesystem Mode)

apiVersion: v1
kind: PersistentVolume
metadata:
  name: enterprise-nfs-pv
  labels:
    environment: production
    tier: db
spec:
  capacity:
    storage: 50Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: manual
  nfs:
    server: 10.96.15.200
    path: /srv/nfs/pgdata

PersistentVolumeClaim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data-pvc
  namespace: production
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 50Gi
  storageClassName: manual
  selector:
    matchLabels:
      environment: production
      tier: db

Raw Block Volume (volumeMode: Block)

High-throughput databases (e.g., Cassandra, RocksDB, or custom key-value engines) often bypass the Linux filesystem layer to eliminate filesystem journaling overhead:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: raw-block-pvc
  namespace: production
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Block
  resources:
    requests:
      storage: 100Gi
  storageClassName: manual
---
apiVersion: v1
kind: Pod
metadata:
  name: raw-db-pod
  namespace: production
spec:
  containers:
  - name: db-engine
    image: custom-db:v2.4
    volumeDevices:
    - name: raw-storage-device
      devicePath: /dev/xvdf
  volumes:
  - name: raw-storage-device
    persistentVolumeClaim:
      claimName: raw-block-pvc

[!NOTE] When consuming a raw block volume (volumeMode: Block), you must declare volumeDevices with devicePath instead of volumeMounts with mountPath.

---

6. Verification & Troubleshooting CLI Playbook

# 1. List all PVs cluster-wide and check claim references
kubectl get pv -o wide

# 2. List PVCs in a target namespace
kubectl get pvc -n production

# 3. Diagnose why a PVC is stuck in Pending status
kubectl describe pvc postgres-data-pvc -n production

# 4. Check if PV claimRef is locking a Released volume
kubectl get pv enterprise-nfs-pv -o jsonpath='{.spec.claimRef}'