5.4 Storage Access Modes, Reclaim Policies & Volume Expansion

5.4 Storage Access Modes, Reclaim Policies & Volume Expansion

Configuring persistent storage for mission-critical enterprise workloads requires balancing data concurrency, lifecycle retention, and runtime capacity elasticity. Administrators must ensure that multi-pod access does not cause filesystem corruption, that accidental deletion of workload manifests does not destroy irrecoverable production databases, and that storage volumes can be expanded online as data volumes grow without requiring workload downtime.

This section examines Kubernetes Access Modes (including the modern ReadWriteOncePod mode), Reclaim Policies and the manual reclamation reuse procedure, and Dynamic Volume Expansion mechanics.

---

1. Storage Access Modes Deep Dive

When declaring a PersistentVolume or PersistentVolumeClaim, the accessModes field specifies how the volume can be mounted by cluster nodes. Crucially, a volume can only be mounted using one access mode at a time, even if the underlying storage hardware physically supports multiple modes.

+-----------------------------------------------------------------------------------------+
|                                KUBERNETES ACCESS MODES                                  |
|                                                                                         |
|   ReadWriteOnce (RWO)                     ReadOnlyMany (ROX)                            |
|   +-----------------------------+         +-----------------------------+               |
|   | Node 1: Pod A (Read-Write)  |         | Node 1: Pod A (Read-Only)   |               |
|   | Node 1: Pod B (Read-Write)  |         | Node 2: Pod B (Read-Only)   |               |
|   | (Single Node Only)          |         | (Multiple Nodes Read)       |               |
|   +-----------------------------+         +-----------------------------+               |
|                                                                                         |
|   ReadWriteMany (RWX)                     ReadWriteOncePod (RWOP)                       |
|   +-----------------------------+         +-----------------------------+               |
|   | Node 1: Pod A (Read-Write)  |         | Node 1: Pod A (Read-Write)  |               |
|   | Node 2: Pod B (Read-Write)  |         | Node 1: Pod B (BLOCKED)     |               |
|   | (Multiple Nodes Write)      |         | (Single POD Cluster-Wide)   |               |
|   +-----------------------------+         +-----------------------------+               |
+-----------------------------------------------------------------------------------------+

Access Modes Breakdown & Backend Compatibility

Why ReadWriteOncePod (RWOP) Is Essential for Stateful Safety

Under standard ReadWriteOnce (RWO), if a StatefulSet pod replica is updated via a rolling deployment and the scheduler places the replacement pod on the same worker node before the terminating pod fully exits, both pods will attach and write to the volume simultaneously. For single-writer database engines (such as MySQL, PostgreSQL, or embedded SQLite), this overlapping access frequently causes irreversible index corruption. ReadWriteOncePod enforces strict single-pod mutual exclusion cluster-wide.

---

2. Reclaim Policies & The Manual PV Reuse Workflow

When an application developer deletes a PersistentVolumeClaim, the bound PersistentVolume executes the reclamation behavior defined in spec.persistentVolumeReclaimPolicy:

persistentVolumeReclaimPolicy: Retain   # Valid Options: Retain, Delete, Recycle

Reclaim Policies Compared:

1. Retain (Manual Reclamation):
   • The PVC is deleted -> The PV transitions to the Released status.
   • The underlying physical data on the SAN/Cloud volume remains 100% intact.
   • No new PVC can bind to the PV because spec.claimRef still points to the deleted claim's UID.
2. Delete (Automatic Destruction):
   • The PVC deletion leads Kubernetes to remove the PV and ask the provisioner to delete the backing asset.
   • Storage deletion is asynchronous and provider-dependent; verify completion before assuming the data is gone.
3. Recycle (Deprecated):
   • Executes a basic filesystem wipe (rm -rf /volume/*) and returns the PV to Available.

Step-by-Step: Reclaiming and Reusing a Released PV

In CKA troubleshooting scenarios, administrators must recover data from a Released PV and make it available for a new application claim without losing underlying storage:

# Step 1: Inspect the Released PV
kubectl get pv enterprise-nfs-pv
# NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM
# enterprise-nfs-pv   50Gi       RWO            Retain           Released   prod/old-claim

# Step 2: Export a backup copy of the PV manifest
kubectl get pv enterprise-nfs-pv -o yaml > pv-backup.yaml

# Step 3: Secure any required backup and scrub or validate retained data for the intended new claimant
# Perform the storage-system-specific sanitization before exposing the PV.

# Step 4: Remove the claimRef locking the old claim
kubectl patch pv enterprise-nfs-pv --type=json -p='[{"op": "remove", "path": "/spec/claimRef"}]'

# Step 5: Verify the PV status transitions from 'Released' to 'Available'
kubectl get pv enterprise-nfs-pv
# NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM
# enterprise-nfs-pv   50Gi       RWO            Retain           Available

# Step 6: Restrict the next claim deliberately with storageClassName, selector, or claimRef as appropriate.

---

3. Dynamic Volume Expansion (Online Resizing)

Kubernetes can expand a PVC, but completion behavior depends on the CSI driver, volume mode, and filesystem. Many supported filesystems grow online while in use; other combinations require the consuming Pod to restart.

+-----------------------------------------------------------------------------------------+
|                               VOLUME EXPANSION LIFECYCLE                                |
|                                                                                         |
|  1. Developer edits PVC: spec.resources.requests.storage: 50Gi -> 100Gi                 |
|        |                                                                                |
|        v                                                                                |
|  2. CSI Resizer sidecar intercepts event and calls ControllerExpandVolume               |
|        |                                                                                |
|        v                                                                                |
|  3. Cloud Storage API expands physical block device to 100Gi                            |
|        |                                                                                |
|        v                                                                                |
|  4. Kubelet/CSI performs supported online resize, or reports a pending node-side resize   |
|        |                                                                                |
|        v                                                                                |
|  5. Verify PVC capacity and conditions; restart the Pod only when offline resize is required                  |
+-----------------------------------------------------------------------------------------+

Step 1: Verify StorageClass Enables Expansion

Dynamic volume expansion requires that the backing StorageClass explicitly enables the feature:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: expandable-sc
provisioner: ebs.csi.aws.com
allowVolumeExpansion: true    # MANDATORY: Must be set to true

Step 2: Expand the PersistentVolumeClaim

Apply an imperative JSON patch to increase the requested storage capacity:

# Patch the PVC storage request from 50Gi to 100Gi
kubectl patch pvc database-storage-pvc -n default --type=merge -p '{"spec":{"resources":{"requests":{"storage":"100Gi"}}}}'

Expansion Constraints & FileSystemResizePending Condition

• Strictly One-Way: Volume expansion can only increase storage capacity. Attempting to decrease volume size (e.g., from 100Gi down to 50Gi) is immediately rejected by the API server with a validation error because the new requested capacity is below the existing value.
• FileSystemResizePending: If the physical block device is expanded by the cloud provider but the Pod is currently offline, the PVC status condition will report FileSystemResizePending. Starting the Pod triggers kubelet to run resize2fs (for ext4) or xfs_growfs (for xfs) online, completing the expansion.