6.10 Storage Failures: PVC Pending, Mount Failures & Multi-Attach Errors

6.10 Storage Failures: PVC Pending, Mount Failures & Multi-Attach Errors

Managing persistent state in a distributed container orchestration system involves coordinating the Kubernetes storage control loops (PV controller, Attach/Detach controller, and Volume Manager) with third-party Container Storage Interface (CSI) drivers, cloud storage APIs, and host filesystem mount points.

Storage misconfigurations cause pods to hang in Pending or ContainerCreating. Diagnosing storage problems requires understanding the binding lifecycle, CSI plugin communication, and reclaim policies.

---

1. Storage Architecture & Binding Lifecycle

+-----------------------------------------------------------------------------------------+
|                        STORAGE BINDING & ATTACHMENT PIPELINE                            |
|                                                                                         |
|  1. DEVELOPER CREATES PVC                                                               |
|     apiVersion: v1, kind: PersistentVolumeClaim (Requests 10Gi, ReadWriteOnce)          |
|                                    |                                                    |
|                                    v                                                    |
|  2. PV CONTROLLER ATTEMPTS BINDING                                                      |
|     - Static Provisioning: Matches existing PV with sufficient capacity, matching       |
|       accessModes, and matching storageClassName.                                       |
|     - Dynamic Provisioning: StorageClass invokes CSI External Provisioner to create PV. |
|                                    |                                                    |
|                                    v                                                    |
|  3. PVC STATUS: BOUND (PV status: Bound)                                                |
|                                    |                                                    |
|                                    v                                                    |
|  4. POD REFERENCING PVC IS SCHEDULED                                                    |
|     - Attach/Detach Controller calls CSI Driver -> Attaches block device to Node.       |
|     - Kubelet Volume Manager mounts filesystem to /var/lib/kubelet/pods/<uid>/volumes.  |
|     - Container launches with mounted volume.                                           |
+-----------------------------------------------------------------------------------------+

---

2. Troubleshooting PVC: Pending State

When a PersistentVolumeClaim remains in Pending, execute kubectl describe pvc <pvc-name> to inspect controller events:

kubectl describe pvc my-claim

High-Frequency PVC Pending Causes & Fixes:

# Example of Default StorageClass Annotation
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard-ssd
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

---

3. Pod Stuck in ContainerCreating: Mount & Multi-Attach Errors

When a PVC is Bound, but the referencing pod remains stuck in ContainerCreating, inspect kubectl describe pod <pod-name>:

Failure Mode 1: Multi-Attach error for volume

Events:
  Warning  FailedAttachVolume  15s  attachdetach-controller  Multi-Attach error for volume "pvc-9a8b7c": Volume is already exclusively attached to one node and cannot be attached to another

• Root Cause: A ReadWriteOnce (RWO) cloud block volume (e.g., AWS EBS, GPD) can only be attached to a single virtual machine node at a time. If the previous node crashed or was abruptly partitioned without completing the detach operation, the cloud provider's control plane locks the volume to the dead node while the scheduler attempts to run the replacement pod on a new node.
• Remediation:
  1. Verify the old node status. If dead, delete the old pod with kubectl delete pod <old-pod> --force --grace-period=0.
  2. Inspect VolumeAttachment objects: kubectl get volumeattachments.
  3. If the old node is conclusively gone, follow the CSI driver and storage-provider force-detach procedure. Deleting a VolumeAttachment object is not a generic cloud detach and can risk concurrent attachment or corruption; do it only when the provider runbook or task explicitly calls for it.

Failure Mode 2: CSI Node Driver Pod Unhealthy

Events:
  Warning  FailedMount  10s  kubelet  MountVolume.SetUp failed for volume "pvc-xxxx" : rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing /var/lib/kubelet/plugins_registry/csi.sock: connect: no such file or directory"

• Root Cause: The CSI node plugin DaemonSet pod on that specific worker node has crashed or restarted, breaking the local gRPC UNIX domain socket.
• Remediation: Inspect CSI DaemonSet health: kubectl get pods -n kube-system -l app=csi-driver and check logs.

---

4. Volume Expansion Failures (FileSystemResizePending)

To increase the size of an existing PVC:

1. Ensure the underlying StorageClass has allowVolumeExpansion: true:

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: expandable-sc
   allowVolumeExpansion: true
   provisioner: ebs.csi.aws.com
   

2. Edit the PVC: kubectl edit pvc data-claim and increase .spec.resources.requests.storage from 10Gi to 20Gi.
3. Symptom: The PVC status changes to FileSystemResizePending.
   • Explanation: The controller-side volume may already be larger while node-side filesystem expansion is incomplete. Many CSI/filesystem combinations support expansion while the Pod is running; if this combination requires offline expansion, restart the consuming Pod and inspect CSI and kubelet events.

---

5. Recovering Data from Released PersistentVolumes

When a PVC is deleted, the bound PV's fate is governed by its persistentVolumeReclaimPolicy:

• Delete: Kubernetes removes the PV and asks the provisioner to delete the backing asset; completion is asynchronous and provider-dependent.
• Retain: The PV is preserved, transitioning to status Released.

+-----------------------------------------------------------------------------------------+
|                        RECLAIMING A 'RELEASED' PERSISTENTVOLUME                         |
|                                                                                         |
|  1. PVC DELETED ---> PV status transitions to 'Released'                                |
|     (PV contains data, but cannot be bound by any new PVC because                       |
|      it still holds a pointer in .spec.claimRef to the deleted PVC).                    |
|                                    |                                                    |
|                                    v                                                    |
|  2. SECURE OR SCRUB RETAINED DATA FOR THE INTENDED NEW CLAIMANT
|                                    |                                                    |
|                                    v                                                    |
|  3. STRIP CLAIMREF POINTER                                                              |
|     $ kubectl patch pv pv-data -p '{"spec":{"claimRef": null}}'                         |
|                                    |                                                    |
|                                    v                                                    |
|  4. PV STATUS TRANSITIONS TO 'Available'                                                |
|     New PVCs can now successfully bind to this PV and access retained data.             |
+-----------------------------------------------------------------------------------------+