5.5 Container Storage Interface (CSI) Architecture & Drivers

5.5 Container Storage Interface (CSI) Architecture & Drivers

In early Kubernetes releases, all storage volume plugins (including AWS EBS, GCE Persistent Disk, Azure Disk, Cinder, Ceph, and GlusterFS) were in-tree. This meant that storage vendor driver code was compiled directly into the core Kubernetes binaries—kube-apiserver, kube-controller-manager, and kubelet. This monolithic architecture introduced severe operational liabilities:

1. Release Coupling: A bug fix or feature enhancement in a third-party storage driver required upgrading the entire Kubernetes cluster.
2. Security & Stability Risks: Vendor code executed with full cluster-level and host root privileges inside kubelet and kube-controller-manager.
3. Code Bloat: Core Kubernetes maintenance was weighed down by dozens of proprietary storage SDKs.

To solve these challenges, the Kubernetes community co-developed the Container Storage Interface (CSI)—an industry-standard specification that establishes an out-of-tree, modular storage interface. Under CSI, storage plugins run as standard containerized workloads communicating with Kubernetes via gRPC over UNIX domain sockets.

---

1. The CSI Architectural Split: Controller vs. Node Plugin

A production CSI driver deployment is decoupled into two distinct operational planes:

+-----------------------------------------------------------------------------------------+
|                         CSI CONTROLLER PLUGIN (CLUSTER SCOPE)                           |
|  Deployed as a Deployment / StatefulSet in kube-system with leader election             |
|                                                                                         |
|  +--------------------+  +--------------------+  +------------------+  +-------------+  |
|  |  csi-provisioner   |  |    csi-attacher    |  |   csi-resizer    |  |csi-snapshot |  |
|  |  Watches PVCs ->   |  |  Watches           |  |  Watches PVC     |  |Watches Snap |  |
|  |  CreateVolume /    |  |  VolumeAttachment  |  |  Expansion ->    |  |CreateSnap / |  |
|  |  DeleteVolume      |  |  -> Attach/Detach  |  |  ExpandVolume    |  |DeleteSnap   |  |
|  +---------+----------+  +---------+----------+  +--------+---------+  +------+------+  |
|            |                       |                      |                   |         |
|            +-----------------------+----------------------+-------------------+         |
|                                    | gRPC over UNIX domain socket                       |
|                                    v                                                    |
|                       +--------------------------+                                      |
|                       | Vendor CSI Controller    |                                      |
|                       +-------------+------------+                                      |
+-------------------------------------|---------------------------------------------------+
                                      | Cloud Storage API (AWS / GCP / Ceph)
+-------------------------------------|---------------------------------------------------+
|                                     v Attaches physical disk to node VM                 |
|                         CSI NODE PLUGIN (NODE SCOPE)                                    |
|  Deployed as a DaemonSet on every worker node (Privileged SecurityContext)              |
|                                                                                         |
|  +--------------------+   gRPC     +--------------------------+                         |
|  |      kubelet       | <--------> | Vendor CSI Node Plugin   |                         |
|  |  Directs Volume    |            | - NodeStageVolume        |                         |
|  |  Mount Operations  |            | - NodePublishVolume      |                         |
|  +--------------------+            +------------+-------------+                         |
|                                                 | Formats & Bind-Mounts                 |
|                                                 v                                       |
|                                    +--------------------------+                         |
|                                    | Host Filesystem & PodDir |                         |
|                                    +--------------------------+                         |
+-----------------------------------------------------------------------------------------+

1. Control Plane Controller Sidecars

Maintained by the CNCF Kubernetes Storage SIG, these helper sidecars run alongside the vendor driver container:

• csi-provisioner: Watches PersistentVolumeClaim objects. Calls CreateVolume and DeleteVolume on the storage provider.
• csi-attacher: Watches VolumeAttachment API objects created by Kubernetes. Calls ControllerPublishVolume and ControllerUnpublishVolume to physically attach/detach cloud disks to/from worker node virtual machines.
• csi-resizer: Watches PVC storage request increases. Calls ControllerExpandVolume to resize underlying cloud block storage.
• csi-snapshotter: Watches VolumeSnapshot CRDs. Calls CreateSnapshot and DeleteSnapshot.
• livenessprobe: Monitors the health of the CSI plugin and exposes standard /healthz endpoints for Kubernetes probes.

2. CSI Node Plugin (DaemonSet)

Runs directly on each worker node with privileged: true and bidirectional mount propagation. The node's kubelet directly invokes the local node plugin over a UNIX domain socket (e.g., /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock).

---

2. Two-Phase Node Mounting: NodeStageVolume vs. NodePublishVolume

When a scheduled Pod requires a persistent volume, the local kubelet orchestrates a two-phase mounting protocol via the CSI Node Plugin:

   [ Attached Block Device on Node: /dev/xvdf ]
                        |
                        v
   +-------------------------------------------------------------+
   | STEP 1: NodeStageVolume                                     |
   | - Formats raw block device with ext4/xfs (if unformatted)   |
   | - Mounts device to global node staging directory:           |
   |   /var/lib/kubelet/plugins/kubernetes.io/csi/pv-name/globalmount
   +-------------------------------------------------------------+
                        |
                        v
   +-------------------------------------------------------------+
   | STEP 2: NodePublishVolume                                   |
   | - Executes a Linux bind-mount from global staging directory |
   |   into the target container's volume directory:             |
   |   /var/lib/kubelet/pods/<pod-uid>/volumes/kubernetes.io~csi/|
   +-------------------------------------------------------------+

Mounting and Teardown Sequences Compared:

---

3. Kubernetes CSI API Objects: CSIDriver, CSINode & VolumeAttachment

Kubernetes introduces dedicated API resources to register drivers and monitor attachment state:

1. CSIDriver (Driver Capabilities)

A cluster-scoped object that informs Kubernetes how to interact with the CSI plugin:

apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  name: ebs.csi.aws.com
spec:
  attachRequired: true
  podInfoOnMount: false
  volumeLifecycleModes:
    - Persistent
  fsGroupPolicy: File

2. CSINode (Node-Specific Driver Registration)

Created automatically for every worker node to record the node's storage identity and volume topology:

apiVersion: storage.k8s.io/v1
kind: CSINode
metadata:
  name: worker-node-01
spec:
  drivers:
    - name: ebs.csi.aws.com
      nodeID: i-0a1b2c3d4e5f6g7h8
      topologyKeys:
        - topology.ebs.csi.aws.com/zone
      allocatable:
        count: 28

3. VolumeAttachment (Attachment Lifecycle Debugging)

Represents the intent to attach a PV to a specific node. When a Pod is scheduled, the attachdetach-controller creates a VolumeAttachment object, which csi-attacher processes:

# Inspect active volume attachments across the cluster
kubectl get volumeattachments

# Describe attachment to diagnose why a volume is failing to attach to a node
kubectl describe volumeattachment csi-9a8b7c6d5e4f3a2b1

---

4. VolumeSnapshot Architecture & Restoration Workflows

Kubernetes provides a standardized Custom Resource Definition (CRD) framework (snapshot.storage.k8s.io/v1) to capture point-in-time copies of storage volumes.

+-----------------------------------------------------------------------------------------+
|                            VOLUMESNAPSHOT ARCHITECTURE                                  |
|                                                                                         |
|  [CLUSTER SCOPE]                                                                        |
|  +-------------------------------------+   +-----------------------------------------+  |
|  |        VolumeSnapshotClass          |   |          VolumeSnapshotContent          |  |
|  |  - driver: ebs.csi.aws.com          |   |  - snapshotHandle: snap-0123456789abcdef|  |
|  |  - deletionPolicy: Delete / Retain  |   |  - volumeSnapshotRef: prod/db-snap      |  |
|  +------------------+------------------+   +--------------------+--------------------+  |
|                     |                                           | (Bound 1-to-1)        |
|=====================|===========================================|=======================|
|  [NAMESPACE SCOPE]  v                                           v                       |
|  +--------------------------------------------------------------+                    |
|  |                      VolumeSnapshot (e.g., db-snapshot)                              |
|  |  - spec.source.persistentVolumeClaimName: production-db-pvc                          |
|  |  - status.readyToUse: true                                                           |
|  +--------------------------------------------------------------+                    |
+-----------------------------------------------------------------------------------------+

Declarative Snapshot Manifests

1. Define VolumeSnapshotClass

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: ebs-snapshot-class
driver: ebs.csi.aws.com
deletionPolicy: Delete

2. Create VolumeSnapshot from Existing PVC

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: postgres-backup-snapshot
  namespace: production
spec:
  volumeSnapshotClassName: ebs-snapshot-class
  source:
    persistentVolumeClaimName: postgres-data-pvc

3. Restore Snapshot into a New PVC via spec.dataSource

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-restored-pvc
  namespace: production
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: high-performance-ebs
  resources:
    requests:
      storage: 50Gi
  dataSource:
    name: postgres-backup-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io

---

5. CSI Troubleshooting & Diagnostic Playbook

# 1. Check health of CSI driver pods in kube-system
kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver

# 2. Inspect logs of csi-attacher sidecar when pods hang in ContainerCreating
kubectl logs -n kube-system deployment/ebs-csi-controller -c csi-attacher

# 3. Inspect logs of csi-provisioner sidecar when PVC stays Pending
kubectl logs -n kube-system deployment/ebs-csi-controller -c csi-provisioner

# 4. Check CSINode registration on worker nodes
kubectl get csinodes -o wide

# 5. Verify snapshot creation status
kubectl get volumesnapshot -n production