Kubernetes Persistent Storage

TL;DR

• Pod storage is ephemeral; persistent storage separates infrastructure (PV) from pod requests (PVC)
• StorageClass defines how to dynamically provision storage via CSI drivers; no manual PV creation needed
• WaitForFirstConsumer volumeBindingMode prevents zone-mismatch deadlocks; Immediate creates volume before pod scheduling
• Access modes vary: ReadWriteOnce for block storage (EBS, Azure Disk), ReadWriteMany only for network filesystems (NFS, EFS)
• reclaimPolicy: Delete removes underlying cloud disk on PVC deletion; Retain leaves disk behind for manual cleanup
• Enable allowVolumeExpansion: true in StorageClass to allow PVC size increases via kubectl patch

Core Concepts

The Three-Layer Storage Model

Pods reference PersistentVolumeClaims (namespaced requests), which bind to PersistentVolumes (cluster-scoped resources backed by actual storage). StorageClass acts as a factory that dynamically provisions volumes on demand.

PersistentVolume (PV): Cluster-scoped resource representing actual storage (EBS volume, NFS export, etc.). Created manually or by a CSI provisioner. Remains independent of pods; survives pod deletion.

PersistentVolumeClaim (PVC): Namespaced request for storage with size and access mode. Pods mount PVCs, not PVs directly. Acts as a claim ticket that binds to an available PV.

StorageClass: Defines provisioning rules: which CSI driver, cloud parameters (volume type, IOPS), reclaim policy, and volumeBindingMode. Enables automatic PV creation on demand.

Access Modes

• ReadWriteOnce (RWO): Mounted read-write on a single node; multiple pods on that node can share it
• ReadOnlyMany (ROX): Mounted read-only on many nodes simultaneously
• ReadWriteMany (RWX): Mounted read-write on many nodes; requires network filesystem (NFS, EFS, CephFS); block storage cannot provide this
• ReadWriteOncePod (RWOP): Strictly one pod cluster-wide; newer, stricter than RWO

volumeBindingMode

Immediate: PV created when PVC is created, potentially in wrong zone if pod lands elsewhere → scheduling deadlock.

WaitForFirstConsumer: PV creation delayed until pod is scheduled, ensuring volume lands in correct zone → no deadlock, eliminates zone-mismatch failures.

Reclaim Policies

Delete: Deletes underlying storage when PVC is deleted. Data is generally unrecoverable without snapshots. Default for dynamically provisioned volumes.

Retain: Leaves underlying storage behind when PVC deleted; disk enters “Released” state requiring manual admin intervention. Use for critical data.

Comparison Table

Command Reference

StorageClass Definition

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "3000"
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

PVC Template

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: db-data
spec:
  storageClassName: fast-ssd
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 20Gi

Pod Mounting a PVC

apiVersion: v1
kind: Pod
metadata:
  name: postgres
spec:
  containers:
  - name: db
    image: postgres:15
    volumeMounts:
    - name: data
      mountPath: /var/lib/postgresql
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: db-data

Essential Commands

# List storage classes and see which is default
kubectl get storageclass

# Check PVC binding status
kubectl get pvc
kubectl describe pvc db-data

# Diagnose pending PVC
kubectl describe pvc <pvc-name>
# Look for: "no persistent volumes available" or provisioner errors

# Expand PVC size (StorageClass must have allowVolumeExpansion: true)
kubectl patch pvc db-data -p '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'

# Change reclaim policy on existing PV
kubectl patch pv <pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

# Check actual cloud storage behind a PV
kubectl describe pv <pv-name>
# Look for source (awsElasticBlockStore, nfs, etc.)

Common Pitfalls

Pitfall: Immediate volumeBindingMode Causes Scheduling Deadlock Using volumeBindingMode: Immediate on cloud block storage (EBS, GCE PD, Azure Disk) causes PV to be created in an arbitrary zone when PVC is created, before pod scheduling.

• Why: If pod’s affinity constraints or node availability force it into a different zone than the pre-created disk, the disk cannot attach (zone-locked storage), and pod remains permanently Pending.
• Fix: Use volumeBindingMode: WaitForFirstConsumer for block storage so PV creation defers until pod is scheduled, guaranteeing correct zone.

Pitfall: Deleting PVC Destroys Unbackedup Data Using reclaimPolicy: Delete (the default) on dynamically provisioned storage, then accidentally deleting a PVC in production.

• Why: PVC deletion immediately triggers CSI driver to delete underlying cloud storage; data is permanently lost unless separate snapshots or backups exist.
• Fix: Use reclaimPolicy: Retain for critical databases or stateful workloads. Maintain independent backup strategy (snapshots, application-level backups) separate from PVC lifecycle.

Pitfall: ReadWriteMany Assumed for Multi-Replica Deployments Designing a Deployment with multiple replicas all mounting the same PVC with ReadWriteMany access mode, discovering too late that the StorageClass uses block storage.

• Why: Block storage (EBS, Azure Disk, GCE PD) only supports ReadWriteOnce. RWX requires network filesystems (NFS, EFS, CephFS); attempting to mount RWO storage with multiple pods fails.
• Fix: Verify StorageClass CSI driver’s supported access modes. For multi-writer workloads, use network filesystem-backed StorageClass or separate PVCs per replica.

Pitfall: PVC Expansion Fails Silently Patching a PVC to expand its size, but expansion never happens.

• Why: StorageClass must have allowVolumeExpansion: true. If missing, expansion requests are silently ignored or rejected.
• Fix: Enable allowVolumeExpansion: true in StorageClass before attempting PVC size increases.

Interview Questions

Q — What is the relationship between a StorageClass, PVC, and PV? Why is this separation useful? StorageClass is a template defining how to provision storage (CSI driver, cloud parameters, reclaim policy). A PVC is a namespaced request for storage; when created, it triggers the StorageClass’s provisioner to dynamically create a matching PV (cluster-scoped resource). This separation allows applications to request storage without knowing infrastructure details. Pods never reference PVs directly—only PVCs. If the pod moves to a different namespace, it references a different PVC that may bind to a different PV, all transparently.

Q — Explain the failure mode of volumeBindingMode: Immediate on cloud block storage. How does WaitForFirstConsumer prevent it? With Immediate binding, the CSI driver creates the PV and cloud disk instantly when the PVC is created—potentially in zone A. Later, the pod’s constraints (node affinity, resource requests) force scheduling onto a node in zone B. Since zone-locked block storage (EBS, GCE PD) cannot be cross-zone attached, the pod permanently hangs Pending. The disk exists but is unreachable. WaitForFirstConsumer defers PV creation until a pod actually references the PVC, so the provisioner knows the pod’s target zone beforehand and creates the disk there.

Q — You delete a PVC with reclaimPolicy: Delete by accident. Is the data recoverable? Generally no. With Delete reclaim policy, deleting the PVC immediately triggers the CSI driver to delete not just the Kubernetes PV object, but the underlying cloud storage volume itself. The disk and its data are gone unless independent snapshots or application-level backups exist. Critical stateful workloads should use reclaimPolicy: Retain (disk survives PVC deletion in “Released” state, allowing reuse) or maintain external backup strategies separate from the PVC lifecycle.