1.3 Storage: volumes, claims, classes

The lab has no storage layer to install because kind ships one: the standard StorageClass backed by the local-path provisioner. That is enough to exercise every concept the exam touches, and its limitations are themselves instructive.

needsmake upmake api

Orientation

competency 1.1 · architecture best practices

Storage tasks on a performance exam are rarely "install a CSI driver". They are: this claim is Pending, say why; this pod lost its data, say why; make this workload survive a reschedule; grow this volume. All four are answered from four fields you can recite.

Exam angle

The graders can only see objects and their status. So the tell for a storage task is almost always a status: PVC Pending, PV Released, pod stuck ContainerCreating with a mount error in events. Learn to map those three states to their causes and you have the competency.

The model: three objects, one relationship

PV · PVC · StorageClass

A PersistentVolume is a piece of real storage. A PersistentVolumeClaim is a request for one. A StorageClass is the recipe for making PVs on demand, and its provisioner field names the code that does it. Static provisioning (an admin pre-creates PVs) still exists but dynamic is the assumed default: the PVC references a class, the provisioner makes the PV, the two bind one-to-one and exclusively.

pod ──mounts──▶ PVC ──binds 1:1──▶ PV ──backed by──▶ real disk / host path / cloud volume
                  │                  ▲
                  └── storageClassName ──▶ StorageClass ──▶ provisioner creates the PV
                                            (binding mode, reclaim policy, expansion, params)

Underneath, CSI is the plugin interface every modern driver implements: a controller component (provision, attach, snapshot) and a node component (mount). You will not be asked to write one, but knowing the split explains error locations: "failed to provision" is a controller-side message, "failed to mount" is node-side, and they point at different logs.

The fields that decide exam tasks

Field	Values	What it changes
accessModes	RWO · ROX · RWX · RWOP	A claim binds only to a PV offering what it asks. local-path only does RWO, so an RWX claim here pends forever; recognising why is the skill.
volumeBindingMode	Immediate · WaitForFirstConsumer	WFFC keeps the PVC Pending until a pod uses it, so topology can be considered. The `standard` class here uses it, so you meet this "problem" immediately, and it is not a problem.
persistentVolumeReclaimPolicy	Delete · Retain	Delete throws data away with the claim; Retain keeps the PV in Released, and it will not rebind until someone clears `spec.claimRef`. Released-but-unusable PVs are a classic troubleshooting scenario.
allowVolumeExpansion	true · false	Only classes that set it let you grow a PVC by editing `spec.resources.requests.storage`. Shrinking is never allowed, anywhere.
volumeMode	Filesystem · Block	Block hands the raw device to the container. Databases sometimes want it; nothing else does.

Access modes are a claim, not a guarantee

RWX means "this PV supports many nodes mounting it read-write", and it is a property of the backing storage, not a wish you can express. Nothing in Kubernetes enforces that two pods writing an RWO volume from the same node behave sensibly, and nothing turns a local disk into a shared filesystem because you asked nicely. RWOP (ReadWriteOncePod) is the strict one: exactly one pod, enforced by the kubelet, which is how you stop two replicas corrupting a single-writer database.

Lifecycle and the states you will be asked to explain

Pending → Bound → Released

Symptom	Usual cause	Check
PVC Pending, no events	WaitForFirstConsumer, no pod yet	normal: schedule a consumer
PVC Pending, provisioner events	unsupported access mode, no capacity, bad class name	kubectl describe pvc
Pod ContainerCreating forever	mount/attach failure, node-side	kubectl describe pod → events
PV Released, not rebinding	Retain policy leaves `claimRef` populated	kubectl patch pv … claimRef=null
PVC Terminating forever	`kubernetes.io/pvc-protection` finalizer: a pod still uses it	kubectl get pods -o json \| grep claimName
Resize stuck	class lacks expansion, or filesystem resize needs a pod restart	status.conditions on the PVC

Two protection finalizers exist for good reasons and both look like bugs the first time: pvc-protection blocks deleting a claim that a pod mounts, and pv-protection blocks deleting a bound PV. The fix is never to strip the finalizer first; it is to remove the consumer, then let the controller clean up. Stripping finalizers to make an object disappear is the storage equivalent of pulling the disk out.

StatefulSets, where storage semantics become visible

volumeClaimTemplates gives every replica its own PVC, named <template>-<sts>-<ordinal>. Deleting the StatefulSet does not delete those PVCs (unless you set persistentVolumeClaimRetentionPolicy, which newer versions offer), so scaling back up reattaches the old data. That retention is deliberate and it is the entire reason StatefulSets exist rather than "a Deployment with a volume": stable identity, stable storage, ordered rollout.

And the part GitOps cannot do for you: data. "Delete the namespace and the controller rebuilds it" restores manifests, never the contents of a PV. That gap is filled by CSI snapshots for point-in-time copies within a cluster, by a backup tool (Velero is the common one; it snapshots volumes and exports object state) for anything that must survive the cluster itself, and by the database operator's own backup story where one exists. Being able to say which of those three you are relying on is the whole DR answer.

Snapshots round out the vocabulary: VolumeSnapshotClass, VolumeSnapshot, VolumeSnapshotContent: the same three-object shape as class/claim/volume, and the way a CSI driver exposes point-in-time copies. local-path has no snapshotter, so this lab teaches the nouns and CloudNativePG teaches the backup story instead.

What a platform engineer actually decides

classes as product

Section 3.1's "APIs as products" idea lands here concretely: StorageClasses are a platform API. You are choosing the menu your tenants order from, and each entry encodes a durability, performance and cost decision they should not have to make.

Name classes for intent (fast-ssd, cheap-hdd, shared-rwx), never for the implementation that happens to back them today.
Set exactly one default class (storageclass.kubernetes.io/is-default-class) and know what it costs, because every PVC without an explicit class silently buys it. If two are marked default, the DefaultStorageClass admission plugin, not the scheduler, picks the most recently created one, which is worse than an error: it is silent, and it changes under you the next time someone adds a class.
Prefer WaitForFirstConsumer in any topology-aware environment, or you will provision volumes in zones your pods cannot reach.
Delete for ephemeral tenant workloads, Retain for anything whose loss ends up in a postmortem.
Cap storage per tenant with quota: requests.storage and <class>.storageclass.storage.k8s.io/requests.storage: per-class quota is how you stop everyone ordering the expensive one (section 1.4).

Command reflex

kubectl get sc first, every time: it shows provisioner, reclaim policy, binding mode and expansion in one line, which is four of the five fields above. Then kubectl get pvc -A and look for anything not Bound.

Exercises

tick the dot when its check passes

kubectl get storageclass standard -o yaml   # read provisioner, bindingMode, reclaimPolicy
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: scratch, namespace: default }
spec:
  accessModes: [ReadWriteOnce]
  resources: { requests: { storage: 1Gi } }
  storageClassName: standard
EOF
kubectl get pvc scratch    # Pending, and that is CORRECT

Now consume it:

kubectl run writer --image=busybox:1.37 --restart=Never \
  --overrides='{"spec":{"containers":[{"name":"writer","image":"busybox:1.37","command":["sh","-c","echo survived > /data/proof && sleep 3600"],"volumeMounts":[{"name":"d","mountPath":"/data"}]}],"volumes":[{"name":"d","persistentVolumeClaim":{"claimName":"scratch"}}]}}'
kubectl get pvc scratch    # Bound, seconds after the pod scheduled

verify: persistence the honest way: delete the pod, recreate it with a command of cat /data/proof, and kubectl logs writer must print survived.

Create a PVC identical to the above but accessModes: [ReadWriteMany], plus a pod that mounts it (without a consumer, WaitForFirstConsumer keeps the events silent and you learn nothing). Now it pends with a reason: kubectl describe pvc events show the provisioner refusing, because local-path only does RWO.

verify: you can state the fix (RWO, or a class whose provisioner supports RWX) and then delete pod and claim. The general lesson: a Pending PVC under WaitForFirstConsumer is normal until a pod consumes it; only then does silence become a finding.

make api installs the CloudNativePG operator; a Postgres exists once you create one (section 3.3 does, and it is worth jumping ahead for its first exercise). With one running:

kubectl get pvc -A | grep -v Bound        # unbound + consumed = a finding (your own scratch claims excepted)
kubectl get pvc -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,SC:.spec.storageClassName,MODE:.spec.accessModes[0],SIZE:.spec.resources.requests.storage'

verify: you can point at each PVC and name what created it (a volumeClaimTemplate, an operator, a human). Note that a CNPG instance's PVC survives kubectl delete pod of the instance, and the new pod mounts the same data. That is the operator relying on exactly the PVC semantics above.

Create a PV of type hostPath (1Gi, RWO, storageClassName: manual), a PVC requesting it by the same class name, and show they bind with no provisioner involved.

verify: kubectl get pv shows STATUS Bound and CLAIM pointing at your PVC. Then delete the PVC and explain what the PV's new status means given its reclaim policy.

Self-check

answer before opening

A PVC has been Pending for ten minutes and describe shows no events at all. Bug or not?

Not a bug if the class uses WaitForFirstConsumer and nothing mounts the claim yet: no scheduling decision exists, so there is nothing to provision and nothing to report. It becomes a finding the moment a pod references it and the claim stays Pending; then the events appear and name the real reason.

A PV sits in Released and your new, identical PVC will not bind to it. Why, and what is the fix?

Retain leaves spec.claimRef pointing at the deleted claim, and a PV with a claimRef is spoken for. Clear it (kubectl patch pv <name> -p '{"spec":{"claimRef":null}}') and it returns to Available. Deciding whether the old data should be wiped first is the actual judgement call.

You delete a StatefulSet and its PVCs remain. Accident or design?

Design. The claims outlive the set so scaling back up reattaches the same data, and so an accidental delete does not destroy the database. Newer clusters can opt in to cleanup via persistentVolumeClaimRetentionPolicy (whenDeleted / whenScaled).

Two pods on different nodes must share a directory. What do you need, and what will this lab do?

A class whose provisioner supports RWX (NFS, CephFS, a cloud file service). local-path cannot, so the claim pends and the events say so. The honest lab answer is "not possible here" plus the fix you would apply in a real cluster, which is also the right exam answer when the storage cannot do what the task implies.

Which quota lines cap tenant storage, and why is per-class quota interesting?

requests.storage and persistentvolumeclaims cap the total and the count; <class>.storageclass.storage.k8s.io/requests.storage caps a specific class. The per-class form is how a platform team offers a fast expensive tier without every tenant defaulting to it: a pricing decision expressed as a Kubernetes object.

Docs to know your way around

study time, not exam time

kubernetes.io: Persistent Volumes (the access-modes and reclaim tables), Storage Classes, Volume Snapshots, StatefulSet volumeClaimTemplates.
Offline: kubectl explain pvc.spec, kubectl explain sc, kubectl get sc -o wide, and PVC status.conditions when a resize misbehaves.