DevOpsStorageKubernetesCSI

StorageClass, dynamic provisioning, and EBS CSI

K
Kai··6 min read
Kubernetes From ScratchPart 44 of 73
PreviousPersistentVolume and PersistentVolumeClaimNextVolumeSnapshot and CSI snapshot

Article 42 exposed a weakness: the admin has to create the PV by hand first, guessing in advance how many volumes users need, what capacity. At real scale that's impossible. Dynamic provisioning flips it: the user declares only a PVC, and the system creates the PV itself — even calling the cloud provider to create a brand-new volume. Its heart is StorageClass + a CSI driver. This article installs a real EBS CSI driver on our AWS cluster and traces every link: what watches what, what calls what for an EBS volume to be born from a line of YAML.

StorageClass: the storage "class" and the provisioner

The docs: "A StorageClass provides a way for administrators to describe the classes of storage they offer." The most important field is provisioner — "determines what volume plugin is used for provisioning PVs. This field must be specified." This points at who will create the PV. For EBS, the provisioner is ebs.csi.aws.com (the CSI driver's name). CSI (Container Storage Interface) is the plug-in standard: instead of stuffing every storage type into Kubernetes code, each provider writes its own driver to the CSI standard. The EBS CSI driver is a program running in the cluster (a pod), not core code.

Install the EBS CSI driver (and IAM for the nodes)

The driver needs AWS permissions to create/attach EBS. Our EC2 nodes don't have IAM yet — so first attach an IAM role:

# create a role trusting ec2, attach the managed policy AmazonEBSCSIDriverPolicy,
# create an instance profile, associate it with the 2 workers
aws iam create-role --role-name k8s-scratch-ebs-csi --assume-role-policy-document file://ec2-trust.json
aws iam attach-role-policy --role-name k8s-scratch-ebs-csi \
  --policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy
aws iam create-instance-profile --instance-profile-name k8s-scratch-ebs-csi
aws iam add-role-to-instance-profile --instance-profile-name k8s-scratch-ebs-csi --role-name k8s-scratch-ebs-csi
aws ec2 associate-iam-instance-profile --instance-id <worker-0> --iam-instance-profile Name=k8s-scratch-ebs-csi
aws ec2 associate-iam-instance-profile --instance-id <worker-1> --iam-instance-profile Name=k8s-scratch-ebs-csi

The CSI driver pod will pull credentials from the node's IAM role via IMDS. Then install the driver (kustomize overlay, pinned to v1.50.0):

kubectl apply -k "github.com/kubernetes-sigs/aws-ebs-csi-driver/deploy/kubernetes/overlays/stable/?ref=v1.50.0"
kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver

ebs-csi-controller-...-lh5sc   6/6   Running   worker-0
ebs-csi-controller-...-qd476   6/6   Running   worker-1
ebs-csi-node-h7ps5             3/3   Running   worker-1
ebs-csi-node-kwkfc             3/3   Running   worker-0

Two parts worth noting. ebs-csi-controller (a Deployment) holds 6/6 containers — most notable are the sidecars external-provisioner (watches PVCs, calls the driver to create a volume), external-attacher (attaches the volume to a node), resizer, snapshotter, plus the main driver container. ebs-csi-node (a DaemonSet, one pod/node — exactly the model from Article 26) handles mounting the volume on each node. Installing the driver also creates a CSIDriver object ebs.csi.aws.com registering the driver with the cluster.

StorageClass + PVC + the provisioning chain

Create a StorageClass pointing at that provisioner, then a PVC referencing it:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: {name: ebs-sc}
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer    # wait for the pod before creating the volume
reclaimPolicy: Delete
parameters: {type: gp3}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: pvc-dyn}
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: ebs-sc                 # <-- points at the StorageClass, NOT a PV
  resources: {requests: {storage: 2Gi}}

kubectl get pvc pvc-dyn ; kubectl get pv

NAME      STATUS    VOLUME   STORAGECLASS
pvc-dyn   Pending   ""       ebs-sc

No resources found        # <-- NO PV yet

The PVC is Pending, with no PV. It differs from Article 42 in two ways: the PVC declares storageClassName (not a fixed capacity to match an existing PV), and no PV exists yet. The PVC stays Pending because of volumeBindingMode: WaitForFirstConsumer, per the docs: "delay the binding and provisioning of a PersistentVolume until a Pod using the PersistentVolumeClaim is created." This matters for EBS: an EBS volume can only attach within one AZ, so it has to wait to know which node/AZ the pod runs on before creating the volume in the right place. Create a pod using the PVC:

  volumes:
  - {name: v, persistentVolumeClaim: {claimName: pvc-dyn}}

Now the chain of cause and effect plays out — this is the core part to trace carefully:

1. user        ──creates──▶ PVC pvc-dyn (storageClassName: ebs-sc)   [Pending]
2. user        ──creates──▶ Pod using pvc-dyn
3. scheduler   ── picks a node for the Pod (e.g. worker-1) → "first consumer" appears
4. external-provisioner (sidecar in ebs-csi-controller)
                 watches PVC + sees it now has a consumer → calls CSI CreateVolume
5. EBS CSI driver ──calls AWS API──▶ AWS CREATES an EBS volume (gp3, 2Gi, right AZ)
6. external-provisioner ──creates──▶ PV (volumeHandle = vol-id) → binds with PVC
7. external-attacher ──calls AWS──▶ attaches EBS to instance worker-1
8. ebs-csi-node (DaemonSet@worker-1) ── mounts the volume into the pod
9. kubelet     ── pod Running, sees /data

Verify each link:

kubectl get pv     # PV now AUTO-BORN
kubectl get pv <name> -o jsonpath='{.spec.csi.volumeHandle}'
aws ec2 describe-volumes --volume-ids <vol-id>
kubectl exec dyn-pod -- cat /data/f.txt

NAME                     CAPACITY   RECLAIM   STATUS   CLAIM             STORAGECLASS
pvc-143e1484-...         2Gi        Delete    Bound    default/pvc-dyn   ebs-sc

volumeHandle = vol-07fb135ac8ea0717a
vol-07fb135ac8ea0717a   2   gp3   in-use   ap-southeast-1a   i-0a33782c408f5bf09

dynamic-ebs-data

Everything lines up: the PV is auto-born named pvc-<uid> (nobody typed it), volumeHandle points at a real EBS volume vol-07fb... (2GB gp3, in-use, attached to instance i-0a33... = worker-1 where the pod runs). One line of PVC made AWS spawn a volume and attach it to the right machine. This is the fundamental difference from static (Article 42): no admin created the PV — external-provisioner did it for us, and external-attacher + ebs-csi-node handle attach/mount.

reclaimPolicy Delete: deleting the PVC deletes the volume too

The StorageClass sets reclaimPolicy: Delete (the default for dynamic provisioning). Delete the PVC:

kubectl delete pod dyn-pod ; kubectl delete pvc pvc-dyn
kubectl get pv                                   # No resources found
aws ec2 describe-volumes --volume-ids vol-07fb135ac8ea0717a

InvalidVolume.NotFound: The volume 'vol-07fb135ac8ea0717a' does not exist.

PVC gone → PV deleted → and the CSI driver calls AWS to delete the EBS volume too (Delete). Handy for ephemeral, but dangerous for precious data — set Retain (Article 42) if you need to keep it. This is also why StatefulSet (Article 25) does not delete PVCs on scale-down: to prevent accidental data loss.

🧹 Cleanup

kubectl delete pod dyn-pod --now ; kubectl delete pvc pvc-dyn    # -> EBS volume auto-deleted

Deleted the test workload (the EBS volume auto-deleted). Keep the EBS CSI driver + StorageClass ebs-sc + IAM role (used again for Article 44 snapshots and StatefulSet with real storage). The cluster now has CoreDNS + metrics-server + ebs-csi. Manifests at github.com/nghiadaulau/kubernetes-from-scratch, directory 43-storageclass-csi.

Wrap-up

Dynamic provisioning drops Article 42's admin-creates-PV-by-hand step: the user creates only a PVC pointing at a StorageClass, the system spawns the PV. The StorageClass declares provisioner (who creates) + parameters (volume type) + volumeBindingMode (Immediate or WaitForFirstConsumer — wait for the pod to pick the right AZ) + reclaimPolicy. The CSI driver (we installed real EBS CSI v1.50.0 + an IAM role for the nodes) is a program running in the cluster: the external-provisioner sidecar watches PVCs → calls the driver → AWS creates an EBS volume → the PV is auto-born (volumeHandle = vol-id) → binds; external-attacher attaches the volume to the instance; ebs-csi-node (DaemonSet) mounts it. We saw one line of PVC spawn a real vol-07fb... volume, and reclaimPolicy: Delete meant deleting the PVC deleted the volume too. The chain PVC→provisioner→AWS→PV→attacher→node is the skeleton of every CSI driver, not just EBS.

Article 44 uses this same driver for VolumeSnapshot: take a point-in-time snapshot of a PVC (CSI calls AWS to create an EBS snapshot), then restore a new PVC from the snapshot — backing up/cloning data at the storage layer.

Kubernetes From ScratchPart 44 of 73
PreviousPersistentVolume and PersistentVolumeClaimNextVolumeSnapshot and CSI snapshot

Related Posts

Inside Hubble: From eBPF Events to Clust...
KubernetesCilium

Inside Hubble: From eBPF Events to Cluster-Wide Network Flows

Hubble lets us see every connection in a Kubernetes cluster by pod name, service, and policy verdict — without a sidecar in any pod. This article dissects the mechanism: Cilium's eBPF datapath (the 74 sched_cls programs from Article 12) calls bpf_perf_event_output to push events into a perf ring buffer named cilium_events; cilium-agent reads them out with numeric identities; then Hubble enriches them — turning identity 18203 into kube-system/coredns — via the identity-to-label store.

K
KaiMay 24, 2026· 1 views
The Tetragon Way: From Observe to Enforc...
SecurityKubernetes

The Tetragon Way: From Observe to Enforce with bpf_send_signal

Tetragon is the Cilium ecosystem's runtime security tool: it observes with kprobe/tracepoint (the very hooks Part II used) and then enforces inside the kernel. Its enforcement uses two helpers — bpf_send_signal sends SIGKILL to kill a process, and bpf_override_return overrides a syscall's return value. This article rebuilds that: an exec tracepoint calls bpf_send_signal(SIGKILL) the moment a process runs — a forbidden binary gets exit 137, a normal binary still runs. No LSM, no reboot.

K
KaiMay 24, 2026· 1 views