The New Storage of v1.36

Part IX stopped at dynamic storage with EBS CSI and snapshots. v1.36 brings a few more storage features to stable, and a from-scratch cluster (which has had EBS CSI + snapshot-controller since Articles 43–44) is the place to try them right away. This article does the two that work — OCI image volume and VolumeAttributesClass — and is direct about the third that doesn't work here, for a reason worth learning.

OCI image volume

Until now a volume has been a disk or data (Articles 41–42). v1.36 stabilizes a new volume source: the content of an OCI image mounted directly into a pod read-only. Useful for packaging config files, an ML model, or an asset bundle into a versioned image, then mounting it into the application container without stuffing them into the application image. Declared at volumes[].image:

spec:
  containers:
  - name: c
    image: busybox:1.36
    command: ["sleep", "100000"]
    volumeMounts:
    - {name: content, mountPath: /oci, readOnly: true}
  volumes:
  - name: content
    image:
      reference: registry.k8s.io/e2e-test-images/busybox:1.36.1-1
      pullPolicy: IfNotPresent

containerd pulls that image the way it pulls a container image, then mounts its filesystem into /oci:

kubectl -n stor-demo exec oci-vol -- ls /oci

bin dev etc home lib lib64 root tmp usr var

/oci contains the rootfs of the referenced image — the main pod runs busybox, but sees the content of a different image at the mount path. This was previously done with an initContainer pulling the data, or by stuffing it into the image; now it's a native volume source, using the node's own registry + image cache infrastructure.

VolumeAttributesClass: changing volume parameters dynamically

Article 43 set EBS IOPS/throughput via the StorageClass — at volume creation, fixed thereafter. VolumeAttributesClass (GA in 1.36) lets you change those parameters on a volume in use, without recreating it. Declare two classes, one low and one high:

apiVersion: storage.k8s.io/v1
kind: VolumeAttributesClass
metadata: {name: ebs-silver}
driverName: ebs.csi.aws.com
parameters: {type: gp3, iops: "3000", throughput: "125"}
---
apiVersion: storage.k8s.io/v1
kind: VolumeAttributesClass
metadata: {name: ebs-gold}
driverName: ebs.csi.aws.com
parameters: {type: gp3, iops: "4000", throughput: "250"}

Create a PVC pointing at volumeAttributesClassName: ebs-silver, and once provisioned the EBS volume has 3000 IOPS. Then move the PVC to ebs-gold:

kubectl -n stor-demo patch pvc vac-pvc --type=merge \
  -p '{"spec":{"volumeAttributesClassName":"ebs-gold"}}'

external-resizer (a sidecar in ebs-csi-controller, with the VolumeAttributesClass=true flag) calls CSI ModifyVolume, and the EBS volume changes on AWS with no disruption to the pod:

# poll PVC + describe the real EBS volume

try 1: currentVAC=ebs-silver | EBS iops=3000 throughput=125
try 2: currentVAC=ebs-gold   | EBS iops=4000 throughput=250

status.currentVolumeAttributesClassName switches to ebs-gold, and aws ec2 describe-volumes confirms IOPS 3000→4000, throughput 125→250 — the change applied straight to the block device that's mounted, no recreation, no remount. One trap we hit while testing: gp3 caps the IOPS/capacity ratio at 500/GB, so 4000 IOPS requires a volume ≥ 8GB — setting 4000 on a 4GB volume makes ModifyVolume return Iops to volume size ratio too high. VolumeAttributesClass is the standard channel for tuning storage performance by load without having to migrate data.

VolumeGroupSnapshot: GA doesn't mean it works everywhere

The third feature, VolumeGroupSnapshot (GA in 1.36), takes a consistent snapshot of multiple PVCs at the same instant — important for applications that spread data across multiple volumes (a database with data and WAL on separate disks). Its CRDs have been on the cluster since Article 44 (external-snapshotter v8.2.0 installs the groupsnapshot.storage.k8s.io group along with it):

kubectl get crd | grep groupsnapshot

volumegroupsnapshotclasses.groupsnapshot.storage.k8s.io
volumegroupsnapshotcontents.groupsnapshot.storage.k8s.io
volumegroupsnapshots.groupsnapshot.storage.k8s.io

But on this cluster it's unusable, and the reason is worth more than the feature itself: a group snapshot requires the CSI driver to implement CreateVolumeGroupSnapshot, which the AWS EBS CSI driver doesn't — because AWS EBS has no consistent group-snapshot API at the lower layer (each EBS snapshot is an independent volume). This is a recurring point from Articles 43–44: a feature being GA in the Kubernetes API still needs the underlying CSI driver to implement that capability. The CRDs are present, the snapshot-controller can enable the --enable-volume-group-snapshots flag, but if the driver can't answer the group RPC nothing happens. On a driver that does support it (some SAN/NetApp/Ceph drivers), the mechanism is like the VolumeSnapshot of Article 44 but gathers multiple PVCs by label into one VolumeGroupSnapshot. On EBS you still have to snapshot each PVC one at a time (Article 44) and handle consistency yourself (e.g., fsfreeze or stopping writes before snapshotting).

🧹 Cleanup

kubectl delete namespace stor-demo
kubectl delete volumeattributesclass ebs-silver ebs-gold

The PVC uses StorageClass ebs-sc (reclaim Delete), so deleting the namespace deletes the EBS volume on AWS too — check aws ec2 describe-volumes returns InvalidVolume.NotFound. Keep EBS CSI + snapshot-controller (Part IX infrastructure). Manifests at github.com/nghiadaulau/kubernetes-from-scratch, directory 70-storage-v136.

Wrap-up

v1.36 adds three storage pieces. OCI image volume (stable) mounts an OCI image's content as a read-only volume (volumes[].image.reference) — we saw the other image's rootfs at /oci, useful for packaging versioned config/model/asset. VolumeAttributesClass (GA) changes the parameters of a volume in use without recreation: moving a PVC from ebs-silver to ebs-gold, external-resizer calls CSI ModifyVolume, and the EBS volume changes IOPS 3000→4000 + throughput 125→250 on AWS with no disruption to the pod (mind the max IOPS/GB ratio of 500). VolumeGroupSnapshot (GA) takes a consistent snapshot of multiple PVCs — the CRDs have been present since Article 44 but it doesn't run on this cluster because EBS CSI doesn't implement CreateVolumeGroupSnapshot; the recurring lesson from Part IX: a feature being GA in the Kubernetes API still depends on the underlying CSI driver's capability.

Article 71 closes Part XIV with two operational/security features of v1.36: querying node logs straight through the kubelet (no need to SSH into each machine), and tightening kubelet API permissions at fine granularity — both touching components we built ourselves in Part I.