Blog
Thoughts on engineering, design, and building great products.
GC, cgroup v2, Swap, and Graceful Node Shutdown
Kubelet does a lot at the node level that we rarely look at while things run fine. This article inspects four of those on a real worker: cleaning up old images when the disk fills, placing each pod in the right cgroup v2 branch and enforcing limits via memory.max/cpu.max, why swap is blocked by default, and graceful node shutdown — the thing that decides whether pods get yanked or shut down cleanly when a node powers off.
Upgrades and Version Skew
Upgrading Kubernetes isn't a single sweep — it follows an order, because components are allowed to skew versions within strict limits: kubelet may be up to three minors older than the apiserver but never newer. This article inspects the cluster's version skew, explains why the apiserver must be upgraded first, then drills the hardest part of upgrading a node: cordon, drain, uncordon — on the real worker-0, fully reversible.
Backing Up etcd and Rotating Certificates
etcd holds all of the cluster's state; lose it and you lose the cluster. Part XIII opens at exactly the scariest spot when things break. We take an etcd snapshot, verify it's valid, and restore it into a fresh data directory to prove the snapshot is usable — all without touching the running etcd. Then we inspect the expiry of the certificate set built in Article 4 and discuss rotating them before they expire.
Device Plugins and Extended Resources
Pods can request CPU and memory, but what about a GPU, a high-speed NIC, or an FPGA? A device plugin is how a node advertises hardware beyond CPU/memory as an extended resource for pods to request and the scheduler to divvy up. This article stands up a real device plugin, captures the full gRPC flow it uses to register with kubelet and advertise devices, watches kubelet call Allocate when a pod runs, then uses the underlying mechanism to see the scheduler split it like CPU.
API Aggregation: Bolting On a Second API Server
A CRD adds a new kind that the main API server itself stores in etcd. API aggregation goes further: it bolts a second API server in behind the main one, serving an API group that it stores and computes on its own terms. Our cluster has been running one example since Article 39 — metrics-server. This article examines it as an aggregated API: how the APIService registers it, where requests get proxied, and why the CPU/memory figures it returns never sit in etcd.
Operator: CRD Plus a Reconcile Loop
A CRD gives us a new data type, but creating a custom resource makes nothing happen. An operator joins a CRD with a controller running a loop: it watches the custom resource and acts to bring reality in line with desire. This article builds a real operator from scratch — an Echo CRD and an in-pod controller — then watches it create a Deployment when we create an Echo, scale when we edit replicas, and let the Deployment be cleaned up when we delete the Echo.
Admission Webhook: Wedge Into the Write Path
Article 54 used a built-in admission controller (Pod Security). This article writes one of your own: an HTTPS service the API server calls before storing each object, returning allow or deny. We build a real validating webhook in Python — self-sign a cert, make the API server trust it via caBundle, and require every pod to have a team label. A pod missing the label is rejected immediately; a pod in a namespace out of scope is untouched.