Blog
Thoughts on engineering, design, and building great products.
Leader Election, Addons, and Node Autoscaling
The cluster runs three control planes, but controller-manager and scheduler keep only one instance active at a time — if all three acted, they'd step on each other. This article looks at the mechanism that prevents that: leader election via a Lease, and proves real failover by taking the leader down and watching another controller win the lock. Then it closes Part XIII with addons and node autoscaling — adding and removing machines by load.
Logging Architecture
kubectl logs sounds simple, but behind it is a chain: the container writes stdout/stderr to a file on the node, kubelet reads that file back, and rotates it when full. This article traces a real log line from kubectl down to the file on the worker's disk, examines the CRI format and the symlink, then separates the two kinds of logs in a self-built cluster — container logs and system-component logs via journald — and why the cluster needs an agent to collect them.
GC, cgroup v2, Swap, and Graceful Node Shutdown
Kubelet does a lot at the node level that we rarely look at while things run fine. This article inspects four of those on a real worker: cleaning up old images when the disk fills, placing each pod in the right cgroup v2 branch and enforcing limits via memory.max/cpu.max, why swap is blocked by default, and graceful node shutdown — the thing that decides whether pods get yanked or shut down cleanly when a node powers off.
Upgrades and Version Skew
Upgrading Kubernetes isn't a single sweep — it follows an order, because components are allowed to skew versions within strict limits: kubelet may be up to three minors older than the apiserver but never newer. This article inspects the cluster's version skew, explains why the apiserver must be upgraded first, then drills the hardest part of upgrading a node: cordon, drain, uncordon — on the real worker-0, fully reversible.
Backing Up etcd and Rotating Certificates
etcd holds all of the cluster's state; lose it and you lose the cluster. Part XIII opens at exactly the scariest spot when things break. We take an etcd snapshot, verify it's valid, and restore it into a fresh data directory to prove the snapshot is usable — all without touching the running etcd. Then we inspect the expiry of the certificate set built in Article 4 and discuss rotating them before they expire.