Blog
Thoughts on engineering, design, and building great products.
Operator: CRD Plus a Reconcile Loop
A CRD gives us a new data type, but creating a custom resource makes nothing happen. An operator joins a CRD with a controller running a loop: it watches the custom resource and acts to bring reality in line with desire. This article builds a real operator from scratch — an Echo CRD and an in-pod controller — then watches it create a Deployment when we create an Echo, scale when we edit replicas, and let the Deployment be cleaned up when we delete the Echo.
Admission Webhook: Wedge Into the Write Path
Article 54 used a built-in admission controller (Pod Security). This article writes one of your own: an HTTPS service the API server calls before storing each object, returning allow or deny. We build a real validating webhook in Python — self-sign a cert, make the API server trust it via caBundle, and require every pod to have a team label. A pod missing the label is rejected immediately; a pod in a namespace out of scope is untouched.
CustomResourceDefinition: Add Your Own Kind
Part XII shifts from using Kubernetes to extending it. The first article is CustomResourceDefinition — declare a new kind of object, and the API server immediately serves it like a native resource: kubectl get works, it validates against a schema, it stores in etcd. We build a Widget CRD with type and value-range constraints, create a valid custom resource, watch two invalid ones get rejected, then update status through a separate subresource.
Secrets, the Detour and Hardening
Part XI closes out at Secrets and the holes still left. We read etcd directly to confirm Secrets are encrypted at-rest since Article 5, then build a real detour: a ServiceAccount with no permission to read a Secret still extracts its value by creating a pod that mounts that Secret and reading the log. The article ends with a table of hardening steps for a self-built cluster — which are done in the series, which are still missing.
Seccomp, AppArmor and Capabilities
Article 54 made pods declare runAsNonRoot, drop ALL capabilities, seccomp RuntimeDefault — but that's only Kubernetes-level policy. This article goes to the kernel layer to see what they actually do: read /proc/self/status from two pods, one default and one hardened, comparing CapEff, Seccomp, NoNewPrivs, AppArmor. Then prove by hand that dropping a capability blocks a specific operation — chown is denied even when the container still runs as root.
Pod Security Standards and Admission
RBAC decides who can create a pod, not what that pod asks for. A pod running privileged or borrowing hostNetwork is an escape hatch onto the node. Pod Security Admission blocks it at creation: one label on a namespace, the API server measures the pod against three levels — privileged/baseline/restricted — and rejects violators. This article turns restricted on for a namespace, watches a plain pod get kicked out, writes a compliant pod that runs, then tries warn mode.
ServiceAccount and Bound Tokens
Articles 51–52 used ServiceAccount without dissecting it. This article gets into the mechanism: every namespace has a default SA, the kubelet auto-injects a short-lived token via a projected volume, and that token is bound to the exact pod and node. To prove it's truly bound, we grab the token in a running pod, call the API successfully, then delete the pod — the old token immediately becomes 401. Plus how to turn off auto-mount and read the JWT claims.