Blog
Thoughts on engineering, design, and building great products.
API Aggregation: Bolting On a Second API Server
A CRD adds a new kind that the main API server itself stores in etcd. API aggregation goes further: it bolts a second API server in behind the main one, serving an API group that it stores and computes on its own terms. Our cluster has been running one example since Article 39 — metrics-server. This article examines it as an aggregated API: how the APIService registers it, where requests get proxied, and why the CPU/memory figures it returns never sit in etcd.
Operator: CRD Plus a Reconcile Loop
A CRD gives us a new data type, but creating a custom resource makes nothing happen. An operator joins a CRD with a controller running a loop: it watches the custom resource and acts to bring reality in line with desire. This article builds a real operator from scratch — an Echo CRD and an in-pod controller — then watches it create a Deployment when we create an Echo, scale when we edit replicas, and let the Deployment be cleaned up when we delete the Echo.
Admission Webhook: Wedge Into the Write Path
Article 54 used a built-in admission controller (Pod Security). This article writes one of your own: an HTTPS service the API server calls before storing each object, returning allow or deny. We build a real validating webhook in Python — self-sign a cert, make the API server trust it via caBundle, and require every pod to have a team label. A pod missing the label is rejected immediately; a pod in a namespace out of scope is untouched.
CustomResourceDefinition: Add Your Own Kind
Part XII shifts from using Kubernetes to extending it. The first article is CustomResourceDefinition — declare a new kind of object, and the API server immediately serves it like a native resource: kubectl get works, it validates against a schema, it stores in etcd. We build a Widget CRD with type and value-range constraints, create a valid custom resource, watch two invalid ones get rejected, then update status through a separate subresource.
Secrets, the Detour and Hardening
Part XI closes out at Secrets and the holes still left. We read etcd directly to confirm Secrets are encrypted at-rest since Article 5, then build a real detour: a ServiceAccount with no permission to read a Secret still extracts its value by creating a pod that mounts that Secret and reading the log. The article ends with a table of hardening steps for a self-built cluster — which are done in the series, which are still missing.
Seccomp, AppArmor and Capabilities
Article 54 made pods declare runAsNonRoot, drop ALL capabilities, seccomp RuntimeDefault — but that's only Kubernetes-level policy. This article goes to the kernel layer to see what they actually do: read /proc/self/status from two pods, one default and one hardened, comparing CapEff, Seccomp, NoNewPrivs, AppArmor. Then prove by hand that dropping a capability blocks a specific operation — chown is denied even when the container still runs as root.
Pod Security Standards and Admission
RBAC decides who can create a pod, not what that pod asks for. A pod running privileged or borrowing hostNetwork is an escape hatch onto the node. Pod Security Admission blocks it at creation: one label on a namespace, the API server measures the pod against three levels — privileged/baseline/restricted — and rejects violators. This article turns restricted on for a namespace, watches a plain pod get kicked out, writes a compliant pod that runs, then tries warn mode.