Kubernetes From Scratch
Build a complete Kubernetes cluster by hand — no kubeadm, no scripts — from the first certificate to a real HA cluster, then use that cluster as a lab to deep-dive every Kubernetes concept. Part one: PKI/TLS, etcd, the control plane, workers, pod networking, CoreDNS. Part two: Pods, workload controllers, scheduling, storage, advanced networking (Cilium eBPF), security, extending the API, operations. Each component is both explained from the inside and stood up/configured by hand. Tested for real on AWS EC2 with Kubernetes v1.36; manifests/scripts at github.com/nghiadaulau/kubernetes-from-scratch. Grounded in the official docs at kubernetes.io.
Ingress: Getting HTTP In From Outside (With Cilium)
NetworkPolicy handles pod-to-pod traffic inside the cluster. This article opens the cluster edge to HTTP from outside, routing by host and path to the right Service — using Cilium's built-in Ingress controller, no extra software to install. Just as important as the mechanics is a real decision: Ingress NGINX was retired in March 2026 and the Ingress API is frozen, so we pick a maintained controller and watch how Cilium translates an Ingress into Envoy config running on eBPF.
Gateway API: The Successor to Ingress
Ingress is frozen at the basics. Gateway API is Kubernetes' new API for inbound traffic, splitting infrastructure and application roles into separate objects, and doing what Ingress can't: weighted traffic splitting, header matching, multi-protocol routing. This article enables Gateway API on Cilium, stands up a Gateway with an HTTPRoute routing by host/path, then splits traffic 80/20 between two versions — tested for real on the EC2 cluster.
LB IPAM and Traffic Policy
Articles 48 and 49 both stopped where the LoadBalancer Service and Gateway hung with external-IP <pending> — nobody hands out addresses on a self-built cluster. This article fills the gap with Cilium's LB IPAM: define an IP range, let Cilium assign it, and the previous Gateway flips to Programmed=True. Then externalTrafficPolicy — Cluster or Local decides whether the client's source IP survives. With a clear line between assigning an IP and advertising it.
Authentication and the Path Into the API Server
Every kubectl command is an HTTPS request to the API server, and before it touches data it must pass three stages: authentication, authorization, admission. This article opens Part XI with the first stage — the API server figuring out who you are. We examine the three ways our self-built cluster authenticates a request: client certificate (the one admin.kubeconfig uses), ServiceAccount token, and anonymous request — using kubectl auth whoami and real commands on the cluster.
RBAC: Turning Identity Into Permission
Article 51 stopped where the API server knows who you are. RBAC answers the rest: what may you do. This article stands up a ServiceAccount that can only read pods in one namespace, verifies it with both kubectl auth can-i and a real token — it lists pods but reading secrets or creating pods returns 403. Then we see how a RoleBinding points at a built-in ClusterRole to scope permission to one namespace, and why the view ClusterRole deliberately can't read secrets.
ServiceAccount and Bound Tokens
Articles 51–52 used ServiceAccount without dissecting it. This article gets into the mechanism: every namespace has a default SA, the kubelet auto-injects a short-lived token via a projected volume, and that token is bound to the exact pod and node. To prove it's truly bound, we grab the token in a running pod, call the API successfully, then delete the pod — the old token immediately becomes 401. Plus how to turn off auto-mount and read the JWT claims.
Pod Security Standards and Admission
RBAC decides who can create a pod, not what that pod asks for. A pod running privileged or borrowing hostNetwork is an escape hatch onto the node. Pod Security Admission blocks it at creation: one label on a namespace, the API server measures the pod against three levels — privileged/baseline/restricted — and rejects violators. This article turns restricted on for a namespace, watches a plain pod get kicked out, writes a compliant pod that runs, then tries warn mode.
Seccomp, AppArmor and Capabilities
Article 54 made pods declare runAsNonRoot, drop ALL capabilities, seccomp RuntimeDefault — but that's only Kubernetes-level policy. This article goes to the kernel layer to see what they actually do: read /proc/self/status from two pods, one default and one hardened, comparing CapEff, Seccomp, NoNewPrivs, AppArmor. Then prove by hand that dropping a capability blocks a specific operation — chown is denied even when the container still runs as root.
Secrets, the Detour and Hardening
Part XI closes out at Secrets and the holes still left. We read etcd directly to confirm Secrets are encrypted at-rest since Article 5, then build a real detour: a ServiceAccount with no permission to read a Secret still extracts its value by creating a pod that mounts that Secret and reading the log. The article ends with a table of hardening steps for a self-built cluster — which are done in the series, which are still missing.
CustomResourceDefinition: Add Your Own Kind
Part XII shifts from using Kubernetes to extending it. The first article is CustomResourceDefinition — declare a new kind of object, and the API server immediately serves it like a native resource: kubectl get works, it validates against a schema, it stores in etcd. We build a Widget CRD with type and value-range constraints, create a valid custom resource, watch two invalid ones get rejected, then update status through a separate subresource.
Admission Webhook: Wedge Into the Write Path
Article 54 used a built-in admission controller (Pod Security). This article writes one of your own: an HTTPS service the API server calls before storing each object, returning allow or deny. We build a real validating webhook in Python — self-sign a cert, make the API server trust it via caBundle, and require every pod to have a team label. A pod missing the label is rejected immediately; a pod in a namespace out of scope is untouched.
Operator: CRD Plus a Reconcile Loop
A CRD gives us a new data type, but creating a custom resource makes nothing happen. An operator joins a CRD with a controller running a loop: it watches the custom resource and acts to bring reality in line with desire. This article builds a real operator from scratch — an Echo CRD and an in-pod controller — then watches it create a Deployment when we create an Echo, scale when we edit replicas, and let the Deployment be cleaned up when we delete the Echo.