Blog
Thoughts on engineering, design, and building great products.
Infrastructure as Code, What Terraform Is, and Getting to Know the CLI
The series opener: why managing infrastructure by hand eventually breaks, what Infrastructure as Code solves, and where Terraform fits in that picture. We dissect the core and provider architecture, install Terraform 1.15, and tour the main CLI commands.
Inside Hubble: From eBPF Events to Cluster-Wide Network Flows
Hubble lets us see every connection in a Kubernetes cluster by pod name, service, and policy verdict — without a sidecar in any pod. This article dissects the mechanism: Cilium's eBPF datapath (the 74 sched_cls programs from Article 12) calls bpf_perf_event_output to push events into a perf ring buffer named cilium_events; cilium-agent reads them out with numeric identities; then Hubble enriches them — turning identity 18203 into kube-system/coredns — via the identity-to-label store.
The Tetragon Way: From Observe to Enforce with bpf_send_signal
Tetragon is the Cilium ecosystem's runtime security tool: it observes with kprobe/tracepoint (the very hooks Part II used) and then enforces inside the kernel. Its enforcement uses two helpers — bpf_send_signal sends SIGKILL to kill a process, and bpf_override_return overrides a syscall's return value. This article rebuilds that: an exec tracepoint calls bpf_send_signal(SIGKILL) the moment a process runs — a forbidden binary gets exit 137, a normal binary still runs. No LSM, no reboot.
LSM BPF: Enforcing Security Right Inside the Kernel
So far our eBPF has only observed. LSM BPF enforces: it attaches to the kernel's security hooks (Linux Security Modules) that SELinux and AppArmor use, and a program returns 0 to allow or -EPERM to block. This article writes an LSM program that blocks opening a file, and hits a lesson: it loaded and attached but blocked nothing — because bpf wasn't an active LSM. After enabling bpf via a boot parameter and rebooting, it blocks for real — both cat and python get Operation not permitted.
Writing a tc Program Yourself: __sk_buff and the tcx Chain
Article 12 read Cilium's tc datapath from the outside. This article writes a tc program ourselves — counting egress packets by protocol — to understand __sk_buff from the inside. The core difference from XDP: tc sees the sk_buff with metadata already filled in (skb->protocol, skb->len), not the raw packet. We attach it with tcx on a real interface, get correct counts, then hit a lesson: attached after Cilium on the NIC it never runs, because of how the tcx chain terminates.
The Verifier: Why eBPF Doesn't Crash the Kernel
Article 1 said the eBPF virtual machine design lets the verifier prove safety. This article watches it for real: we compile an XDP program that reads the first byte of a packet but forgets the bounds check, load it — the verifier rejects it with a log naming the exact register and reason. Add one data_end check and it goes through. The verifier is a safety prover at load time, tracking each register's state across every branch — letting eBPF load foreign code into the kernel safely.
eBPF From Scratch: Running Programs Inside the Linux Kernel
Right now, on a worker of the Kubernetes cluster we built in the previous series, 140 eBPF programs are running inside the Linux kernel — routing every packet, controlling device access, collecting metrics. eBPF lets you load code into the kernel and run it safely at hooks, without changing kernel source and without loading a module. This opening article explains what eBPF is, why it changes how the kernel is extended, and how a program goes from code to native machine code.