Blog
Thoughts on engineering, design, and building great products.
Inside Hubble: From eBPF Events to Cluster-Wide Network Flows
Hubble lets us see every connection in a Kubernetes cluster by pod name, service, and policy verdict — without a sidecar in any pod. This article dissects the mechanism: Cilium's eBPF datapath (the 74 sched_cls programs from Article 12) calls bpf_perf_event_output to push events into a perf ring buffer named cilium_events; cilium-agent reads them out with numeric identities; then Hubble enriches them — turning identity 18203 into kube-system/coredns — via the identity-to-label store.
The Tetragon Way: From Observe to Enforce with bpf_send_signal
Tetragon is the Cilium ecosystem's runtime security tool: it observes with kprobe/tracepoint (the very hooks Part II used) and then enforces inside the kernel. Its enforcement uses two helpers — bpf_send_signal sends SIGKILL to kill a process, and bpf_override_return overrides a syscall's return value. This article rebuilds that: an exec tracepoint calls bpf_send_signal(SIGKILL) the moment a process runs — a forbidden binary gets exit 137, a normal binary still runs. No LSM, no reboot.
LSM BPF: Enforcing Security Right Inside the Kernel
So far our eBPF has only observed. LSM BPF enforces: it attaches to the kernel's security hooks (Linux Security Modules) that SELinux and AppArmor use, and a program returns 0 to allow or -EPERM to block. This article writes an LSM program that blocks opening a file, and hits a lesson: it loaded and attached but blocked nothing — because bpf wasn't an active LSM. After enabling bpf via a boot parameter and rebooting, it blocks for real — both cat and python get Operation not permitted.
Writing a tc Program Yourself: __sk_buff and the tcx Chain
Article 12 read Cilium's tc datapath from the outside. This article writes a tc program ourselves — counting egress packets by protocol — to understand __sk_buff from the inside. The core difference from XDP: tc sees the sk_buff with metadata already filled in (skb->protocol, skb->len), not the raw packet. We attach it with tcx on a real interface, get correct counts, then hit a lesson: attached after Cilium on the NIC it never runs, because of how the tcx chain terminates.
The Verifier: Why eBPF Doesn't Crash the Kernel
Article 1 said the eBPF virtual machine design lets the verifier prove safety. This article watches it for real: we compile an XDP program that reads the first byte of a packet but forgets the bounds check, load it — the verifier rejects it with a log naming the exact register and reason. Add one data_end check and it goes through. The verifier is a safety prover at load time, tracking each register's state across every branch — letting eBPF load foreign code into the kernel safely.
eBPF From Scratch: Running Programs Inside the Linux Kernel
Right now, on a worker of the Kubernetes cluster we built in the previous series, 140 eBPF programs are running inside the Linux kernel — routing every packet, controlling device access, collecting metrics. eBPF lets you load code into the kernel and run it safely at hooks, without changing kernel source and without loading a module. This opening article explains what eBPF is, why it changes how the kernel is extended, and how a program goes from code to native machine code.
Capstone: Writing connmon — A Node-Wide TCP Connection Monitor
The final article: assembling everything learned into a real tool. connmon attaches a kprobe to tcp_connect in the kernel, pushes every new TCP connection through a ring buffer, and a Go loader prints them in real time — pid, process, destination IP:port. Just over a hundred lines, a single static binary, run it on the cluster and immediately see coredns, kubelet, curl connecting out. Includes a real kprobe build trap. Then a look back over the whole eBPF journey from scratch.