Blog
Thoughts on engineering, design, and building great products.
Inside Hubble: From eBPF Events to Cluster-Wide Network Flows
Hubble lets us see every connection in a Kubernetes cluster by pod name, service, and policy verdict — without a sidecar in any pod. This article dissects the mechanism: Cilium's eBPF datapath (the 74 sched_cls programs from Article 12) calls bpf_perf_event_output to push events into a perf ring buffer named cilium_events; cilium-agent reads them out with numeric identities; then Hubble enriches them — turning identity 18203 into kube-system/coredns — via the identity-to-label store.
The Tetragon Way: From Observe to Enforce with bpf_send_signal
Tetragon is the Cilium ecosystem's runtime security tool: it observes with kprobe/tracepoint (the very hooks Part II used) and then enforces inside the kernel. Its enforcement uses two helpers — bpf_send_signal sends SIGKILL to kill a process, and bpf_override_return overrides a syscall's return value. This article rebuilds that: an exec tracepoint calls bpf_send_signal(SIGKILL) the moment a process runs — a forbidden binary gets exit 137, a normal binary still runs. No LSM, no reboot.
Case Study: A Packet Through Cilium's eBPF Datapath
Nineteen articles dissected each piece: verifier, maps, XDP, tc, tail call, perf ring, identity. This article assembles them into one seamless story — following a single packet as a pod calls the cluster's DNS Service, from leaving the source pod to reaching the CoreDNS pod, through every eBPF program and every BPF map it touches. No new concepts; just seeing the whole machine run as one unified thing, with real data from the same cluster used throughout the series.
tc/sched_cls and Dissecting a Live Cilium Datapath
After XDP comes tc — the hook where the packet already has an sk_buff, where both ingress and egress are visible, and where Cilium puts almost its entire datapath. This article dissects the 74 sched_cls programs actually running on a cluster node: where they attach (NIC, each pod), how they call each other via tail calls, and which BPF maps they look up to load-balance a Service or apply a NetworkPolicy. kube-proxy-less load balancing is one map lookup.
cilium/ebpf: Loading eBPF From Go
Article 9 built execsnoop in C with libbpf. This post rewrites the exact same tool but loads it from Go with the cilium/ebpf library — how the Kubernetes ecosystem (Cilium, Tetragon, Falco) builds eBPF applications. The kernel side is unchanged; bpf2go compiles it and embeds the object straight into the Go code, then a Go program attaches the tracepoint and reads the ring buffer. The result is a single static binary with no dependency on libbpf.so — and we hit the real traps building it.
uprobe, USDT, and Inspecting a Pod From the Host
So far we've attached to the kernel. eBPF can also reach into userspace: uprobe attaches to a function in an ordinary program or library, USDT attaches to a tracepoint the application has baked in. This post traces getaddrinfo in libc to see which program is resolving which domain, then uses the same technique to inspect a real pod on the cluster from the host — seeing the syscalls it makes without touching the pod. This is why eBPF became the observability foundation for Kubernetes.
Full Teardown and Wrap-up
The final article. The EC2 cluster has done its job, and leaving it running burns money every hour. This article tears down all the infrastructure in order — terminate the six instances (taking their disks with them), release the Elastic IP, remove the IAM role — with a real cost table for three choices: keep running, stop to save it, or delete entirely. Then a look back: from a single self-signed certificate to a fully operational HA Kubernetes cluster, built by hand.