Blog
Thoughts on engineering, design, and building great products.
Inside Hubble: From eBPF Events to Cluster-Wide Network Flows
Hubble lets us see every connection in a Kubernetes cluster by pod name, service, and policy verdict — without a sidecar in any pod. This article dissects the mechanism: Cilium's eBPF datapath (the 74 sched_cls programs from Article 12) calls bpf_perf_event_output to push events into a perf ring buffer named cilium_events; cilium-agent reads them out with numeric identities; then Hubble enriches them — turning identity 18203 into kube-system/coredns — via the identity-to-label store.
The Tetragon Way: From Observe to Enforce with bpf_send_signal
Tetragon is the Cilium ecosystem's runtime security tool: it observes with kprobe/tracepoint (the very hooks Part II used) and then enforces inside the kernel. Its enforcement uses two helpers — bpf_send_signal sends SIGKILL to kill a process, and bpf_override_return overrides a syscall's return value. This article rebuilds that: an exec tracepoint calls bpf_send_signal(SIGKILL) the moment a process runs — a forbidden binary gets exit 137, a normal binary still runs. No LSM, no reboot.
eBPF From Scratch: Running Programs Inside the Linux Kernel
Right now, on a worker of the Kubernetes cluster we built in the previous series, 140 eBPF programs are running inside the Linux kernel — routing every packet, controlling device access, collecting metrics. eBPF lets you load code into the kernel and run it safely at hooks, without changing kernel source and without loading a module. This opening article explains what eBPF is, why it changes how the kernel is extended, and how a program goes from code to native machine code.
Case Study: A Packet Through Cilium's eBPF Datapath
Nineteen articles dissected each piece: verifier, maps, XDP, tc, tail call, perf ring, identity. This article assembles them into one seamless story — following a single packet as a pod calls the cluster's DNS Service, from leaving the source pod to reaching the CoreDNS pod, through every eBPF program and every BPF map it touches. No new concepts; just seeing the whole machine run as one unified thing, with real data from the same cluster used throughout the series.
tc/sched_cls and Dissecting a Live Cilium Datapath
After XDP comes tc — the hook where the packet already has an sk_buff, where both ingress and egress are visible, and where Cilium puts almost its entire datapath. This article dissects the 74 sched_cls programs actually running on a cluster node: where they attach (NIC, each pod), how they call each other via tail calls, and which BPF maps they look up to load-balance a Service or apply a NetworkPolicy. kube-proxy-less load balancing is one map lookup.
XDP: Processing Packets at the Earliest Point, Writing a Firewall
XDP attaches an eBPF program to the network driver, running on every incoming packet before the kernel even allocates an sk_buff — the earliest point you can touch a packet. It returns a verdict: PASS, DROP, TX, REDIRECT. This post builds a small XDP firewall that drops ICMP on a real interface, attaches it to a node's NIC with bpftool, then watches ping fall from 0% to 100% loss while SSH stays alive — and sees how it sits ahead of Cilium's tc datapath on the same interface.
LB IPAM and Traffic Policy
Articles 48 and 49 both stopped where the LoadBalancer Service and Gateway hung with external-IP <pending> — nobody hands out addresses on a self-built cluster. This article fills the gap with Cilium's LB IPAM: define an IP range, let Cilium assign it, and the previous Gateway flips to Programmed=True. Then externalTrafficPolicy — Cluster or Local decides whether the client's source IP survives. With a clear line between assigning an IP and advertising it.