Blog
Thoughts on engineering, design, and building great products.
Case Study: A Packet Through Cilium's eBPF Datapath
Nineteen articles dissected each piece: verifier, maps, XDP, tc, tail call, perf ring, identity. This article assembles them into one seamless story — following a single packet as a pod calls the cluster's DNS Service, from leaving the source pod to reaching the CoreDNS pod, through every eBPF program and every BPF map it touches. No new concepts; just seeing the whole machine run as one unified thing, with real data from the same cluster used throughout the series.
Off-CPU and Scheduler Latency: Measuring the Time a Process Is NOT Running
On-CPU profiling (Article 17) only sees the CPU when busy. But most latency an app feels is time it is NOT running: waiting for disk, a lock, or its CPU turn. eBPF measures that off-CPU interval via scheduler tracepoints. On a real node we measure two things: run-queue latency — from wakeup to actually running, exposing the 16-32ms tail under CPU contention; and off-CPU time — how long a task stays off the CPU each time, with a tail reaching several seconds for blocked tasks.
CPU Profiling with perf_event: Sampling Stacks, the Foundation of Flame Graphs
To know what the CPU is busy doing, we sample: a few dozen times per second, freeze each CPU and record the stack that's running. eBPF does this through the perf_event program type — attached to a kernel sampling counter, each time it fires it captures the stack and aggregates in the kernel. This article profiles a real node at 99Hz, sees dd eating CPU reading /dev/zero while idle cores sit in the idle loop, aggregates by process to get dd at 479 samples — the data a flame graph draws.
seccomp-bpf: Classic BPF Filtering Syscalls in Every Container
Before eBPF there was cBPF — classic BPF, the thing tcpdump uses. And it's still running: seccomp-bpf filters syscalls with cBPF, the foundational sandbox layer of containers. This article distinguishes cBPF from eBPF, inspects real seccomp on the cluster (pause containers and CSI sidecars restricted, privileged pods not), then writes a cBPF filter that blocks mkdir with EPERM — eight instructions on struct seccomp_data, installed with prctl, blocking for real while printf still runs.
tc/sched_cls and Dissecting a Live Cilium Datapath
After XDP comes tc — the hook where the packet already has an sk_buff, where both ingress and egress are visible, and where Cilium puts almost its entire datapath. This article dissects the 74 sched_cls programs actually running on a cluster node: where they attach (NIC, each pod), how they call each other via tail calls, and which BPF maps they look up to load-balance a Service or apply a NetworkPolicy. kube-proxy-less load balancing is one map lookup.
XDP: Processing Packets at the Earliest Point, Writing a Firewall
XDP attaches an eBPF program to the network driver, running on every incoming packet before the kernel even allocates an sk_buff — the earliest point you can touch a packet. It returns a verdict: PASS, DROP, TX, REDIRECT. This post builds a small XDP firewall that drops ICMP on a real interface, attaches it to a node's NIC with bpftool, then watches ping fall from 0% to 100% loss while SSH stays alive — and sees how it sits ahead of Cilium's tc datapath on the same interface.
cilium/ebpf: Loading eBPF From Go
Article 9 built execsnoop in C with libbpf. This post rewrites the exact same tool but loads it from Go with the cilium/ebpf library — how the Kubernetes ecosystem (Cilium, Tetragon, Falco) builds eBPF applications. The kernel side is unchanged; bpf2go compiles it and embeds the object straight into the Go code, then a Go program attaches the tracepoint and reads the ring buffer. The result is a single static binary with no dependency on libbpf.so — and we hit the real traps building it.