Blog
Thoughts on engineering, design, and building great products.
Capstone: Writing connmon — A Node-Wide TCP Connection Monitor
The final article: assembling everything learned into a real tool. connmon attaches a kprobe to tcp_connect in the kernel, pushes every new TCP connection through a ring buffer, and a Go loader prints them in real time — pid, process, destination IP:port. Just over a hundred lines, a single static binary, run it on the cluster and immediately see coredns, kubelet, curl connecting out. Includes a real kprobe build trap. Then a look back over the whole eBPF journey from scratch.
Case Study: A Packet Through Cilium's eBPF Datapath
Nineteen articles dissected each piece: verifier, maps, XDP, tc, tail call, perf ring, identity. This article assembles them into one seamless story — following a single packet as a pod calls the cluster's DNS Service, from leaving the source pod to reaching the CoreDNS pod, through every eBPF program and every BPF map it touches. No new concepts; just seeing the whole machine run as one unified thing, with real data from the same cluster used throughout the series.
Off-CPU and Scheduler Latency: Measuring the Time a Process Is NOT Running
On-CPU profiling (Article 17) only sees the CPU when busy. But most latency an app feels is time it is NOT running: waiting for disk, a lock, or its CPU turn. eBPF measures that off-CPU interval via scheduler tracepoints. On a real node we measure two things: run-queue latency — from wakeup to actually running, exposing the 16-32ms tail under CPU contention; and off-CPU time — how long a task stays off the CPU each time, with a tail reaching several seconds for blocked tasks.
CPU Profiling with perf_event: Sampling Stacks, the Foundation of Flame Graphs
To know what the CPU is busy doing, we sample: a few dozen times per second, freeze each CPU and record the stack that's running. eBPF does this through the perf_event program type — attached to a kernel sampling counter, each time it fires it captures the stack and aggregates in the kernel. This article profiles a real node at 99Hz, sees dd eating CPU reading /dev/zero while idle cores sit in the idle loop, aggregates by process to get dd at 479 samples — the data a flame graph draws.
seccomp-bpf: Classic BPF Filtering Syscalls in Every Container
Before eBPF there was cBPF — classic BPF, the thing tcpdump uses. And it's still running: seccomp-bpf filters syscalls with cBPF, the foundational sandbox layer of containers. This article distinguishes cBPF from eBPF, inspects real seccomp on the cluster (pause containers and CSI sidecars restricted, privileged pods not), then writes a cBPF filter that blocks mkdir with EPERM — eight instructions on struct seccomp_data, installed with prctl, blocking for real while printf still runs.
tc/sched_cls and Dissecting a Live Cilium Datapath
After XDP comes tc — the hook where the packet already has an sk_buff, where both ingress and egress are visible, and where Cilium puts almost its entire datapath. This article dissects the 74 sched_cls programs actually running on a cluster node: where they attach (NIC, each pod), how they call each other via tail calls, and which BPF maps they look up to load-balance a Service or apply a NetworkPolicy. kube-proxy-less load balancing is one map lookup.
XDP: Processing Packets at the Earliest Point, Writing a Firewall
XDP attaches an eBPF program to the network driver, running on every incoming packet before the kernel even allocates an sk_buff — the earliest point you can touch a packet. It returns a verdict: PASS, DROP, TX, REDIRECT. This post builds a small XDP firewall that drops ICMP on a real interface, attaches it to a node's NIC with bpftool, then watches ping fall from 0% to 100% loss while SSH stays alive — and sees how it sits ahead of Cilium's tc datapath on the same interface.