Blog
Thoughts on engineering, design, and building great products.
libbpf and CO-RE: Writing an eBPF Tool Yourself
bpftrace is good for quick questions. When you need a real tool — packaged, distributed, long-running — you write the eBPF program in C with libbpf and CO-RE. This post builds execsnoop from scratch: a kernel program pushing exec events through a ring buffer, bpftool generating the skeleton, and a C loader using libbpf to load and read events. Build the full chain clang → skeleton → libbpf, then watch every exec on the cluster appear with pid, ppid, filename — plus a real buffer trap.
uprobe, USDT, and Inspecting a Pod From the Host
So far we've attached to the kernel. eBPF can also reach into userspace: uprobe attaches to a function in an ordinary program or library, USDT attaches to a tracepoint the application has baked in. This post traces getaddrinfo in libc to see which program is resolving which domain, then uses the same technique to inspect a real pod on the cluster from the host — seeing the syscalls it makes without touching the pod. This is why eBPF became the observability foundation for Kubernetes.
bpftrace: Maps, Counting and Histograms
Printing line by line floods the screen when events are dense. bpftrace's real power is aggregating data right inside the kernel: counting by key, building distribution charts, then returning only a small summary. This post uses bpftrace's @ map to count syscalls by process, then builds a real vfs_read latency histogram with a kprobe/kretprobe pair — seeing the distribution as ASCII bars, including the slow tail that an average would hide.
bpftrace: Tracing in a Single Line
Part I wrote eBPF programs by hand in C, with clang and bpftool — many steps for a simple question like 'which process is opening which file'. bpftrace is the shortcut: one command line answers immediately, no C, no clang. But underneath it's still eBPF — this post proves it (bpftool sees the bpftrace program load then disappear), then walks through the probe/filter/action syntax, the 122 thousand probes you can attach to, and the built-in variables, via one-liners that run for real.
BTF and CO-RE: Compile Once, Run on Every Kernel
Kernel data structures like task_struct have different layouts across kernel versions — a field sits at a different offset per build. So how does a pre-compiled eBPF program read the right field on every kernel? BTF and CO-RE. This final Part I post generates vmlinux.h from the kernel's BTF, writes a program that reads ppid by walking task->real_parent->tgid, compiles once and runs — libbpf finds the right offset from the running kernel's BTF. The foundation for Part III.
Program Types and Hooks: Where You Attach, What You See
An eBPF program doesn't run in a vacuum — it attaches to a hook in the kernel, and the kind of hook decides three things: when the program runs, what context it receives, and which helpers it may call. This post lists the program types the kernel supports, then attaches a tracepoint to the openat syscall to watch it run for real on every file open — contrasted with the XDP packet receiver from the previous post, to show why two programs that are both eBPF each see a completely different world.
Maps: Memory and the Bridge to Userspace
An eBPF program runs per event then shuts off, keeping no variables between runs. Maps are how it remembers state and talks to userspace. This article writes a program that counts every process exec into a map, loads it into the kernel, runs a few commands, then reads the map from userspace with bpftool — watching the number actually go up. Plus inspecting a real Cilium map holding per-CPU metrics, and distinguishing a plain array from a per-CPU one.