Blog
Thoughts on engineering, design, and building great products.
uprobe, USDT, and Inspecting a Pod From the Host
So far we've attached to the kernel. eBPF can also reach into userspace: uprobe attaches to a function in an ordinary program or library, USDT attaches to a tracepoint the application has baked in. This post traces getaddrinfo in libc to see which program is resolving which domain, then uses the same technique to inspect a real pod on the cluster from the host — seeing the syscalls it makes without touching the pod. This is why eBPF became the observability foundation for Kubernetes.
bpftrace: Maps, Counting and Histograms
Printing line by line floods the screen when events are dense. bpftrace's real power is aggregating data right inside the kernel: counting by key, building distribution charts, then returning only a small summary. This post uses bpftrace's @ map to count syscalls by process, then builds a real vfs_read latency histogram with a kprobe/kretprobe pair — seeing the distribution as ASCII bars, including the slow tail that an average would hide.
bpftrace: Tracing in a Single Line
Part I wrote eBPF programs by hand in C, with clang and bpftool — many steps for a simple question like 'which process is opening which file'. bpftrace is the shortcut: one command line answers immediately, no C, no clang. But underneath it's still eBPF — this post proves it (bpftool sees the bpftrace program load then disappear), then walks through the probe/filter/action syntax, the 122 thousand probes you can attach to, and the built-in variables, via one-liners that run for real.
BTF and CO-RE: Compile Once, Run on Every Kernel
Kernel data structures like task_struct have different layouts across kernel versions — a field sits at a different offset per build. So how does a pre-compiled eBPF program read the right field on every kernel? BTF and CO-RE. This final Part I post generates vmlinux.h from the kernel's BTF, writes a program that reads ppid by walking task->real_parent->tgid, compiles once and runs — libbpf finds the right offset from the running kernel's BTF. The foundation for Part III.
Program Types and Hooks: Where You Attach, What You See
An eBPF program doesn't run in a vacuum — it attaches to a hook in the kernel, and the kind of hook decides three things: when the program runs, what context it receives, and which helpers it may call. This post lists the program types the kernel supports, then attaches a tracepoint to the openat syscall to watch it run for real on every file open — contrasted with the XDP packet receiver from the previous post, to show why two programs that are both eBPF each see a completely different world.
Maps: Memory and the Bridge to Userspace
An eBPF program runs per event then shuts off, keeping no variables between runs. Maps are how it remembers state and talks to userspace. This article writes a program that counts every process exec into a map, loads it into the kernel, runs a few commands, then reads the map from userspace with bpftool — watching the number actually go up. Plus inspecting a real Cilium map holding per-CPU metrics, and distinguishing a plain array from a per-CPU one.
The eBPF Virtual Machine: Registers, Instruction Set, and Bytecode
Last article we saw an eBPF program with 'xlated 512B' (verified bytecode) and 'jited 333B' (machine code). This article goes inside that bytecode: eBPF is a RISC-style virtual machine with 11 64-bit registers and a small instruction set, designed to translate quickly to native code and be verifiable for safety. We read a running Cilium program's bytecode directly, see how each instruction maps to registers and its class, then why this design lets the verifier prove safety.