Blog
Thoughts on engineering, design, and building great products.
Off-CPU and Scheduler Latency: Measuring the Time a Process Is NOT Running
On-CPU profiling (Article 17) only sees the CPU when busy. But most latency an app feels is time it is NOT running: waiting for disk, a lock, or its CPU turn. eBPF measures that off-CPU interval via scheduler tracepoints. On a real node we measure two things: run-queue latency — from wakeup to actually running, exposing the 16-32ms tail under CPU contention; and off-CPU time — how long a task stays off the CPU each time, with a tail reaching several seconds for blocked tasks.
uprobe, USDT, and Inspecting a Pod From the Host
So far we've attached to the kernel. eBPF can also reach into userspace: uprobe attaches to a function in an ordinary program or library, USDT attaches to a tracepoint the application has baked in. This post traces getaddrinfo in libc to see which program is resolving which domain, then uses the same technique to inspect a real pod on the cluster from the host — seeing the syscalls it makes without touching the pod. This is why eBPF became the observability foundation for Kubernetes.
bpftrace: Maps, Counting and Histograms
Printing line by line floods the screen when events are dense. bpftrace's real power is aggregating data right inside the kernel: counting by key, building distribution charts, then returning only a small summary. This post uses bpftrace's @ map to count syscalls by process, then builds a real vfs_read latency histogram with a kprobe/kretprobe pair — seeing the distribution as ASCII bars, including the slow tail that an average would hide.
bpftrace: Tracing in a Single Line
Part I wrote eBPF programs by hand in C, with clang and bpftool — many steps for a simple question like 'which process is opening which file'. bpftrace is the shortcut: one command line answers immediately, no C, no clang. But underneath it's still eBPF — this post proves it (bpftool sees the bpftrace program load then disappear), then walks through the probe/filter/action syntax, the 122 thousand probes you can attach to, and the built-in variables, via one-liners that run for real.