eBPF From Scratch
Learn eBPF from the ground up to writing real programs — the eBPF virtual machine, the verifier, maps, the hooks (XDP/tc/kprobe/tracepoint/LSM); tracing with bpftrace; writing programs in C with libbpf + CO-RE then loading them from Go (cilium/ebpf); networking, observability and security. A real Kubernetes cluster (kernel 6.17, Cilium 1.19 eBPF kube-proxy-less with hundreds of BPF programs running) is the lab throughout. Everything is tested on real hardware and grounded in official docs (ebpf.io, kernel.org, libbpf, cilium). Source at github.com/nghiadaulau/ebpf-from-scratch.
eBPF From Scratch: Running Programs Inside the Linux Kernel
Right now, on a worker of the Kubernetes cluster we built in the previous series, 140 eBPF programs are running inside the Linux kernel — routing every packet, controlling device access, collecting metrics. eBPF lets you load code into the kernel and run it safely at hooks, without changing kernel source and without loading a module. This opening article explains what eBPF is, why it changes how the kernel is extended, and how a program goes from code to native machine code.
The eBPF Virtual Machine: Registers, Instruction Set, and Bytecode
Last article we saw an eBPF program with 'xlated 512B' (verified bytecode) and 'jited 333B' (machine code). This article goes inside that bytecode: eBPF is a RISC-style virtual machine with 11 64-bit registers and a small instruction set, designed to translate quickly to native code and be verifiable for safety. We read a running Cilium program's bytecode directly, see how each instruction maps to registers and its class, then why this design lets the verifier prove safety.
The Verifier: Why eBPF Doesn't Crash the Kernel
Article 1 said the eBPF virtual machine design lets the verifier prove safety. This article watches it for real: we compile an XDP program that reads the first byte of a packet but forgets the bounds check, load it — the verifier rejects it with a log naming the exact register and reason. Add one data_end check and it goes through. The verifier is a safety prover at load time, tracking each register's state across every branch — letting eBPF load foreign code into the kernel safely.
Maps: Memory and the Bridge to Userspace
An eBPF program runs per event then shuts off, keeping no variables between runs. Maps are how it remembers state and talks to userspace. This article writes a program that counts every process exec into a map, loads it into the kernel, runs a few commands, then reads the map from userspace with bpftool — watching the number actually go up. Plus inspecting a real Cilium map holding per-CPU metrics, and distinguishing a plain array from a per-CPU one.
Program Types and Hooks: Where You Attach, What You See
An eBPF program doesn't run in a vacuum — it attaches to a hook in the kernel, and the kind of hook decides three things: when the program runs, what context it receives, and which helpers it may call. This post lists the program types the kernel supports, then attaches a tracepoint to the openat syscall to watch it run for real on every file open — contrasted with the XDP packet receiver from the previous post, to show why two programs that are both eBPF each see a completely different world.
BTF and CO-RE: Compile Once, Run on Every Kernel
Kernel data structures like task_struct have different layouts across kernel versions — a field sits at a different offset per build. So how does a pre-compiled eBPF program read the right field on every kernel? BTF and CO-RE. This final Part I post generates vmlinux.h from the kernel's BTF, writes a program that reads ppid by walking task->real_parent->tgid, compiles once and runs — libbpf finds the right offset from the running kernel's BTF. The foundation for Part III.
bpftrace: Tracing in a Single Line
Part I wrote eBPF programs by hand in C, with clang and bpftool — many steps for a simple question like 'which process is opening which file'. bpftrace is the shortcut: one command line answers immediately, no C, no clang. But underneath it's still eBPF — this post proves it (bpftool sees the bpftrace program load then disappear), then walks through the probe/filter/action syntax, the 122 thousand probes you can attach to, and the built-in variables, via one-liners that run for real.
bpftrace: Maps, Counting and Histograms
Printing line by line floods the screen when events are dense. bpftrace's real power is aggregating data right inside the kernel: counting by key, building distribution charts, then returning only a small summary. This post uses bpftrace's @ map to count syscalls by process, then builds a real vfs_read latency histogram with a kprobe/kretprobe pair — seeing the distribution as ASCII bars, including the slow tail that an average would hide.
uprobe, USDT, and Inspecting a Pod From the Host
So far we've attached to the kernel. eBPF can also reach into userspace: uprobe attaches to a function in an ordinary program or library, USDT attaches to a tracepoint the application has baked in. This post traces getaddrinfo in libc to see which program is resolving which domain, then uses the same technique to inspect a real pod on the cluster from the host — seeing the syscalls it makes without touching the pod. This is why eBPF became the observability foundation for Kubernetes.
libbpf and CO-RE: Writing an eBPF Tool Yourself
bpftrace is good for quick questions. When you need a real tool — packaged, distributed, long-running — you write the eBPF program in C with libbpf and CO-RE. This post builds execsnoop from scratch: a kernel program pushing exec events through a ring buffer, bpftool generating the skeleton, and a C loader using libbpf to load and read events. Build the full chain clang → skeleton → libbpf, then watch every exec on the cluster appear with pid, ppid, filename — plus a real buffer trap.
cilium/ebpf: Loading eBPF From Go
Article 9 built execsnoop in C with libbpf. This post rewrites the exact same tool but loads it from Go with the cilium/ebpf library — how the Kubernetes ecosystem (Cilium, Tetragon, Falco) builds eBPF applications. The kernel side is unchanged; bpf2go compiles it and embeds the object straight into the Go code, then a Go program attaches the tracepoint and reads the ring buffer. The result is a single static binary with no dependency on libbpf.so — and we hit the real traps building it.
XDP: Processing Packets at the Earliest Point, Writing a Firewall
XDP attaches an eBPF program to the network driver, running on every incoming packet before the kernel even allocates an sk_buff — the earliest point you can touch a packet. It returns a verdict: PASS, DROP, TX, REDIRECT. This post builds a small XDP firewall that drops ICMP on a real interface, attaches it to a node's NIC with bpftool, then watches ping fall from 0% to 100% loss while SSH stays alive — and sees how it sits ahead of Cilium's tc datapath on the same interface.