Blog
Thoughts on engineering, design, and building great products.
LSM BPF: Enforcing Security Right Inside the Kernel
So far our eBPF has only observed. LSM BPF enforces: it attaches to the kernel's security hooks (Linux Security Modules) that SELinux and AppArmor use, and a program returns 0 to allow or -EPERM to block. This article writes an LSM program that blocks opening a file, and hits a lesson: it loaded and attached but blocked nothing — because bpf wasn't an active LSM. After enabling bpf via a boot parameter and rebooting, it blocks for real — both cat and python get Operation not permitted.
Writing a tc Program Yourself: __sk_buff and the tcx Chain
Article 12 read Cilium's tc datapath from the outside. This article writes a tc program ourselves — counting egress packets by protocol — to understand __sk_buff from the inside. The core difference from XDP: tc sees the sk_buff with metadata already filled in (skb->protocol, skb->len), not the raw packet. We attach it with tcx on a real interface, get correct counts, then hit a lesson: attached after Cilium on the NIC it never runs, because of how the tcx chain terminates.
The Verifier: Why eBPF Doesn't Crash the Kernel
Article 1 said the eBPF virtual machine design lets the verifier prove safety. This article watches it for real: we compile an XDP program that reads the first byte of a packet but forgets the bounds check, load it — the verifier rejects it with a log naming the exact register and reason. Add one data_end check and it goes through. The verifier is a safety prover at load time, tracking each register's state across every branch — letting eBPF load foreign code into the kernel safely.
eBPF From Scratch: Running Programs Inside the Linux Kernel
Right now, on a worker of the Kubernetes cluster we built in the previous series, 140 eBPF programs are running inside the Linux kernel — routing every packet, controlling device access, collecting metrics. eBPF lets you load code into the kernel and run it safely at hooks, without changing kernel source and without loading a module. This opening article explains what eBPF is, why it changes how the kernel is extended, and how a program goes from code to native machine code.
Off-CPU and Scheduler Latency: Measuring the Time a Process Is NOT Running
On-CPU profiling (Article 17) only sees the CPU when busy. But most latency an app feels is time it is NOT running: waiting for disk, a lock, or its CPU turn. eBPF measures that off-CPU interval via scheduler tracepoints. On a real node we measure two things: run-queue latency — from wakeup to actually running, exposing the 16-32ms tail under CPU contention; and off-CPU time — how long a task stays off the CPU each time, with a tail reaching several seconds for blocked tasks.
CPU Profiling with perf_event: Sampling Stacks, the Foundation of Flame Graphs
To know what the CPU is busy doing, we sample: a few dozen times per second, freeze each CPU and record the stack that's running. eBPF does this through the perf_event program type — attached to a kernel sampling counter, each time it fires it captures the stack and aggregates in the kernel. This article profiles a real node at 99Hz, sees dd eating CPU reading /dev/zero while idle cores sit in the idle loop, aggregates by process to get dd at 479 samples — the data a flame graph draws.
seccomp-bpf: Classic BPF Filtering Syscalls in Every Container
Before eBPF there was cBPF — classic BPF, the thing tcpdump uses. And it's still running: seccomp-bpf filters syscalls with cBPF, the foundational sandbox layer of containers. This article distinguishes cBPF from eBPF, inspects real seccomp on the cluster (pause containers and CSI sidecars restricted, privileged pods not), then writes a cBPF filter that blocks mkdir with EPERM — eight instructions on struct seccomp_data, installed with prctl, blocking for real while printf still runs.