Blog
Thoughts on engineering, design, and building great products.
Off-CPU and Scheduler Latency: Measuring the Time a Process Is NOT Running
On-CPU profiling (Article 17) only sees the CPU when busy. But most latency an app feels is time it is NOT running: waiting for disk, a lock, or its CPU turn. eBPF measures that off-CPU interval via scheduler tracepoints. On a real node we measure two things: run-queue latency — from wakeup to actually running, exposing the 16-32ms tail under CPU contention; and off-CPU time — how long a task stays off the CPU each time, with a tail reaching several seconds for blocked tasks.
The scheduler and the scheduling framework
Every pod we create has someone quietly picking a node for it — that's kube-scheduler, the thing we stood up in Article 8 but never looked at closely. This article opens Part VII by digging into exactly how it picks: filter out nodes that don't fit, score the remaining nodes, then bind. We test for real a pod stuck because no node has room, a pod that gets a node, and watch scoring pile pods onto the less-loaded node — not a naive round-robin.
controller-manager and scheduler: Control Loops and Leader Election
The two components that turn desired state into action: kube-controller-manager runs dozens of control loops, kube-scheduler picks the node for a pod. This article stands up both on three controllers, explains how the scheduler filters and scores nodes, then watches leader election work for real — three instances running, but only one doing the work.