uprobe, USDT, and Inspecting a Pod From the Host

Articles 6–7 attached to the kernel — tracepoint, kprobe. But plenty of things worth observing live in userspace: a library function, an application's logic. eBPF reaches there through uprobe and USDT. And because a container is just a process on the host, the same technique lets us inspect a pod from the host without installing anything in the pod — closing Part II with exactly the thing that makes eBPF the observability foundation of Kubernetes.

uprobe: attaching to a userspace function

uprobe (user probe) attaches to a function in a userspace program or library, just as kprobe attaches to a kernel function. A common example: trace getaddrinfo in libc — the function every libc-based program calls to resolve domain names:

LIBC=/usr/lib/x86_64-linux-gnu/libc.so.6
sudo bpftrace -e "uprobe:$LIBC:getaddrinfo { printf(\"%s resolving %s\n\", comm, str(arg0)); }"

python3 resolving localhost
python3 resolving github.com

uprobe:<binary-path>:<function-name> is the probe; arg0 is the function's first argument — for getaddrinfo that's a pointer to the host name, which str() reads out. Any process on the machine that calls getaddrinfo (through libc) gets caught — here it's python3 resolving localhost and github.com. This is userspace observability without modifying or recompiling the program: you attach to the binary already on disk. (There's uretprobe for when the function returns, like kretprobe.)

USDT: probes the application bakes in

uprobe attaches to any function, but function names can change between versions. USDT (User Statically-Defined Tracing) is a set of stable probe points the application's author bakes directly into the code — like tracepoints but in userspace. Many large runtimes ship USDT (Node.js, Python, PostgreSQL, JVM, libc). List the USDT in libc:

sudo bpftrace -l 'usdt:/usr/lib/x86_64-linux-gnu/libc.so.6:*'

usdt:/usr/lib/x86_64-linux-gnu/libc.so.6:libc:cond_broadcast
usdt:/usr/lib/x86_64-linux-gnu/libc.so.6:libc:cond_destroy
usdt:/usr/lib/x86_64-linux-gnu/libc.so.6:libc:cond_init
usdt:/usr/lib/x86_64-linux-gnu/libc.so.6:libc:cond_signal

Attach to USDT with usdt:<binary>:<provider>:<name>. Unlike uprobe, USDT is a contract the author commits to keeping, so it's more stable for observing application logic (e.g. a database's queries, a runtime's GC).

Inspecting a pod from the host

This is the crux for Kubernetes: a container is just a process on the host (same kernel, different namespace). eBPF attaches to the host kernel, so it sees every process of every pod — filter by PID or cgroup and you can inspect a specific pod, with no sidecar to install and no pod to modify. Find a pod's process (cilium-agent), then count the syscalls it makes:

cpid=$(pgrep -x cilium-agent | head -1)        # PID of the in-pod process, seen from the host
sudo bpftrace -e "tracepoint:syscalls:sys_enter_* /pid == $cpid/ { @[probe] = count(); }"

@[...sys_enter_write]:      1897
@[...sys_enter_waitid]:      172
@[...sys_enter_socket]:       18
@[...sys_enter_tgkill]:       12
@[...sys_enter_setsockopt]:   10
@[...sys_enter_unlinkat]:      2

The filter /pid == $cpid/ narrows down to exactly the pod's process; @[probe] = count() counts by syscall name (readable, not a number). We see cilium-agent mostly does write (logs/maps) and opens sockets — observed from the outside, without touching the pod, without installing an agent in the container. This generalizes: filter by cgroup (to catch a multi-process pod) or by network namespace — which is exactly how tools like Pixie, Cilium Hubble, and Parca observe Kubernetes workloads without instrumenting each app. One attach point on the host kernel sees every pod on it.

🧹 Cleanup

bpftrace detaches itself on exit; there's nothing to clean, the node returns to 140 programs. Commands are at github.com/nghiadaulau/ebpf-from-scratch, directory 08-uprobe-usdt.

Wrap-up

eBPF reaches beyond the kernel: uprobe attaches to a userspace function in an existing binary/library (we traced getaddrinfo in libc, seeing python3 resolve github.com — via arg0 + str()), USDT attaches to the stable probes an application bakes in (libc, Node, Python, PostgreSQL...) via usdt:<binary>:<vendor>:<name>. And because a container is a process on the host's shared kernel, eBPF on the host kernel can inspect every pod: filter /pid == .../ (or by cgroup) to see the syscalls a pod makes — we saw cilium-agent mostly doing write/socket without installing anything in the pod. This is the foundation of modern Kubernetes observability (Hubble, Pixie): one attach point on the host, the whole cluster in view.

Part II closes — we've observed the system with bpftrace from one-liner to histogram to inspecting a pod, all without writing C. Part III goes back to writing full eBPF programs ourselves with libbpf + CO-RE (C), then loading them from Go (cilium/ebpf) — for when you need a real tool, not just an ad-hoc command line.