Capstone: Writing connmon — A Node-Wide TCP Connection Monitor

The final article. No new concepts — just assembling the pieces of the whole series into a real observation tool: connmon, which prints every new TCP connection on the entire node the moment it happens. It uses kprobe (Article 6), ring buffer (Article 9), CO-RE (Article 5), and the cilium/ebpf Go loader (Article 10) — the true "from scratch" spirit, ending in something that runs.

The idea: catch every tcp_connect

Every outgoing TCP connection calls the kernel function tcp_connect(). Attach a kprobe there and we catch every connection of every process on the node — without modifying or knowing in advance which application it is. From the struct sock *sk argument, read the destination address/port and push it up to userspace.

The kernel side: kprobe + ring buffer

struct event {                       // shares its layout with the Go side
    __u32 pid;
    __u32 saddr, daddr;          // __be32 (network order)
    __u16 dport;                 // host order
    __u16 af;
    char  comm[16];
};

struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);
} events SEC(".maps");

SEC("kprobe/tcp_connect")
int BPF_KPROBE(trace_tcp_connect, struct sock *sk)
{
    if (BPF_CORE_READ(sk, __sk_common.skc_family) != AF_INET)
        return 0;                                // IPv4 to keep it short

    struct event *e = bpf_ringbuf_reserve(&events, sizeof(*e), 0);
    if (!e) return 0;
    e->pid   = bpf_get_current_pid_tgid() >> 32;
    e->saddr = BPF_CORE_READ(sk, __sk_common.skc_rcv_saddr);  // CO-RE (Article 5)
    e->daddr = BPF_CORE_READ(sk, __sk_common.skc_daddr);
    e->dport = bpf_ntohs(BPF_CORE_READ(sk, __sk_common.skc_dport));
    bpf_get_current_comm(&e->comm, sizeof(e->comm));
    bpf_ringbuf_submit(e, 0);                        // push up to userspace (Article 9)
    return 0;
}

BPF_KPROBE (a libbpf macro) expands the kprobe signature to extract struct sock *sk from pt_regs. The socket fields are read with BPF_CORE_READ — relocated against this kernel's BTF (Article 5), so the same binary runs on a kernel with a different layout. Events are pushed through the ring buffer just like execsnoop (Article 9).

The userspace side: a Go loader

//go:generate go run github.com/cilium/ebpf/cmd/bpf2go -cc clang connmon connmon.bpf.c -- -I. -D__TARGET_ARCH_x86

func main() {
    rlimit.RemoveMemlock()
    objs := connmonObjects{}
    loadConnmonObjects(&objs, nil)                              // load + verifier + CO-RE + JIT
    defer objs.Close()

    kp, _ := link.Kprobe("tcp_connect", objs.TraceTcpConnect, nil)  // attach kprobe
    defer kp.Close()

    rd, _ := ringbuf.NewReader(objs.Events)                    // read the ring buffer
    defer rd.Close()

    for {
        rec, err := rd.Read()
        if errors.Is(err, ringbuf.ErrClosed) { return }
        var e event
        binary.Read(bytes.NewReader(rec.RawSample), binary.LittleEndian, &e)
        fmt.Printf("%-12s %-16s %-7d %-15s -> %s:%d\n",
            time.Now().Format("15:04:05.000"), cstr(e.Comm[:]), e.Pid,
            ipv4(e.Saddr), ipv4(e.Daddr), e.Dport)
    }
}

Exactly the skeleton of Article 10: bpf2go embeds the object + generates connmonObjects/loadConnmonObjects; link.Kprobe replaces link.Tracepoint; ringbuf.Reader reads events. ipv4() converts a __be32 into an a.b.c.d string (the low byte is the first octet).

A real build trap: kprobe needs to know the architecture

The first go generate reported a strange error:

GCC error "The eBPF is using target specific macros, please provide -target
that is not bpf, bpfel or bpfeb"

The cause: BPF_KPROBE takes its arguments from pt_regs, and the pt_regs layout differs by CPU architecture — bpf_tracing.h needs to know the target architecture via the macro __TARGET_ARCH_x86. Articles 9-10 used a tracepoint (which does not touch pt_regs) so they did not hit this; kprobe does need it. The fix: add -D__TARGET_ARCH_x86 to bpf2go's compile flags (see the go:generate directive above). A very real trap when moving from tracepoint to kprobe.

Run it: see where the whole node is connecting

go generate && go build -o connmon .
sudo ./connmon

TIME         COMM             PID     SADDR           -> DADDR:PORT
03:39:43.301 coredns          2555    127.0.0.1       -> 127.0.0.1:8080
03:39:44.082 kubelet          817     10.200.0.64     -> 10.200.0.180:9808
03:39:44.093 curl             7812    10.0.1.20       -> 10.0.1.10:6443
03:39:45.868 kubelet          817     10.200.0.64     -> 10.200.0.30:8181
03:39:48.803 kubelet          817     10.0.1.20       -> 10.0.1.20:9808

A static binary, ~5.4MB, and running it immediately shows the node's network life: coredns self-checking on port 8080, kubelet health-probing pods (:9808, :8181), and our curl going to the kube-apiserver through the LB (10.0.1.10:6443). Each line is a real tcp_connect, caught in the kernel, pushed through the ring buffer, printed in real time — without installing anything into those processes. Exit (Ctrl+C) and the kprobe detaches itself, the node returns to 140 programs. This is a miniature tcpconnect — exactly the kind of tool that production observability suites (Cilium, Pixie, Falco) build at scale.

🧹 Cleanup

connmon detaches the kprobe + ring buffer itself when it exits; the node returns to 140 programs. The full source (connmon.bpf.c, main.go, go.mod/go.sum) is at github.com/nghiadaulau/ebpf-from-scratch, directory 21-capstone-connmon; run go generate && go build.

Looking back: the eBPF journey from scratch

Twenty-one articles, one through-line:

• Part I — Foundations: the eBPF virtual machine (registers, instructions, JIT), the verifier proving safety, maps sharing state, program types & hooks, BTF + CO-RE so one binary runs across kernels. Each concept anchored to the 140 real Cilium programs on the node.
• Part II — Tracing: bpftrace from one-liner to histogram to inspecting a pod from the host — observation without writing C.
• Part III — Writing real tools: execsnoop in C (libbpf + skeleton + ring buffer) then in Go (cilium/ebpf + bpf2go).
• Part IV — Networking: XDP (firewall, the earliest hook), tc/sched_cls, dissecting Cilium's real datapath, writing our own tc.
• Part V — Security: LSM BPF (block before the fact, requires the bpf LSM enabled), seccomp (cBPF, every container), Tetragon-style (kprobe + bpf_send_signal).
• Part VI — Observability: CPU profiling (perf_event), off-CPU/scheduler latency, Hubble internals.
• Part VII — Putting it together: the journey of a packet through the datapath, and connmon in this article.

The thread through all of it: eBPF is one technology — load a small program into the kernel, the verifier proves it safe, the JIT compiles it, it attaches to a hook and shares state through maps. From that foundation grow tracing, networking, security, observability — and a project like Cilium. Starting from scratch, you can now read 140 programs on a real node, write your own tools in both C and Go, and understand why modern cloud-native infrastructure is built on eBPF.

Thanks for following to the end. All the hands-on source is at github.com/nghiadaulau/ebpf-from-scratch — clone it, run it, hack on it, and write your own eBPF programs.