KubernetesCiliumObservabilityeBPFHubble

Inside Hubble: From eBPF Events to Cluster-Wide Network Flows

K
Kai··5 min read·1 views
eBPF From ScratchPart 20 of 22
PreviousOff-CPU and Scheduler Latency: Measuring the Time a Process Is NOT RunningNextCase Study: A Packet Through Cilium's eBPF Datapath

Articles 17-18 observed the performance of a single node. This article closes Part VI with cluster-wide network observation: Hubble — the thing that lets us see every connection between pods, by service name, with the policy verdict, without installing anything into a pod. It is the top layer of the "eBPF as a foundation for observability" chain. We dissect how it works, from the datapath up to labeled flows.

Three layers: datapath → perf ring → enrichment

Hubble is not a separate agent plugged into a pod. It is the top layer of a three-step chain, the lower two of which we have already met:

   [1] eBPF datapath (74 sched_cls, Article 12) handles each packet
            │  bpf_perf_event_output()  -- pushes trace/drop/policy events
            ▼
   [2] cilium_events  (BPF map perf_event_array, one slot per CPU)
            │  cilium-agent opens a perf reader, reads raw events
            ▼  (cilium monitor shows this layer: NUMERIC identity)
   [3] Hubble: enrich identity -> pod/service name + verdict
            ▼
   readable flow: "host -> kube-system/coredns:8080 FORWARDED"

Layer 2: the cilium_events perf ring

The map that the datapath pushes events into is exactly a perf_event_array we met in Article 12:

sudo bpftool map show | grep cilium_events

23: perf_event_array  name cilium_events  max_entries 2     <- one slot per CPU (node has 2 CPUs)

Cilium's sched_cls programs (cil_from_container, cil_to_netdev...) call the helper bpf_perf_event_output() to write an event record into cilium_events whenever something worth reporting happens (a packet forwarded, dropped, a policy verdict). The perf_event_array has one slot per CPU (max_entries 2 = 2 cores) — each CPU pushes into its own slot, avoiding contention on the hot path. This is a relative of the ring buffer (Article 9): an efficient mechanism for pushing events from the kernel up to userspace.

Reading layer 2 raw: cilium monitor

cilium monitor opens exactly that perf reader on cilium_events and prints raw events — precisely what the datapath pushes up, not yet enriched:

sudo cilium monitor

-> endpoint 645  identity 16777217->18203 state reply  ifindex lxc9d33...  10.0.1.11:6443 -> 10.200.0.140:59902 tcp ACK
-> endpoint 1797 identity host->35393    state new    ifindex lxca0f2...  10.200.0.64:36830 -> 10.200.0.180:9808 tcp SYN
-> stack         identity 35393->host    state reply  ifindex 0           10.200.0.180:9808 -> 10.200.0.64:36830 tcp SYN,ACK

This reveals the structure of a datapath event: direction (-> endpoint delivers into a pod, -> stack up to the host stack, -> network out to the NIC), source→destination identity (numeric!), state (new/established/reply — from conntrack, Article 12), ifindex (which pod's veth), and the actual packet (IP:port, TCP flags). Note: identities here are numeric — 18203, 35393, 16777217. Those are Cilium's security identities (Article 12), not yet human-readable names. This layer is the raw eBPF event.

Layer 3: Hubble enriches into names

hubble observe reads the same event stream but enriches the numeric identities into pod/service names:

hubble observe --last 12 -o compact

10.200.0.64:54764 (host) -> kube-system/coredns-87bb947d6-v29lc:8080 (ID:18203) to-endpoint FORWARDED (TCP Flags: SYN)
10.200.0.64:54764 (host) <- kube-system/coredns-87bb947d6-v29lc:8080 (ID:18203) to-stack    FORWARDED (TCP Flags: SYN,ACK)
10.200.0.64:55954 (host) -> kube-system/ebs-csi-node-qq4nk:9808     (ID:35393) to-endpoint FORWARDED (TCP Flags: ACK,FIN)
10.0.1.20:47106   (host) -> 10.0.1.10:6443                          (world)    to-network  FORWARDED (TCP Flags: ACK,PSH)

The same flows as cilium monitor, but now identity 18203 shows up as kube-system/coredns-87bb947d6-v29lc, 35393 as ebs-csi-node-qq4nk, and 16777217 as world (outside the cluster). The verdict FORWARDED (or DROPPED if a policy blocks) is explicit. The source of the enrichment is the identity→label store (Article 12):

cilium identity get 18203

18203   k8s:k8s-app=kube-dns
        k8s:io.kubernetes.pod.namespace=kube-system
        k8s:io.cilium.k8s.policy.serviceaccount=coredns

Hubble looks up the number 18203 to get this k8s label set, matches it with the corresponding endpoint/pod, and produces a readable name. All the original information — who talks to whom, what verdict — is already present in the eBPF event; Hubble only translates numbers into names.

Why this model is powerful

No pod gets a sidecar, no application is modified. The eBPF datapath already handles every packet of every pod (Article 12) — adding one bpf_perf_event_output immediately gives a complete event stream, labeled by the identity that Cilium already uses to enforce policy. That is why Hubble-style network observation is nearly free architecturally: it reuses the very datapath that is already routing and enforcing policy. Compared to the old model (one packet-capturing agent per node using libpcap, or a sidecar in every pod), eBPF gives the same visibility from a single attachment point in the kernel, at much lower cost.

🧹 Cleanup

This article only reads the events already flowing (cilium monitor, hubble observe, cilium identity get) — it attaches and modifies nothing. The node still has 140 programs. Commands at github.com/nghiadaulau/ebpf-from-scratch, directory 19-hubble.

Wrap-up

Hubble builds the cluster-wide network flow picture from a three-layer chain, reusing the existing eBPF datapath. [1] Cilium's sched_cls programs (Article 12) call bpf_perf_event_output() to push trace/drop/policy events into [2] cilium_events, a one-slot-per-CPU perf_event_array (a relative of the Article 9 ring buffer). [3] cilium-agent reads that perf ring — cilium monitor shows the raw events: direction (endpoint/stack/network), source→destination identity as numbers, state (from conntrack), ifindex, the actual packet. Hubble enriches: it looks up the security identity (18203) to get k8s labels (k8s-app=kube-dns) from the identity→label store and resolves it into a pod name (kube-system/coredns-...), along with the verdict FORWARDED/DROPPED. All the data is already in the eBPF event; Hubble only translates numbers into names. Because the datapath already sees every packet, this kind of observation is nearly free — no sidecar, no app changes, one attachment point in the kernel sees the whole cluster.

Part VI closes — we observed performance from on-CPU profiling, off-CPU latency, through to cluster-wide network flows. Part VII is the final part: a Cilium end-to-end case study — assembling every piece of the series (verifier, maps, XDP, tc, tail call, perf ring, identity) into the complete picture of one pod packet traveling from one container to another, then a wrap-up of the eBPF journey from scratch.

eBPF From ScratchPart 20 of 22
PreviousOff-CPU and Scheduler Latency: Measuring the Time a Process Is NOT RunningNextCase Study: A Packet Through Cilium's eBPF Datapath

Related Posts

The Tetragon Way: From Observe to Enforc...
SecurityKubernetes

The Tetragon Way: From Observe to Enforce with bpf_send_signal

Tetragon is the Cilium ecosystem's runtime security tool: it observes with kprobe/tracepoint (the very hooks Part II used) and then enforces inside the kernel. Its enforcement uses two helpers — bpf_send_signal sends SIGKILL to kill a process, and bpf_override_return overrides a syscall's return value. This article rebuilds that: an exec tracepoint calls bpf_send_signal(SIGKILL) the moment a process runs — a forbidden binary gets exit 137, a normal binary still runs. No LSM, no reboot.

K
KaiMay 24, 2026· 1 views
LSM BPF: Enforcing Security Right Inside...
SecurityLinux

LSM BPF: Enforcing Security Right Inside the Kernel

So far our eBPF has only observed. LSM BPF enforces: it attaches to the kernel's security hooks (Linux Security Modules) that SELinux and AppArmor use, and a program returns 0 to allow or -EPERM to block. This article writes an LSM program that blocks opening a file, and hits a lesson: it loaded and attached but blocked nothing — because bpf wasn't an active LSM. After enabling bpf via a boot parameter and rebooting, it blocks for real — both cat and python get Operation not permitted.

K
KaiMay 24, 2026