NetworkPolicy: A Firewall By Label

K
Kai··8 min read

In Articles 13–14 we built a flat network model: every pod can ping every pod, every port. That's a baseline requirement of Kubernetes — but it also means one compromised pod can scan the whole cluster. Article 46 replaced the datapath with Cilium eBPF, and Cilium implements NetworkPolicy. This article uses it to block traffic at the pod layer by label, and watches Hubble print each blocked packet.

The kubernetes.io docs spell it out: NetworkPolicy only takes effect if the CNI implements it. On a cluster with a CNI that doesn't support policy (like the plain bridge of Article 14 before migration), NetworkPolicy objects can still be created but block nothing. Our cluster now runs Cilium so policy has real effect — that's why this article comes after Article 46 rather than in Part I.

The model: a pod is "isolated" or not

NetworkPolicy is a namespaced object, selecting target pods via podSelector. The core rule lives in one word: isolated:

        Pod NOT selected by any policy        Pod selected by a policy (Ingress)
        ───────────────────────────          ──────────────────────────────────
   ┌────────────────────────────┐        ┌────────────────────────────────┐
   │  non-isolated               │        │  isolated for Ingress           │
   │  → accepts ALL inbound       │        │  → DENY all, except what's      │
   │  → sends ALL outbound        │        │     listed in ingress.from      │
   └────────────────────────────┘        └────────────────────────────────┘
        (the cluster default)               (on the moment a policy selects it)

By default, every pod is non-isolated — accepts and sends freely (just like our cluster so far). A pod becomes isolated in one direction the moment a NetworkPolicy selects it and lists that direction in policyTypes. Once isolated for Ingress, only sources matching ingress.from can get in; every other source is blocked. The egress direction works the same way. Reply traffic of an already-allowed connection is always allowed, because policy is evaluated at the connection level.

Two consequences people forget: policies are additive (union) — multiple policies selecting one pod allow it if any policy allows; and for A to talk to B, both A's egress and B's ingress have to permit it (if both are restricted).

To test, we use a web pod, two clients frontend (will be allowed) and other (will be blocked).

Step 1 — no policy yet: every pod can reach web

Stand up a dedicated namespace for tidiness and a simple HTTP server (agnhost netexec, like Article 20):

kubectl create namespace np-demo
kubectl -n np-demo run web --image=registry.k8s.io/e2e-test-images/agnhost:2.45 \
  --labels="app=web" --port=8080 -- netexec --http-port=8080
kubectl -n np-demo run frontend --image=busybox:1.36 --labels="role=frontend" -- sleep 100000
kubectl -n np-demo run other    --image=busybox:1.36 --labels="role=other"    -- sleep 100000
kubectl -n np-demo get pods -o wide
NAME       READY   STATUS    IP             NODE
frontend   1/1     Running   10.200.1.107   worker-1
other      1/1     Running   10.200.0.160   worker-0
web        1/1     Running   10.200.0.141   worker-0

No policy yet → web is non-isolated → both clients reach it (/hostname returns the pod name web):

kubectl -n np-demo exec frontend -- wget -qO- --timeout=4 http://10.200.0.141:8080/hostname
kubectl -n np-demo exec other    -- wget -qO- --timeout=4 http://10.200.0.141:8080/hostname
web      # frontend reaches it
web      # other reaches it

A flat network, exactly like Article 14. Now lock it down with a first policy.

Step 2 — default-deny: block all ingress to web

The first policy only selects app=web and declares policyTypes: [Ingress], listing no from. The docs call this the default-deny-ingress pattern: the selected pod becomes isolated for Ingress, and since no source is listed, no source can get in:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web-deny-ingress
  namespace: np-demo
spec:
  podSelector:
    matchLabels:
      app: web
  policyTypes:
  - Ingress

No ingress: block means the list of allowed sources is empty → web is isolated, no source gets through. Apply it and try again:

kubectl -n np-demo apply -f web-deny-ingress.yaml
kubectl -n np-demo exec frontend -- wget -qO- --timeout=4 http://10.200.0.141:8080/hostname
kubectl -n np-demo exec other    -- wget -qO- --timeout=4 http://10.200.0.141:8080/hostname
wget: download timed out      # frontend blocked
wget: download timed out      # other blocked

Both time out. One policy selecting one target pod is enough to cut every path into web from within the namespace. Note the result is a timeout rather than connection refused: Cilium silently drops the SYN packet so the client waits until it expires, unlike a live server that actively refuses.

Step 3 — open exactly one door for frontend

Add a second policy that allows ingress, but only from pods labeled role=frontend, only on port 8080:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web-allow-frontend
  namespace: np-demo
spec:
  podSelector:
    matchLabels:
      app: web
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          role: frontend
    ports:
    - protocol: TCP
      port: 8080
kubectl -n np-demo apply -f web-allow-frontend.yaml
kubectl -n np-demo exec frontend -- wget -qO- --timeout=4 http://10.200.0.141:8080/hostname
kubectl -n np-demo exec other    -- wget -qO- --timeout=4 http://10.200.0.141:8080/hostname
kubectl -n np-demo get netpol
web                           # frontend can reach it again
wget: download timed out      # other still blocked
NAME                 POD-SELECTOR   AGE
web-allow-frontend   app=web        8s
web-deny-ingress     app=web        29s

Both policies select app=web, and they're additive: web-deny-ingress opens no source, web-allow-frontend opens role=frontend, the union is "only frontend gets in". other matches no source so it still drops. In practice you usually write only the allow policy — a single policy selecting a pod already isolates it, so the allow policy itself also denies the rest; the separate web-deny-ingress pattern here is just to show the two steps clearly.

On selector-combination semantics, the docs draw a sharp distinction that's very easy to get wrong:

ingress:
- from:
  - podSelector: {role: frontend}      ┐  two items WITHIN the same from[]
  - namespaceSelector: {team: a}       ┘  = OR  (frontend  OR  pod in ns team=a)

- from:
  - namespaceSelector: {team: a}       ┐  two selectors WITHIN one item
    podSelector: {role: frontend}      ┘  = AND (pod role=frontend  AND  in ns team=a)

The difference is one dash: two separate - is OR, merged under one - is AND (intersection).

Hubble's verdict

Because the cluster runs Cilium eBPF (Article 46), there's no need to guess why it timed out — Hubble records the verdict for each flow. Regenerate traffic, then inspect flows to web right on worker-0's cilium-agent (where web runs):

CIL=$(kubectl -n kube-system get pod -l k8s-app=cilium \
  --field-selector spec.nodeName=worker-0 -o jsonpath='{.items[0].metadata.name}')
kubectl -n kube-system exec $CIL -c cilium-agent -- hubble observe --to-pod np-demo/web --last 12 -o compact
np-demo/other:58020 (ID:46753) <> np-demo/web:8080 (ID:60437) Policy denied DROPPED (TCP Flags: SYN)
np-demo/frontend:54530 (ID:26159) -> np-demo/web:8080 (ID:60437) to-endpoint FORWARDED (TCP Flags: SYN)
np-demo/frontend:54530 (ID:26159) -> np-demo/web:8080 (ID:60437) to-endpoint FORWARDED (TCP Flags: ACK)
np-demo/frontend:54530 (ID:26159) -> np-demo/web:8080 (ID:60437) to-endpoint FORWARDED (TCP Flags: ACK, FIN)
np-demo/other:55938 (ID:46753) <> np-demo/web:8080 (ID:60437) Policy denied DROPPED (TCP Flags: SYN)

Read it straight off: frontend's SYN is FORWARDED, other's SYN is Policy denied DROPPED. No tcpdump, no reading iptables rules (which are gone since Article 46) — the policy verdict shows up right at the observability layer. To debug a policy that isn't behaving as intended, this is the first tool to reach for.

Policy translates labels into identity

Note the ID:xxxxx numbers in the Hubble log. Those are Cilium identities, exactly the mechanism described in Article 45. Cilium doesn't write rules by pod IP, because IPs change every time a pod is recreated; instead it assigns each set of labels a numeric identity, and policy is translated into "which identity can talk to which identity". Inspect:

kubectl -n kube-system exec $CIL -c cilium-agent -- cilium-dbg identity get 26159 | grep role
kubectl -n kube-system exec $CIL -c cilium-agent -- cilium-dbg identity get 46753 | grep role
k8s:role=frontend      # identity 26159  ← this is the label role=frontend
k8s:role=other         # identity 46753  ← this is the label role=other

Identity 26159 is role=frontend. When frontend is deleted and recreated with a different IP but the same labels, it still carries identity 26159 and policy still applies correctly — no gap while the IP catches up in the rules. This is the fundamental difference from an IP-based firewall: we declare who can talk to whom by label, and Cilium handles translating it to the datapath.

🧹 Cleanup

kubectl delete namespace np-demo

Deleting the namespace sweeps away all 3 pods and 2 NetworkPolicies. The cluster returns to baseline: only kube-system running, every pod Running, no test workloads. Keep Cilium + Hubble intact. Manifests at github.com/nghiadaulau/kubernetes-from-scratch, directory 47-networkpolicy.

Wrap-up

NetworkPolicy turns Article 14's flat network into one segmented by label. The root rule: a non-isolated pod accepts everything, but the moment a policy selects it and declares policyTypes, the pod becomes isolated for that direction — only sources matching ingress.from/egress.to get through, the rest are blocked; multiple policies are additive. We tested all three states for real: allow-all (no policy) → deny-all (select app=web, open no door → both clients time out) → selective allow (open for role=frontend → only frontend gets in, other still drops). Two things Cilium adds: Hubble prints the FORWARDED/Policy denied DROPPED verdict per packet, and cilium-dbg identity shows policy attaching to an identity derived from labels (26159 = role=frontend) rather than IPs, exactly the "identity-based security" mechanism from Article 45. One thing to remember: on a CNI that doesn't support policy, these YAMLs do nothing.

NetworkPolicy controls pod-to-pod traffic inside the cluster. Article 48 heads to the cluster edge with Ingress and an Ingress controller, bringing HTTP traffic from outside into Services and routing by host/path.