Gateway API: The Successor to Ingress

Article 48 ended where Ingress is frozen: header-based routing, percentage traffic splitting, or separating permissions between the infrastructure owner and the application author — none of these will ever make it into Ingress. Gateway API is where they get done. It's the successor API SIG Network recommends, still running on the same Cilium we used for Ingress in the previous article.

Three objects instead of one

Ingress crams everything into one object. Gateway API splits it into three, matching three different roles in an organization:

   GatewayClass   ── whoever installs the controller (like Cilium) registers it; like IngressClass
        │
        ▼
   Gateway        ── the cluster operator opens a "gate": listener port 80/443, which hosts
        │             may attach routes. This is where the address + TLS cert are provided.
        ▼
   HTTPRoute      ── the app team attaches routes to the Gateway: host, path, header → backend Service.
                     Each team owns its own routes without touching the shared Gateway.

This split solves a real Ingress problem: in Ingress, the application author has to edit the same object as the person managing TLS and addresses. Gateway API lets the infrastructure owner hold the Gateway while teams only create HTTPRoutes pointing at it.

Enable Gateway API on Cilium

Unlike Ingress, Gateway API has no built-in CRDs in Kubernetes — you install them separately. Cilium 1.19 needs the standard channel CRDs at v1.4.1:

BASE=https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.4.1/config/crd/standard
for r in gatewayclasses gateways httproutes referencegrants grpcroutes; do
  kubectl apply -f $BASE/gateway.networking.k8s.io_${r}.yaml
done

Then enable it on Cilium (still --reuse-values to keep the kube-proxy-less + ingress config from the previous articles), and restart operator + agent like in Article 48 — helm upgrade only edits the ConfigMap, the pods don't reload on their own:

helm upgrade cilium cilium/cilium --version 1.19.4 -n kube-system --reuse-values \
  --set gatewayAPI.enabled=true
kubectl -n kube-system rollout restart deploy/cilium-operator ds/cilium

Cilium registers a GatewayClass named cilium:

kubectl get gatewayclass

NAME     CONTROLLER                     ACCEPTED   AGE
cilium   io.cilium/gateway-controller   True       23s

ACCEPTED True means the controller accepts this class. Gateway API requires kubeProxyReplacement=true and the Envoy L7 proxy — both in place since Article 46.

Gateway and HTTPRoute

Create a Gateway opening an HTTP listener on port 80, allowing only routes in the same namespace to attach:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: web-gw
  namespace: gw-demo
spec:
  gatewayClassName: cilium
  listeners:
  - name: http
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: Same

Then an HTTPRoute attached to that Gateway (parentRefs), routing by host apps.kkloud.local and path:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: fanout
  namespace: gw-demo
spec:
  parentRefs:
  - name: web-gw
  hostnames:
  - apps.kkloud.local
  rules:
  - matches:
    - path: {type: PathPrefix, value: /foo}
    backendRefs:
    - {name: foo, port: 80}
  - matches:
    - path: {type: PathPrefix, value: /bar}
    backendRefs:
    - {name: bar, port: 80}

The backends are two Services foo and bar (each a http-echo Deployment returning FOO/BAR, like Article 48). Once applied, Cilium creates a Service for the Gateway and an Envoy listener:

kubectl -n gw-demo get gateway web-gw
kubectl -n gw-demo get svc | grep gateway

NAME     CLASS    ADDRESS   PROGRAMMED   AGE
web-gw   cilium             False        52s

NAME                    TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
cilium-gateway-web-gw   LoadBalancer   10.32.0.205   <pending>     80:30731/TCP   12s

Why the Gateway isn't Programmed yet

PROGRAMMED False and EXTERNAL-IP <pending> aren't config errors — they're a consequence of running on bare metal. Check the specific reason:

kubectl -n gw-demo get gateway web-gw -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'

Accepted=True Accepted
Programmed=False AddressNotAssigned

The Gateway creates a LoadBalancer-type Service and waits to be assigned an external address. The self-built EC2 cluster has no cloud controller to hand out an IP, and we haven't enabled Cilium's LB IPAM yet (saved for Article 50), so the Service stays <pending> and the Gateway reports AddressNotAssigned. But the data plane is ready — the Envoy listener is set up:

kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg envoy admin listeners | grep gateway

gw-demo/cilium-gateway-web-gw/listener::127.0.0.1:17803

A LoadBalancer Service always gets a NodePort assigned alongside it (here 30731), so we test through that. Remember from Article 48: Cilium's NodePort doesn't listen on loopback, so use the node IP.

Routing and traffic splitting

Test from a worker, sending requests with a Host header:

IP=10.0.1.20 ; NP=30731
curl -s -H "Host: apps.kkloud.local" $IP:$NP/foo
curl -s -H "Host: apps.kkloud.local" $IP:$NP/bar
curl -s -o /dev/null -w "HTTP %{http_code}\n" -H "Host: apps.kkloud.local" $IP:$NP/baz

FOO          # /foo -> Service foo
BAR          # /bar -> Service bar
HTTP 404     # path matches no rule

So far Gateway API does exactly what Ingress does. The part Ingress can't do is here: a second HTTPRoute splits traffic between two backends by weight, with no controller-specific annotation:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: split
  namespace: gw-demo
spec:
  parentRefs:
  - name: web-gw
  hostnames:
  - split.kkloud.local
  rules:
  - backendRefs:
    - {name: foo, port: 80, weight: 80}
    - {name: bar, port: 80, weight: 20}

Call it 20 times and count:

for i in $(seq 20); do curl -s -H "Host: split.kkloud.local" $IP:$NP/; done | sort | uniq -c

   3 BAR
  17 FOO

17/3 out of 20 calls, tracking the declared 80/20 ratio. This is how you do a canary straight in the standard API: lower the weight of the old version, gradually raise the new one, with no dependency on a specific controller's annotations — which is exactly what makes Ingress config hard to port between controllers.

🧹 Cleanup

kubectl delete namespace gw-demo        # removes Gateway, 2 HTTPRoutes, foo, bar

The cilium-gateway-web-gw Service belongs to the Gateway, so it goes with the namespace. Keep the Gateway API CRDs and gatewayAPI.enabled on Cilium — Article 50 uses them next when assigning a real address to the LoadBalancer. Manifests at github.com/nghiadaulau/kubernetes-from-scratch, folder 49-gateway-api.

Wrap-up

Gateway API splits inbound traffic into three objects across three roles: GatewayClass (controller registration, Cilium creates the cilium class), Gateway (opens a listener, provides the address + TLS), HTTPRoute (the team attaches routes by host/path/header to the Gateway). We installed the v1.4.1 CRDs, enabled gatewayAPI.enabled on Cilium (remember to restart operator + agent), then stood up an HTTP Gateway on port 80 with two HTTPRoutes: path fanout (/foo→foo, /bar→bar, unknown path → 404) and weighted traffic splitting (80/20 → counted out to 17/3). On bare metal, the Gateway stops at Programmed=False / AddressNotAssigned because the LoadBalancer Service has no external IP yet, but the Envoy listener is running so we test through the accompanying NodePort. Compared to Ingress: same host/path routing, but Gateway API adds traffic splitting and role separation right in the standard API, not through per-controller annotations.

There's still a loose end: the Gateway and the LoadBalancer Service from Article 48 both wait for an external IP the bare-metal cluster can't provide. Article 50 fills that gap — Cilium's LB IPAM assigns IPs to LoadBalancer Services, along with the inbound traffic routing options (externalTrafficPolicy, topology-aware routing) that complete the path into the cluster.