Admission Webhook: Wedge Into the Write Path

Article 54 used Pod Security Admission, an admission controller built into the API server. But admission has a dynamic form too: the API server can call out to an external service for each write request, asking "is this object okay, or does it need modifying". That's the admission webhook — a way to wedge your own logic into the write path without modifying the API server. This article builds one from scratch, including the TLS part.

Two types, in order

There are two types of webhook, running at two different points in the admission stage:

   request ──► [Mutating webhooks] ──► [schema/builtin] ──► [Validating webhooks] ──► etcd
                  can MODIFY the object        validate            only allow / deny
                  (inject sidecar, set default)                    (sees the final object)

MutatingWebhookConfiguration runs first and may modify the object (returning a JSONPatch) — used to inject a sidecar, set default values. ValidatingWebhookConfiguration runs after and only allows or denies — used to enforce policy on the finished object. This article does the validating type, because it shows the request-response loop clearly without the object-patching part.

The API server calls a webhook with a POST containing an AdmissionReview, and waits for an AdmissionReview back with response.uid (matching the request), response.allowed (bool), plus response.status.message if denied. Because it's an HTTPS call, the webhook must run TLS and the API server must be able to trust that cert via caBundle.

The webhook server and cert

The webhook logic here is compact: deny any pod without a team label. Written in Python, read the AdmissionReview and reply:

def do_POST(self):
    body = json.loads(self.rfile.read(int(self.headers['Content-Length'])))
    req = body['request']
    labels = (req['object'].get('metadata') or {}).get('labels') or {}
    allowed = 'team' in labels
    resp = {'apiVersion':'admission.k8s.io/v1','kind':'AdmissionReview',
            'response':{'uid':req['uid'],'allowed':allowed}}
    if not allowed:
        resp['response']['status'] = {'code':403,'message':'pod phai co label "team"'}
    # ...return JSON over TLS...

The harder part is the cert. The API server calls the service via the name webhook.webhook-demo.svc, so the cert must have exactly that SAN, and be signed by a CA we'll put into the webhook config:

openssl req -x509 -new -nodes -key ca.key -subj "/CN=webhook-ca" -out ca.crt
# server cert with subjectAltName = DNS:webhook.webhook-demo.svc, signed by the CA above
openssl x509 -req -in tls.csr -CA ca.crt -CAkey ca.key -extfile csr.cnf -out tls.crt

Bring the cert into the cluster and run the server: cert into a Secret, the script into a ConfigMap, a python:3.12-alpine Deployment mounting both and listening on :8443, a Service webhook:443 pointing at it:

kubectl -n webhook-demo create secret tls webhook-tls --cert=tls.crt --key=tls.key
kubectl -n webhook-demo create configmap webhook-src --from-file=server.py
kubectl -n webhook-demo apply -f webhook-deploy.yaml   # Deployment + Service

Register the webhook

ValidatingWebhookConfiguration connects the API server to the service. caBundle is the CA that signed the cert (so the API server trusts the server), rules filter which requests call the webhook, and namespaceSelector limits scope — the key point so the webhook doesn't interfere across the whole cluster:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata: {name: require-team-label}
webhooks:
- name: require-team.kkloud.io
  rules:
  - apiGroups: [""], apiVersions: ["v1"], operations: ["CREATE"], resources: ["pods"]
  clientConfig:
    service: {namespace: webhook-demo, name: webhook, port: 443, path: "/"}
    caBundle: <base64 ca.crt>
  admissionReviewVersions: ["v1"]
  sideEffects: None
  failurePolicy: Fail
  namespaceSelector:
    matchLabels: {webhook: enabled}

failurePolicy: Fail means that if the webhook doesn't reply, the request is rejected — safer than Ignore (which lets it through when the webhook is broken) but also means a dead webhook blocks pods entirely, so it has to be kept highly available. sideEffects: None declares the webhook causes no side effects beyond replying.

Try it

Label a namespace webhook=enabled to bring it into scope, then create two pods:

kubectl label namespace wh-test webhook=enabled
kubectl -n wh-test run nolabel  --image=busybox:1.36 --command -- sleep 100000
kubectl -n wh-test run haslabel --image=busybox:1.36 --labels="team=blue" --command -- sleep 100000

Error from server: admission webhook "require-team.kkloud.io" denied the request: pod phai co label "team"
pod/haslabel created

The pod missing the label is rejected with exactly the message from the webhook; the pod with team=blue is created. The API server called out to our Python server mid-request and acted on the answer. Check scope: create a label-less pod in default (which has no webhook=enabled label):

kubectl -n default run free --image=busybox:1.36 --command -- sleep 100000

pod/free created

This pod is created — namespaceSelector makes the webhook apply only to labeled namespaces. A scope that narrow matters: a failurePolicy: Fail webhook applied cluster-wide that dies would block every pod, including system pods.

🧹 Cleanup

kubectl delete validatingwebhookconfiguration require-team-label    # delete the config FIRST
kubectl delete namespace webhook-demo wh-test

Delete the ValidatingWebhookConfiguration before deleting the server: if you delete the server first while the config remains (failurePolicy Fail), every pod CREATE in a matching namespace will be blocked because the webhook no longer replies. Manifests are at github.com/nghiadaulau/kubernetes-from-scratch, directory 58-admission-webhook.

Wrap-up

An admission webhook wedges your logic into the write path: the API server POSTs an AdmissionReview to an HTTPS service and acts on response.allowed. A mutating webhook runs first and can modify the object (via JSONPatch); a validating webhook runs after and only allows/denies. We built a real validating webhook in Python — self-signing a cert for webhook.webhook-demo.svc, putting the CA into caBundle so the API server trusts it — requiring pods to have a team label: a pod without it is rejected immediately with the message from the webhook, a pod with it passes. namespaceSelector limits scope, failurePolicy: Fail blocks when the webhook is broken (safe but demands high availability), sideEffects: None declares no side effects. The cleanup order is itself an operational lesson: remove the config first, the server after.

A CRD (Article 57) gives a data structure, a webhook (this article) wedges in at write time — but both are still passive, doing nothing on their own when an object changes. Article 59 combines them into an operator: a controller running a loop, watching a custom resource and acting to bring reality back in line with desire.