DevOpsArchitectureKuberneteskube-apiserver

kube-apiserver: The Cluster's Entry Point and the Request Pipeline

K
Kai··7 min read
Kubernetes From ScratchPart 8 of 73
Previousetcd: Quorum, Raft, and Standing Up a Three-Node ClusterNextcontroller-manager and scheduler: Control Loops and Leader Election

etcd is running, but nobody is allowed to touch it except one component: kube-apiserver. This article stands it up — the central component that every request, from your kubectl to the kubelet on a worker, must pass through. Before configuring it, we need to understand more precisely why it's the single entry point and what a request encounters as it passes through.

What the api-server does in a request

In Article 1 we saw the authn → authz → admission diagram. Now let's look closely at each stage, because the flags we're about to write are exactly what enable each one:

   request ──► [1] authn ──► [2] authz ──► [3] admission ──► [4] validate ──► etcd
               "who?"        "allowed?"     "valid/mutate?"   "right schema?"
  1. Authentication — the api-server determines who you are. In our cluster, the main method is a client certificate: the api-server reads the CN/O from the cert (the --client-ca-file flag says which CA to trust). A ServiceAccount token, meanwhile, is verified with --service-account-key-file.
  2. Authorization — what you're allowed to do. We enable two modes: Node (the Node authorizer, which scopes each kubelet to its own node) and RBAC. The flag is --authorization-mode=Node,RBAC.
  3. Admission control — plugins that intervene before the write: rejecting, or modifying the object. We enable NodeRestriction so a kubelet can't modify another node's objects.
  4. Validate and write — check the object matches the schema, then encrypt (if configured) and write it down to etcd.

The api-server has another role too: it's a client in two conversations — calling down to the kubelet (logs, exec, metrics) with the apiserver-kubelet-client cert, and reading/writing etcd with the apiserver-etcd-client cert. This is why it needs so many certs. And because the api-server holds no state of its own (everything is in etcd), we can run three instances in parallel — that's the HA part of the picture.

Step 1 — Download the control plane binaries

On each controller, download the three v1.36.1 control plane binaries (this article uses kube-apiserver; kube-controller-manager and kube-scheduler are for Article 8, but we grab them all at once for convenience):

# run on EACH controller
VER=v1.36.1
BASE=https://dl.k8s.io/release/${VER}/bin/linux/amd64
for b in kube-apiserver kube-controller-manager kube-scheduler; do
  sudo curl -fSL -o /usr/local/bin/$b ${BASE}/$b
  sudo chmod +x /usr/local/bin/$b
done
sudo mkdir -p /var/lib/kubernetes

Verify binaries after downloading. The binaries are fairly large (~90MB each), and a single mid-transfer curl network error can leave a truncated file with no clear warning. Always confirm before moving on — use -fSL so curl reports an error on HTTP failure, and check the version:

kube-apiserver --version

Kubernetes v1.36.1

Step 2 — Get the certs and encryption-config onto the controllers

The api-server needs quite a few files from the pki directory. Copy them into /var/lib/kubernetes on each controller:

# from the workstation, in ~/k8s-scratch/pki
FILES="ca.pem ca-key.pem kube-apiserver.pem kube-apiserver-key.pem \
  apiserver-kubelet-client.pem apiserver-kubelet-client-key.pem \
  apiserver-etcd-client.pem apiserver-etcd-client-key.pem etcd-ca.pem \
  service-account.pem service-account-key.pem \
  front-proxy-ca.pem front-proxy-client.pem front-proxy-client-key.pem \
  encryption-config.yaml"

for h in controller-0 controller-1 controller-2; do
  scp $FILES ${h}:/tmp/
  ssh $h 'sudo mkdir -p /var/lib/kubernetes && sudo mv /tmp/*.pem /tmp/encryption-config.yaml /var/lib/kubernetes/'
done

Each controller now has all the paperwork: its own serving cert (kube-apiserver.pem), the CA to authenticate clients (ca.pem), the certs to call the kubelet and etcd, the service-account key, the front-proxy pair for aggregation, and the encryption file.

Step 3 — systemd unit for kube-apiserver

This is the unit with the most flags in the whole series. To make it breathable, the flags are grouped by function. The unit is for controller-0 (IP 10.0.1.11); the other two machines just change --advertise-address:

[Unit]
Description=Kubernetes API Server
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \
  --advertise-address=10.0.1.11 \
  --allow-privileged=true \
  --apiserver-count=3 \
  --authorization-mode=Node,RBAC \
  --bind-address=0.0.0.0 \
  --client-ca-file=/var/lib/kubernetes/ca.pem \
  --enable-admission-plugins=NodeRestriction \
  --encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml \
  --etcd-cafile=/var/lib/kubernetes/etcd-ca.pem \
  --etcd-certfile=/var/lib/kubernetes/apiserver-etcd-client.pem \
  --etcd-keyfile=/var/lib/kubernetes/apiserver-etcd-client-key.pem \
  --etcd-servers=https://10.0.1.11:2379,https://10.0.1.12:2379,https://10.0.1.13:2379 \
  --kubelet-certificate-authority=/var/lib/kubernetes/ca.pem \
  --kubelet-client-certificate=/var/lib/kubernetes/apiserver-kubelet-client.pem \
  --kubelet-client-key=/var/lib/kubernetes/apiserver-kubelet-client-key.pem \
  --runtime-config=api/all=true \
  --service-account-key-file=/var/lib/kubernetes/service-account.pem \
  --service-account-signing-key-file=/var/lib/kubernetes/service-account-key.pem \
  --service-account-issuer=https://10.0.1.10:6443 \
  --service-cluster-ip-range=10.32.0.0/24 \
  --service-node-port-range=30000-32767 \
  --tls-cert-file=/var/lib/kubernetes/kube-apiserver.pem \
  --tls-private-key-file=/var/lib/kubernetes/kube-apiserver-key.pem \
  --requestheader-client-ca-file=/var/lib/kubernetes/front-proxy-ca.pem \
  --requestheader-allowed-names=front-proxy-client \
  --requestheader-extra-headers-prefix=X-Remote-Extra- \
  --requestheader-group-headers=X-Remote-Group \
  --requestheader-username-headers=X-Remote-User \
  --proxy-client-cert-file=/var/lib/kubernetes/front-proxy-client.pem \
  --proxy-client-key-file=/var/lib/kubernetes/front-proxy-client-key.pem \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Read it by group to make it stick:

  • etcd connection (--etcd-servers, --etcd-cafile/certfile/keyfile): points at all three etcds, using the apiserver-etcd-client cert. Each api-server can call any etcd.
  • Serving TLS (--tls-cert-file, --tls-private-key-file, --bind-address=0.0.0.0): the cert clients see when calling :6443. This is the cert with the full SAN from Article 4.
  • Authenticating clients (--client-ca-file): trust certs signed by the Kubernetes CA.
  • Calling down to the kubelet (--kubelet-client-certificate/key, --kubelet-certificate-authority).
  • Authorization (--authorization-mode=Node,RBAC, --enable-admission-plugins=NodeRestriction).
  • ServiceAccount (--service-account-*): the keys to sign/verify tokens, plus --service-account-issuer (the token issuer URL, for which we use the load balancer's address).
  • Encryption (--encryption-provider-config): points at the file created in Article 5.
  • Service network (--service-cluster-ip-range=10.32.0.0/24): the virtual ClusterIP range; recall that 10.32.0.1 is already in the SAN.
  • front-proxy / requestheader: serves the aggregation layer.

Write the file to /etc/systemd/system/kube-apiserver.service on each controller (change --advertise-address), then:

sudo systemctl daemon-reload
sudo systemctl enable kube-apiserver

Step 4 — Start and verify

Unlike etcd, the api-server doesn't need to wait for the others, so start them normally on all three:

for h in controller-0 controller-1 controller-2; do
  ssh $h 'sudo systemctl start kube-apiserver'
done

After a few seconds, check the /healthz endpoint on each machine — call 127.0.0.1:6443 with the CA so TLS is valid:

for h in controller-0 controller-1 controller-2; do
  printf "%-14s " "$h"
  ssh $h 'echo "healthz=$(curl -s --cacert /var/lib/kubernetes/ca.pem https://127.0.0.1:6443/healthz); active=$(systemctl is-active kube-apiserver)"'
done

controller-0   healthz=ok; active=active
controller-1   healthz=ok; active=active
controller-2   healthz=ok; active=active

/healthz requires no authentication (that's by design, so the load balancer can probe it). Now try an authenticated call with the admin cert to make sure both authn and authz work — list namespaces and check the version:

# temporarily copy the admin cert to controller-0 to test
curl -s --cacert ca.pem --cert admin.pem --key admin-key.pem \
  https://127.0.0.1:6443/api/v1/namespaces -o /dev/null -w "HTTP %{http_code}\n"
curl -s --cacert ca.pem --cert admin.pem --key admin-key.pem \
  https://127.0.0.1:6443/version

HTTP 200
{
  "major": "1",
  "minor": "36",
  "gitVersion": "v1.36.1",
  "gitCommit": "756939600b9a7180fc2df6550a4585b638875e67",
  "goVersion": "go1.26.2",
  "compiler": "gc",
  "platform": "linux/amd64"
}

HTTP 200 means the admin cert (with O=system:masters) passed both authn and authz. The api-server is alive and talking to etcd.

Step 5 — Verify that Secret encryption actually happens

In Article 5 we promised to prove that a Secret is encrypted in etcd, not just trust the configuration. Now let's do it for real: create a Secret through the api-server, then read the raw bytes in etcd (bypassing the api-server) to see what comes out.

# create a Secret "test-enc" with a pw field
PW=$(echo -n 'supersecret' | base64)
curl -s --cacert ca.pem --cert admin.pem --key admin-key.pem \
  -XPOST -H 'Content-Type: application/json' \
  https://127.0.0.1:6443/api/v1/namespaces/default/secrets \
  -d '{"apiVersion":"v1","kind":"Secret","metadata":{"name":"test-enc"},"data":{"pw":"'$PW'"}}' \
  -o /dev/null -w "POST HTTP %{http_code}\n"

POST HTTP 201

Read the corresponding key directly from etcd and look at the hexdump:

E="--cacert=/etc/etcd/etcd-ca.pem --cert=/etc/etcd/etcd.pem --key=/etc/etcd/etcd-key.pem"
sudo etcdctl get /registry/secrets/default/test-enc $E | hexdump -C | head -5

00000000  2f 72 65 67 69 73 74 72  79 2f 73 65 63 72 65 74  |/registry/secret|
00000010  73 2f 64 65 66 61 75 6c  74 2f 74 65 73 74 2d 65  |s/default/test-e|
00000020  6e 63 0a 6b 38 73 3a 65  6e 63 3a 61 65 73 63 62  |nc.k8s:enc:aescb|
00000030  63 3a 76 31 3a 6b 65 79  31 3a a9 0c 67 35 32 90  |c:v1:key1:..g52.|
00000040  7e cc 22 cb 1d 69 9c 5b  e3 bd b6 35 e4 dd 7d d4  |~."..i.[...5..}.|

Look at the right-hand column: after the key name comes k8s:enc:aescbc:v1:key1: followed by a pile of random bytes. The string supersecret appears nowhere — it has been encrypted with AES-CBC using key1, exactly as declared in encryption-config.yaml. If we had not enabled encryption, you'd be able to read supersecret here with the naked eye. Delete the test Secret to clean up:

curl -s --cacert ca.pem --cert admin.pem --key admin-key.pem \
  -XDELETE https://127.0.0.1:6443/api/v1/namespaces/default/secrets/test-enc \
  -o /dev/null -w "DELETE HTTP %{http_code}\n"

DELETE HTTP 200

🧹 Cleanup

The api-server is now a permanent component; don't shut it down. Note that we're still calling through 127.0.0.1 on each controller — there's no load balancer yet, so kubectl from your laptop can't reach it. Standing up HAProxy and configuring kubectl remotely is Article 9. Remember to delete the temporary cert files (admin.pem...) if you copied them up to a controller to test.

Wrap-up

The cluster's entry point is open: three api-servers running in parallel, connected to etcd over TLS, authorizing with Node and RBAC, and encrypting Secrets before writing — which we just verified down to the bytes. This is the component we'll interact with most going forward, so understanding how it takes a request and processes it through four stages is valuable.

But the api-server only stores the desired state; it doesn't turn that desire into reality on its own. That's the job of the two components in Article 8: kube-controller-manager, which runs the control loops, and kube-scheduler, which picks the node for a pod. Both are clients of the api-server (using the kubeconfigs created in Article 5), and both must elect a leader since we run three instances — we'll see leader election work for real.

Kubernetes From ScratchPart 8 of 73
Previousetcd: Quorum, Raft, and Standing Up a Three-Node ClusterNextcontroller-manager and scheduler: Control Loops and Leader Election

Related Posts

Inside Hubble: From eBPF Events to Clust...
KubernetesCilium

Inside Hubble: From eBPF Events to Cluster-Wide Network Flows

Hubble lets us see every connection in a Kubernetes cluster by pod name, service, and policy verdict — without a sidecar in any pod. This article dissects the mechanism: Cilium's eBPF datapath (the 74 sched_cls programs from Article 12) calls bpf_perf_event_output to push events into a perf ring buffer named cilium_events; cilium-agent reads them out with numeric identities; then Hubble enriches them — turning identity 18203 into kube-system/coredns — via the identity-to-label store.

K
KaiMay 24, 2026· 1 views
The Tetragon Way: From Observe to Enforc...
SecurityKubernetes

The Tetragon Way: From Observe to Enforce with bpf_send_signal

Tetragon is the Cilium ecosystem's runtime security tool: it observes with kprobe/tracepoint (the very hooks Part II used) and then enforces inside the kernel. Its enforcement uses two helpers — bpf_send_signal sends SIGKILL to kill a process, and bpf_override_return overrides a syscall's return value. This article rebuilds that: an exec tracepoint calls bpf_send_signal(SIGKILL) the moment a process runs — a forbidden binary gets exit 137, a normal binary still runs. No LSM, no reboot.

K
KaiMay 24, 2026· 1 views