HAProxy Consolidates Three API Servers, and Remote kubectl
The control plane has all its components, but so far we still call the api-server through 127.0.0.1 on each controller. From outside — your laptop, or the workers we're about to add — nobody knows which of the three api-servers to call. This article solves that: stand up a load balancer to consolidate the three api-servers into one address, configure kubectl on your machine, and add the RBAC needed before the workers join.
Why we need a load balancer, and why it must be TCP passthrough
The three api-servers run in parallel and as equals (Article 1: they hold no state of their own). A client just needs one stable address, and if one api-server dies the traffic should automatically shift to the other two. That's exactly the job of a load balancer.
But there's an important constraint: this load balancer must not decrypt TLS. Recall from Article 2 — the cluster uses mTLS, and the api-server reads the client's identity from the client certificate itself. If the load balancer terminates TLS (the way an HTTP LB typically does for web traffic), it breaks mTLS: the api-server no longer sees the client's original cert, can't read the identity, and all authentication collapses.
So we configure HAProxy in TCP mode (mode tcp) — it just forwards the raw byte stream to an api-server, never looking inside. TLS is still established end to end between the client and the api-server, passing through the load balancer like a pipe.
kubectl ──TLS──┐ ┌── controller-0:6443
(client cert) │ HAProxy :6443 │
├── forward bytes ───┼── controller-1:6443
worker ──TLS──┘ (mode tcp, └── controller-2:6443
does NOT open TLS)
▲ ▲
└──────── end-to-end mTLS, LB stays out of it ┘
Step 1 — Install and configure HAProxy on lb-0
HAProxy is available in the Ubuntu repositories. Install it, then write the configuration:
# on lb-0
sudo apt-get update
sudo apt-get install -y haproxy
haproxy -v | head -1
HAProxy version 2.8.16-0ubuntu0.24.04.2 2026/04/15 - https://haproxy.org/
The config has one frontend listening on :6443 and one backend listing the three api-servers. All in mode tcp:
sudo tee /etc/haproxy/haproxy.cfg >/dev/null <<'EOF'
global
log /dev/log local0
maxconn 4096
daemon
defaults
log global
mode tcp
option tcplog
timeout connect 10s
timeout client 30s
timeout server 30s
frontend kubernetes
bind *:6443
default_backend kube-apiservers
backend kube-apiservers
option tcp-check
balance roundrobin
server controller-0 10.0.1.11:6443 check
server controller-1 10.0.1.12:6443 check
server controller-2 10.0.1.13:6443 check
EOF
sudo systemctl enable haproxy
sudo systemctl restart haproxy
systemctl is-active haproxy
active
option tcp-check together with check on each server makes HAProxy probe which api-servers are alive and only send traffic to the healthy ones — the basis of fault tolerance.
Step 2 — Test the load balancer from the workstation
Now from your machine, call the api-server through lb-0's Elastic IP (which is 203.0.113.10 from Article 3). Since the api-server cert's SAN already includes this IP (Article 4), TLS will be valid:
# in ~/k8s-scratch/pki
curl -s --cacert ca.pem https://203.0.113.10:6443/healthz; echo
curl -s --cacert ca.pem --cert admin.pem --key admin-key.pem \
https://203.0.113.10:6443/version | jq -r '.gitVersion'
ok
v1.36.1
/healthz returns ok and the authenticated call sees v1.36.1 — the load balancer is forwarding correctly, and mTLS stays intact through it. If one api-server were to go down right now, the two commands above would still work thanks to the other two.
Step 3 — Configure kubectl on the laptop
Up to now we've used only curl. Let's configure kubectl properly: create a kubeconfig pointing at the Elastic IP, using the admin cert. Unlike admin.kubeconfig from Article 5 (which points at 127.0.0.1 to run on a controller), this one points out at the load balancer:
KUBECONFIG_FILE=admin-remote.kubeconfig
kubectl config set-cluster k8s-scratch \
--certificate-authority=ca.pem --embed-certs=true \
--server=https://203.0.113.10:6443 --kubeconfig=$KUBECONFIG_FILE
kubectl config set-credentials admin \
--client-certificate=admin.pem --client-key=admin-key.pem \
--embed-certs=true --kubeconfig=$KUBECONFIG_FILE
kubectl config set-context k8s-scratch \
--cluster=k8s-scratch --user=admin --kubeconfig=$KUBECONFIG_FILE
kubectl config use-context k8s-scratch --kubeconfig=$KUBECONFIG_FILE
Try a few real commands — the first time kubectl talks to the cluster through the official route:
kubectl --kubeconfig=$KUBECONFIG_FILE version
kubectl --kubeconfig=$KUBECONFIG_FILE get namespaces
kubectl --kubeconfig=$KUBECONFIG_FILE get nodes
Client Version: v1.36.1
Kustomize Version: v5.8.1
Server Version: v1.36.1
NAME STATUS AGE
default Active 14m
kube-node-lease Active 14m
kube-public Active 14m
kube-system Active 14m
No resources found
The four default namespaces are there (the api-server creates them on startup), and get nodes returns No resources found — correct, since we haven't added any workers. That's for the later articles. (To avoid typing --kubeconfig=... every time, you can export KUBECONFIG=$PWD/admin-remote.kubeconfig.)
Step 4 — RBAC so the api-server can call down to the kubelet
There's one more piece of RBAC to put in place before the workers join. When you run kubectl logs or kubectl exec, the api-server has to call down to the kubelet of the node holding the pod. The kubelet side (Article 11) will delegate the decision back to the api-server: "does the identity calling me have permission to access the node API?". For the answer to be yes, we need a ClusterRole granting permission over the kubelet's resources, bound to the identity the api-server uses when calling the kubelet — which is the CN of the apiserver-kubelet-client cert, namely kube-apiserver-kubelet-client.
cat <<'EOF' | kubectl --kubeconfig=admin-remote.kubeconfig apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: system:kube-apiserver-to-kubelet
rules:
- apiGroups: [""]
resources: ["nodes/proxy", "nodes/stats", "nodes/log", "nodes/spec", "nodes/metrics"]
verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: system:kube-apiserver
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:kube-apiserver-to-kubelet
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: kube-apiserver-kubelet-client
EOF
clusterrole.rbac.authorization.k8s.io/system:kube-apiserver-to-kubelet created
clusterrolebinding.rbac.authorization.k8s.io/system:kube-apiserver created
Our
apiserver-kubelet-clientcert hasO=system:masters(Article 4), so it already has full access and this binding is technically redundant. But this is the proper way: in a tightened environment, people don't put that cert insystem:mastersand instead grant it exactly the minimum kubelet permissions through this ClusterRole. Set it up now to get familiar with it, and sokubectl logs/execkeeps working even if you later pull the cert out ofsystem:masters.
🧹 Cleanup
HAProxy is a permanent component. Keep the admin-remote.kubeconfig file safe — it embeds the admin cert with O=system:masters, i.e. full access to the cluster. Don't commit or share it. When you stop/start the EC2 cluster, the Elastic IP stays the same so this kubeconfig still works; only the internal private IPs are unchanged by default anyway.
Wrap-up
The cluster now has a real front: one stable address through HAProxy, kubectl from the laptop controlling it, and the RBAC for the api-server to call down to the kubelet in place. What matters in terms of understanding is the reason the load balancer must be in TCP mode — it's a direct consequence of the cluster using mTLS, and a common misconfiguration if someone accidentally lets the LB terminate TLS.
The control plane section closes here. Article 10 opens the worker section with its foundational layer: the container runtime. We'll explore CRI — the interface between the kubelet and the runtime — then install containerd and runc on the two workers, preparing a place for the kubelet to put containers in Article 11.