DevOpsKubernetesNodekubelet

kubelet: Bringing the Workers Into the Cluster

K
Kai··9 min read
Kubernetes From ScratchPart 12 of 73
PreviousContainer Runtime and CRI: Installing containerd on the WorkersNextkube-proxy: Turning a ClusterIP Into a Real Destination

In Article 10, the two workers got a container runtime: containerd listening on /run/containerd/containerd.sock, ready to take CRI commands. But crictl is just a tool we type by hand; nobody automatically watches the cluster and tells containerd which pods to run. The one that does that is the kubelet, and this article stands it up on both workers.

The kubelet is the only process on a worker that talks directly to the api-server. It registers the node into the cluster, watches the pods assigned to its node, calls containerd over CRI to create containers, and continuously reports the health of the node and its pods back up to the api-server. Once the kubelet is installed, kubectl get nodes will return something for the first time.

Where the kubelet sits in the picture

Recall the delegation chain from Article 10. The kubelet is the link connecting the control plane to the runtime:

   api-server  ◄──────── kubelet kubeconfig (cert system:node:worker-0)
       │  ▲                   │
       │  │ report node/pod   │ CRI (gRPC)
       │  │ status            ▼
       │  └───────────────  kubelet ──► containerd ──► runc ──► [container]
       │                       ▲
       └── call down ──────────┘
          (logs/exec/proxy, RBAC from Article 9)

There are two separate directions of communication between the api-server and the kubelet, and we've prepared for both ahead of time:

  • kubelet → api-server (the upward direction): the kubelet is the client, authenticating with a kubeconfig holding the worker-0 cert. The api-server reads the identity system:node:worker-0, and the Node authorizer (enabled since Article 7) decides exactly what this kubelet may read/write related to its own node.
  • api-server → kubelet (the downward direction): when you run kubectl logs/exec, the api-server is the client calling into the kubelet. Now the kubelet must authenticate and authorize the api-server — that's the authentication/authorization part of the kubelet config below, backed by the ClusterRole we created in Article 9.

Understand these two directions and the configuration fields ahead stop being magic.

Step 1 — Distribute the certificate and kubeconfig

Each worker needs four files, all already generated earlier:

  • ca.pem — the cluster CA, for the kubelet to verify the api-server's cert (and to verify clients calling down to it).
  • worker-0.pem + worker-0-key.pem — the node's cert. This cert has CN=system:node:worker-0, O=system:nodes (Article 4) — exactly the format the Node authorizer expects. It serves both as a client cert (upward) and as the kubelet's serving cert (downward), because the profile grants both clientAuth and serverAuth.
  • worker-0.kubeconfig — the kubeconfig pointing at the load balancer https://10.0.1.10:6443 (Article 5), embedding the node cert as its identity.

The worker's kubeconfig points at lb-0's internal IP (10.0.1.10), not the Elastic IP. The workers and load balancer are in the same VPC; going through the internal IP is shorter and doesn't loop out to the Internet. The Elastic IP is only for kubectl from your laptop (Article 9).

Copy them up to each worker, then place them where the kubelet will look. Rename the node cert to a neutral name (kubelet.pem) so the unit and config use the same path on every node:

# from the machine with the pki directory (change worker-0 -> worker-1 for the second node)
W=worker-0
scp ca.pem $W.pem $W-key.pem $W.kubeconfig $W:/tmp/

ssh $W
sudo mkdir -p /var/lib/kubelet /var/lib/kubernetes
sudo mv /tmp/ca.pem            /var/lib/kubernetes/ca.pem
sudo mv /tmp/$W.pem            /var/lib/kubelet/kubelet.pem
sudo mv /tmp/$W-key.pem        /var/lib/kubelet/kubelet-key.pem
sudo mv /tmp/$W.kubeconfig     /var/lib/kubelet/kubeconfig
sudo chmod 600 /var/lib/kubelet/kubelet-key.pem /var/lib/kubelet/kubeconfig

Step 2 — Install the kubelet binary

A single static binary, pinned to v1.36.1 to match the control plane:

# on worker-0
cd /tmp
curl -fsSL -o kubelet https://dl.k8s.io/v1.36.1/bin/linux/amd64/kubelet
ls -la kubelet
sudo install -m 755 kubelet /usr/local/bin/kubelet
kubelet --version

-rw-rw-r-- 1 ubuntu ubuntu 60080393 kubelet
Kubernetes v1.36.1

60MB, glance at the size again, because as the control plane articles noted, a truncated binary doesn't report an error and only breaks at runtime.

Step 3 — Write the KubeletConfiguration

The modern kubelet reads most of its parameters from a KubeletConfiguration file instead of a forest of command-line flags — easier to read and to version-manage. Create /var/lib/kubelet/kubelet-config.yaml:

# on worker-0
sudo tee /var/lib/kubelet/kubelet-config.yaml >/dev/null <<'EOF'
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
  x509:
    clientCAFile: "/var/lib/kubernetes/ca.pem"
authorization:
  mode: Webhook
cgroupDriver: systemd
clusterDomain: "cluster.local"
clusterDNS:
  - "10.32.0.10"
containerRuntimeEndpoint: "unix:///run/containerd/containerd.sock"
resolvConf: "/run/systemd/resolve/resolv.conf"
runtimeRequestTimeout: "15m"
tlsCertFile: "/var/lib/kubelet/kubelet.pem"
tlsPrivateKeyFile: "/var/lib/kubelet/kubelet-key.pem"
registerNode: true
EOF

Each field is worth explaining:

  • authentication + authorization handle the api-server → kubelet direction. anonymous.enabled: false turns off anonymous access. x509.clientCAFile tells the kubelet to trust clients presenting a cert signed by our CA (namely the api-server with the apiserver-kubelet-client cert). webhook.enabled: true together with authorization.mode: Webhook makes the kubelet not decide permissions itself but ask the api-server back: "is this identity allowed to do X on the node API?", and the api-server answers based on the RBAC we built in Article 9. If you leave authorization.mode at the default (AlwaysAllow), anyone who can reach the kubelet's port has full access; don't do that.
  • cgroupDriver: systemd must match containerd's SystemdCgroup = true (Article 10). They're a pair: a driver mismatch makes the node unstable under resource pressure.
  • containerRuntimeEndpoint points straight at containerd's CRI socket. This is the wire connecting the kubelet to the runtime we built in the previous article.
  • clusterDNS + clusterDomain are the internal DNS address (10.32.0.10, within the Service range 10.32.0.0/24 from Article 7) that the kubelet writes into each pod's /etc/resolv.conf. CoreDNS itself isn't running yet (that's Article 15), but declaring it here ahead of time saves us from editing this later.
  • resolvConf: /run/systemd/resolve/resolv.conf: Ubuntu runs systemd-resolved, and the default /etc/resolv.conf points at 127.0.0.53. If the kubelet used that file as the upstream source for pods, DNS inside a pod would resolve back to a loopback address that's meaningless to the pod. Point straight at systemd's resolv file to avoid that.
  • tlsCertFile/tlsPrivateKeyFile are the serving cert for the api-server → kubelet direction.

Step 4 — systemd unit and startup

The unit depends on containerd: if the runtime isn't up, starting the kubelet is pointless, so declare Requires + After.

# on worker-0
sudo tee /etc/systemd/system/kubelet.service >/dev/null <<'EOF'
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=containerd.service
Requires=containerd.service

[Service]
ExecStart=/usr/local/bin/kubelet \
  --config=/var/lib/kubelet/kubelet-config.yaml \
  --kubeconfig=/var/lib/kubelet/kubeconfig \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now kubelet
sleep 5
systemctl is-active kubelet

active

There are only two command-line flags left: --config pointing at the file we just wrote, and --kubeconfig, the identity for calling up to the api-server. Look at what the kubelet does right after it comes up:

sudo journalctl -u kubelet --no-pager -n 6

kubelet_node_status.go:75] "Attempting to register node" node="worker-0"
kubelet_node_status.go:78] "Successfully registered node" node="worker-0"
apiserver.go:51] "Watching apiserver"
reflector.go:507] "Caches populated" type="*v1.Pod"
kubelet.go:2709] "SyncLoop ADD" source="api" pods=[]
desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"

The kubelet registers node worker-0 with the api-server itself (thanks to registerNode: true), then opens a watch to track which pods get assigned to it, showing pods=[] because there's nothing yet.

Step 5 — Repeat on worker-1

worker-1 is done identically, differing only in the cert set (worker-1.*) already copied in Step 1. Download the binary, write the same KubeletConfiguration and kubelet.service as above (these two files are identical between the nodes because the paths are neutral), then:

# on worker-1
sudo systemctl daemon-reload
sudo systemctl enable --now kubelet
systemctl is-active kubelet
kubelet --version

active
Kubernetes v1.36.1

Step 6 — See the nodes from the cluster

Now back on the laptop, using the admin kubeconfig (via the Elastic IP):

kubectl get nodes -o wide

NAME       STATUS     ROLES    AGE   VERSION   INTERNAL-IP   CONTAINER-RUNTIME
worker-0   NotReady   <none>   44s   v1.36.1   10.0.1.20     containerd://2.3.1
worker-1   NotReady   <none>   10s   v1.36.1   10.0.1.21     containerd://2.3.1

Both workers have joined the cluster: the right version, the right internal IP, and CONTAINER-RUNTIME showing containerd://2.3.1, proof the kubelet handshook with containerd over CRI. But the status is NotReady. That's not a bug; it's something we left in deliberately. Ask the node why:

kubectl get node worker-0 -o jsonpath='{range .status.conditions[*]}{.type}={.status} -> {.message}{"\n"}{end}'

MemoryPressure=False -> kubelet has sufficient memory available
DiskPressure=False -> kubelet has no disk pressure
PIDPressure=False -> kubelet has sufficient PID available
Ready=False -> container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

Every resource condition is healthy; only Ready=False, and the reason says it plainly: cni plugin not initialized. The kubelet refuses to report the node ready while there's no pod network config, because a Ready node implies it can run pods with networking, which isn't true yet. We installed the CNI binaries into /opt/cni/bin in Article 10 but deliberately didn't write the config; that's the job of Article 14. By then, these two nodes will flip to Ready on their own without touching the kubelet again.

One more thing we can verify right away: the api-server → kubelet path, i.e. the RBAC we built in Article 9. Call the kubelet's healthz proxy through the api-server:

kubectl get --raw "/api/v1/nodes/worker-0/proxy/healthz"; echo

ok

The api-server presented its apiserver-kubelet-client cert to the kubelet, the kubelet asked the api-server back over the webhook whether that identity had permission, the RBAC from Article 9 said yes, and the kubelet replied ok. Both directions of communication work, exactly as the diagram at the start showed.

🧹 Cleanup

The kubelet is a permanent service; leave it (it's enabled, so it comes back up after a reboot). Clean up the downloaded binary:

# on each worker
rm -f /tmp/kubelet

Security note: the files in /var/lib/kubelet, especially kubelet-key.pem and kubeconfig, are the worker's system:node identity. They're already set to 600; don't copy them off the machine. If you stop the EC2 cluster, the kubelet re-registers on boot (the internal private IPs don't change).

The full script is at github.com/nghiadaulau/kubernetes-from-scratch, in the 11-kubelet directory.

Wrap-up

The two workers are now real members of the cluster: the kubelet is running, the node is registered, it's connected to containerd, and both directions of communication with the api-server work under exactly the authentication/authorization mechanisms we built in the earlier articles. What's worth remembering is why a NotReady node is the right signal at this point: it faithfully reflects that the pod network doesn't exist yet, not that the kubelet is broken.

There's one more component that must run on the worker before we get to networking: kube-proxy, the thing that turns Kubernetes' virtual Services into forwarding rules on each node. Article 12 installs kube-proxy on the two workers, explains how it turns a ClusterIP into the real pods behind it, and lays the groundwork for the next three networking articles.

Kubernetes From ScratchPart 12 of 73
PreviousContainer Runtime and CRI: Installing containerd on the WorkersNextkube-proxy: Turning a ClusterIP Into a Real Destination

Related Posts

Inside Hubble: From eBPF Events to Clust...
KubernetesCilium

Inside Hubble: From eBPF Events to Cluster-Wide Network Flows

Hubble lets us see every connection in a Kubernetes cluster by pod name, service, and policy verdict — without a sidecar in any pod. This article dissects the mechanism: Cilium's eBPF datapath (the 74 sched_cls programs from Article 12) calls bpf_perf_event_output to push events into a perf ring buffer named cilium_events; cilium-agent reads them out with numeric identities; then Hubble enriches them — turning identity 18203 into kube-system/coredns — via the identity-to-label store.

K
KaiMay 24, 2026· 1 views
The Tetragon Way: From Observe to Enforc...
SecurityKubernetes

The Tetragon Way: From Observe to Enforce with bpf_send_signal

Tetragon is the Cilium ecosystem's runtime security tool: it observes with kprobe/tracepoint (the very hooks Part II used) and then enforces inside the kernel. Its enforcement uses two helpers — bpf_send_signal sends SIGKILL to kill a process, and bpf_override_return overrides a syscall's return value. This article rebuilds that: an exec tracepoint calls bpf_send_signal(SIGKILL) the moment a process runs — a forbidden binary gets exit 137, a normal binary still runs. No LSM, no reboot.

K
KaiMay 24, 2026· 1 views