Logging Architecture

Part XIII turns to observability: how to know what's happening inside the cluster. We start with the most-used thing — logs. kubectl logs is so familiar we forget to ask where it gets the logs from. This article traces a real log line from kubectl down to the file on the node's disk, then separates the cluster's log types.

Where kubectl logs gets its logs

A container writes stdout/stderr, the container runtime catches it and writes it to a file on the node, and kubelet reads that file back for kubectl logs. Trace it: create a pod that prints a line every couple of seconds, pinned to worker-0:

kubectl -n log-demo run talker --image=busybox:1.36 -- \
  sh -c 'i=0; while true; do echo "dong log so $i tu talker"; i=$((i+1)); sleep 2; done'
kubectl -n log-demo logs talker --tail=2

dong log so 1 tu talker
dong log so 2 tu talker

kubectl logs reads through kubelet. The actual file sits at /var/log/pods/<ns>_<pod>_<uid>/<container>/:

ssh worker-0 'sudo ls /var/log/pods/log-demo_talker_<uid>/talker/'
ssh worker-0 'sudo tail -2 /var/log/pods/log-demo_talker_<uid>/talker/0.log'

0.log
2026-05-24T00:05:16.381709653Z stdout F dong log so 2 tu talker
2026-05-24T00:05:18.381816019Z stdout F dong log so 3 tu talker

The 0.log file is in CRI format: each line is <timestamp> <stream> <tag> <content> — stdout/stderr is the stream, F (full) or P (partial, a long line that was split). kubectl logs strips out the content again. There's an extra symlink layer at /var/log/containers/ pointing at this file, so a log-collection agent can find it in one fixed place:

ssh worker-0 'ls -l /var/log/containers/ | grep talker'

talker_log-demo_talker-6b5707f2....log -> /var/log/pods/log-demo_talker_<uid>/talker/0.log

Kubelet rotates logs

Logs can't grow forever. Kubelet rotates the file by size:

kubectl get --raw /api/v1/nodes/worker-0/proxy/configz | jq '.kubeletconfig |
  {containerLogMaxSize, containerLogMaxFiles}'

containerLogMaxSize: 10Mi
containerLogMaxFiles: 5

Each log file is at most 10Mi, keeping at most 5 files (the current 0.log + the rotated ones). One thing to remember: kubectl logs reads only the current file — rotated logs have to be pulled straight from the node. So container logs are ephemeral: they die with the pod and are size-limited, so you can't rely on them as long-term log storage.

System-component logs: journald

Container logs are one kind; the logs of the Kubernetes components themselves are another. On this cluster they run under systemd (Part I), so their logs go to journald:

ssh worker-0    'sudo journalctl -u kubelet        -n 1 -o short-iso'
ssh controller-0 'sudo journalctl -u kube-apiserver -n 1 -o short-iso'

... worker-0 kubelet[23367]: ... "Finished parsing log file" path="/var/log/pods/log-demo_talker_.../0.log"
... controller-0 kube-apiserver[7449]: ... apf_controller.go:493 "Update CurrentCL" plName="exempt" ...

This is where a self-built cluster differs from a kubeadm one: kubeadm runs the control plane as static pods, so apiserver/scheduler logs sit in /var/log/pods like a container. Our cluster runs them as systemd services (Articles 7–8), so their logs are in journald — viewed with journalctl -u <service> rather than kubectl logs. (The apiserver line above happens to reveal APF — the topic of Article 66.)

The cluster doesn't collect logs on its own

There's a common misconception: Kubernetes has no cluster-level logging built in. Logs are scattered across each node, ephemeral and rotated. To collect them into one queryable place, the common model is a node logging agent — a DaemonSet (Article 26) running on every node, mounting hostPath: /var/log, tailing the files in /var/log/containers/ and pushing them to an external storage system (Loki, Elasticsearch, CloudWatch...):

   each node:  /var/log/containers/*.log
                   │ tail
                   ▼
   agent (DaemonSet, hostPath /var/log)  ──►  log store outside the cluster

Because it's a DaemonSet, the agent shows up on a new node automatically and disappears when a node leaves — exactly why a DaemonSet fits per-node work. An alternative is a sidecar (Article 19) for applications that log to a file instead of stdout. The common thread: logs must leave the node, because storing them on the node means they're lost with the node.

🧹 Cleanup

kubectl delete namespace log-demo

This article only creates a pod that prints logs and reads them; the rest is viewing files and journald. No real agent is installed (that's an infrastructure choice that varies by site). The commands used in this article are at github.com/nghiadaulau/kubernetes-from-scratch, directory 65-logging.

Wrap-up

kubectl logs reads through kubelet from the file /var/log/pods/<ns>_<pod>_<uid>/<container>/0.log that the runtime writes, in CRI format (timestamp stream F/P content), with a symlink at /var/log/containers/ for an agent. Kubelet rotates logs by containerLogMaxSize (10Mi) and containerLogMaxFiles (5), and kubectl logs only sees the current file — so container logs are ephemeral. System-component logs go a different route: a self-built cluster runs them under systemd so their logs are in journald (journalctl -u kubelet/-u kube-apiserver), unlike a kubeadm cluster where the control plane is static pods. Kubernetes doesn't collect logs at the cluster level; the standard model is a node logging agent running as a DaemonSet, mounting /var/log, pushing logs to an external store — because logs stored on a node are lost with the node.

Logs give us discrete events. Article 66 turns to continuous metrics: the Prometheus-format /metrics endpoint that the apiserver and kubelet expose, traces, and API Priority and Fairness — the mechanism by which the apiserver protects itself from overload, which we just glimpsed in the log above.