Node Log Query and Fine-grained Kubelet Authorization

Article 65 fetched kubelet and apiserver logs with journalctl — you had to SSH into the right node. v1.36 brings two features that touch exactly that approach: querying node logs through the API instead of SSH, and tightening kubelet API access at the per-endpoint level. Both close Part XIV and connect straight back to Part I (the kubelet in Article 11, apiserver→kubelet RBAC in Article 9).

Node Log Query: node logs through the API

The kubelet can serve the logs of systemd services on the node via the /logs endpoint, gated behind a config flag. The flag is off by default — check it on the cluster:

kubectl get --raw /api/v1/nodes/worker-0/proxy/configz | jq '.kubeletconfig.enableSystemLogQuery'

false

Turn it on (add enableSystemLogQuery: true to the kubelet config then restart the kubelet — an operation on the worker, the way the kubelet was configured in Article 11), then query a service's logs straight through the API server:

kubectl get --raw "/api/v1/nodes/worker-0/proxy/logs/?query=kubelet&tailLines=3"

May 24 01:08:54 worker-0 kubelet[300312]: I0524 ... "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/cilium-operator-..."
May 24 01:09:00 worker-0 kubelet[300312]: I0524 ... "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/coredns-..."

The query=kubelet parameter selects the service; switch to containerd and you get runtime logs:

kubectl get --raw "/api/v1/nodes/worker-0/proxy/logs/?query=containerd&tailLines=1"

May 24 01:08:49 worker-0 containerd[2034]: ... level=info msg="container event discarded" ...

These are the journald logs from Article 65, but fetched through the API server instead of SSH. On a cluster with many nodes, you don't have to log into each machine to see why the kubelet or containerd is misbehaving — query through one entry point. In exchange, it opens a path to read node logs through the API, so access to that path needs to be controlled — which leads straight into the next section.

Kubelet API permissions: from one lump to fine-grained

All access to the kubelet API (logs, metrics, exec, stats, etc.) goes through the API server proxy, and the API server asks the authorizer first. Previously that question was one lump: anyone with get on nodes/proxy could reach every kubelet endpoint — logs, exec, metrics, all of it. A monitoring agent that only needs /metrics had to be granted nodes/proxy, which inadvertently gave it the right to exec into pods on the node too. v1.36 stabilizes fine-grained kubelet authorization: each endpoint becomes its own subresource (nodes/metrics, nodes/log, nodes/configz, nodes/stats, nodes/healthz, etc.), gated independently.

Build a ServiceAccount that only gets nodes/metrics, nothing else:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: {name: kubelet-metrics-only}
rules:
- {apiGroups: [""], resources: ["nodes/metrics"], verbs: ["get"]}

(plus a ClusterRoleBinding attaching it to the SA metrics-reader.) Ask the authorizer about each subresource — note the --subresource syntax, not nodes/metrics:

SA=system:serviceaccount:kauthz:metrics-reader
for sr in metrics proxy log healthz configz stats; do
  echo "nodes/$sr: $(kubectl auth can-i get nodes --subresource=$sr --as=$SA)"
done

nodes/metrics  -> yes
nodes/proxy    -> no
nodes/log      -> no
nodes/healthz  -> no
nodes/configz  -> no
nodes/stats    -> no

This SA can read the kubelet's /metrics but can't exec (nodes/proxy), can't read logs (nodes/log) or config (nodes/configz). Before fine-grained authz, giving it /metrics meant granting nodes/proxy — and with that, handing over both exec and logs. Now you grant exactly what's needed, true to the least-privilege principle of Article 52. This article's two features fit together: Node Log Query opens a path to read node logs through the API, and fine-grained authz lets you grant nodes/log separately to whoever needs to view logs without handing over exec.

🧹 Cleanup

# revert worker-0 kubelet to its old config (remove enableSystemLogQuery)
ssh worker-0 'sudo mv /var/lib/kubelet/kubelet-config.yaml.bak /var/lib/kubelet/kubelet-config.yaml; sudo systemctl restart kubelet'
kubectl delete namespace kauthz
kubectl delete clusterrole kubelet-metrics-only
kubectl delete clusterrolebinding kubelet-metrics-only

Node Log Query is a config flag — this article turned it on to try it, then reverts, returning worker-0 to its original state (enableSystemLogQuery: false). Fine-grained authz is just RBAC objects. Manifests at github.com/nghiadaulau/kubernetes-from-scratch, directory 71-node-log-kubelet-authz.

Wrap-up

Two v1.36 features touch the Part I kubelet. Node Log Query fetches node service logs (kubelet, containerd, etc.) via the kubelet's /logs?query=<service> endpoint, after enabling enableSystemLogQuery: true — we queried worker-0's kubelet and containerd logs through the API server, no SSH (versus the journalctl of Article 65). Fine-grained kubelet authorization splits kubelet API permissions, previously lumped under nodes/proxy, into independent subresources (nodes/metrics, nodes/log, nodes/configz, etc.): an SA with only nodes/metrics can read metrics but can't exec or read logs — true least-privilege instead of handing over cluster-wide access. The two complement each other: querying node logs is more convenient but opens a new surface, and fine-grained authz lets you grant exactly the narrow permission for that surface.

Part XIV closes — four groups of features that just graduated in the very v1.36 the cluster runs: admission via CEL, in-place resize, new storage, and kubelet observability/security. There's exactly one thing left for the whole series: Article 72 tears down all the infrastructure and looks back on the journey from the first certificate to here.