Secrets, the Detour and Hardening

Article 31 introduced the Secret as a kind of object, and Article 52 showed that view can't read it. This article closes Part XI with the two questions still left: where a Secret lives and whether it's actually safe, and who can read it. The second answer has a detour many people don't anticipate, so we'll rebuild it by hand.

Secrets in etcd: encrypted

Article 5 set --encryption-provider-config on the API server to encrypt Secrets before writing them to etcd. Verify it by reading etcd directly, bypassing the API server:

kubectl -n sec56 create secret generic topsecret --from-literal=password=hunter2
# read straight from etcd on controller-0
sudo etcdctl --cacert=/etc/etcd/etcd-ca.pem --cert=/etc/etcd/etcd.pem --key=/etc/etcd/etcd-key.pem \
  get /registry/secrets/sec56/topsecret

Prefix stored in etcd: /registry/secrets/sec56/topsecret k8s:enc:...
Is there a plaintext hunter2 string in etcd?  0

The value starts with k8s:enc: — marking it as having gone through the encryption provider, and hunter2 doesn't appear in raw form (grep counts 0). If Article 5 hadn't enabled encryption, anyone who could read the etcd file or a backup would read the whole Secret. The point to remember in reverse: base64 in a Secret manifest is not encryption — data.password: aHVudGVyMg== decodes straight back to hunter2, so Secret manifests shouldn't be pushed to Git like ordinary code.

The detour: whoever can create a pod can read the Secret

RBAC blocks reading a Secret directly (Article 52), but there's another way. The docs say it outright: whoever can create a pod that uses a Secret can read that Secret's value, even without permission to read the Secret. Rebuild it: a ServiceAccount creator that can only create pods and read logs, and cannot read Secrets:

kind: Role
metadata: {namespace: sec56, name: pod-creator}
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["create", "get", "list"]

SA=system:serviceaccount:sec56:creator
kubectl auth can-i get secrets --as=$SA -n sec56   # no
kubectl auth can-i create pods --as=$SA -n sec56   # yes

Reading the Secret directly with creator's token is blocked, exactly as RBAC says:

kubectl --token="$TOK" ... -n sec56 get secret topsecret

Error from server (Forbidden): secrets "topsecret" is forbidden:
  User "system:serviceaccount:sec56:creator" cannot get resource "secrets" ...

But creator can create a pod. Create a pod that puts the Secret into an environment variable and prints it to the log:

spec:
  restartPolicy: Never
  containers:
  - name: c
    image: busybox:1.36
    command: ["sh", "-c", "echo LEAKED=$SECRET"]
    env:
    - name: SECRET
      valueFrom: {secretKeyRef: {name: topsecret, key: password}}

kubectl --token="$TOK" ... -n sec56 apply -f leak.yaml     # pod/leak created
kubectl --token="$TOK" ... -n sec56 logs leak

LEAKED=hunter2

creator just read hunter2 — the Secret value it has no get permission for. The mechanism: it's the kubelet that loads the Secret into the pod, using the node's credentials, not creator's; creator's RBAC only guards the gate of creating the pod, not what the pod mounts. The practical upshot: permission to create pods (or Deployments, Jobs...) in a namespace is nearly equivalent to permission to read every Secret in that namespace. So controlling who can create workloads in each namespace matters as much as controlling who can read Secrets — and this is why you should split namespaces along real trust boundaries.

A hardening table for a self-built cluster

Gather the tightening steps Part XI and the earlier parts have touched, checked against this self-built cluster:

Hardening step	In the series	Cluster status
TLS on every component (API, etcd, kubelet)	Article 4	done
Encrypt Secrets at-rest	Article 5	done (k8s:enc:)
--authorization-mode=Node,RBAC	Article 7	done
RBAC least-privilege	Article 52	pattern exists; applying it per workload is an ops task
Bound tokens, disable auto-mount when not needed	Article 53	mechanism exists; automountServiceAccountToken:false applied per pod
Pod Security Admission (enforce restricted)	Article 54	PSA on by default; labeling namespaces is an ops task
seccomp/AppArmor/drop capabilities	Article 55	runtime applies default AppArmor; seccomp/cap declared in securityContext
Control permission to create pods (because of the Secret detour)	this article	belongs to RBAC design + namespace splitting
--anonymous-auth=false	Article 51	still on by default; RBAC is blocking anonymous
Audit logging	—	not yet on; coming in Part XIII
etcd backup + cert rotation	—	Part XIII

The cluster has the foundations (TLS, encryption, RBAC, PSA) but most of the tightening is continuous operations: labeling namespaces for PSA, writing minimal RBAC, disabling auto-mount, splitting namespaces by trust. Security isn't a single switch but the sum of many correct defaults.

🧹 Cleanup

kubectl delete namespace sec56

This article only creates one namespace with a Secret + SA + pod, touching no cluster configuration. The commands used in the article are at github.com/nghiadaulau/kubernetes-from-scratch, directory 56-secrets-hardening.

Wrap-up

Secrets are encrypted at-rest since Article 5 — reading etcd directly shows the k8s:enc: prefix and no raw value, but base64 in a manifest is not encryption. The most important detour: whoever can create a pod in a namespace can extract every Secret in it, because the kubelet loads the Secret with the node's credentials and doesn't check the pod creator's RBAC — we rebuilt it with an SA blocked from get secrets that still printed hunter2 to a log via a pod it created itself. The consequence: permission to create workloads approximates permission to read Secrets, so you must control it and split namespaces along trust boundaries. The hardening table shows the self-built cluster has its foundations (TLS, encryption, RBAC, PSA) but most of the tightening is continuous operations, with audit logging and backup/cert-rotation left for Part XIII.

Part XI closes here — from a request coming in (Article 51), through permissions (52), workload identity (53), blocking dangerous pods (54), kernel mechanisms (55), to Secrets and their detour. Part XII changes direction: extending Kubernetes itself — CustomResourceDefinition, admission webhook, operator — turning the API server into a foundation for your own API.