Sign Every Certificate by Hand with cfssl

Article 3 left us six empty machines and a fully-equipped workstation. Today is the first truly "Kubernetes" step, and also the part kubeadm hides most: signing all the cluster's certificates. In Article 2 we drew the diagram of who presents which cert to whom; this article turns that diagram into concrete .pem files.

We use CloudFlare's cfssl instead of plain openssl, because with a large number of certs cfssl is much tidier: each cert is described by a short JSON file and signed with one command. Everything is done on the workstation, in the directory ~/k8s-scratch/pki, and only later articles push each file up to the machine that needs it.

What we're about to create

   Kubernetes CA ──┬── kube-apiserver        (server, full SAN)
                   ├── apiserver-kubelet-client
                   ├── admin                 (O=system:masters)
                   ├── kube-controller-manager
                   ├── kube-scheduler
                   ├── kube-proxy
                   ├── worker-0  (CN=system:node:worker-0, O=system:nodes)
                   ├── worker-1  (CN=system:node:worker-1, O=system:nodes)
                   └── service-account       (token-signing key pair)

   etcd CA ────────┬── etcd                  (server+peer, SAN of 3 controllers)
                   └── apiserver-etcd-client

   front-proxy CA ─── front-proxy-client

Prepare the directory and the signing config file

cfssl needs a config file describing how to sign: what the cert is for (server auth, client auth) and how long it's valid. We use one "profile" named kubernetes for everything, valid one year (8760h):

mkdir -p ~/k8s-scratch/pki && cd ~/k8s-scratch/pki

cat > ca-config.json <<'EOF'
{
  "signing": {
    "default": { "expiry": "8760h" },
    "profiles": {
      "kubernetes": {
        "usages": ["signing", "key encipherment", "server auth", "client auth"],
        "expiry": "8760h"
      }
    }
  }
}
EOF

We let the profile combine both server auth and client auth for simplicity. In production people usually split the two so a cert plays exactly one role, but here combining them is less fuss while still being correct.

Step 1 — Three Certificate Authorities

Each CA is a self-signed key pair. cfssl gencert -initca takes a CSR describing the CA's CN then produces <name>.pem (cert) and <name>-key.pem (private key). Create three CSR files for the three CAs:

cat > ca-csr.json <<'EOF'
{ "CN": "kubernetes-ca", "key": {"algo":"rsa","size":2048},
  "names": [{"O":"Kubernetes","OU":"CA","C":"VN"}] }
EOF
cat > etcd-ca-csr.json <<'EOF'
{ "CN": "etcd-ca", "key": {"algo":"rsa","size":2048},
  "names": [{"O":"etcd","OU":"CA","C":"VN"}] }
EOF
cat > front-proxy-ca-csr.json <<'EOF'
{ "CN": "front-proxy-ca", "key": {"algo":"rsa","size":2048},
  "names": [{"O":"front-proxy","OU":"CA","C":"VN"}] }
EOF

Then generate all three:

cfssl gencert -initca ca-csr.json          | cfssljson -bare ca
cfssl gencert -initca etcd-ca-csr.json     | cfssljson -bare etcd-ca
cfssl gencert -initca front-proxy-ca-csr.json | cfssljson -bare front-proxy-ca

2026/05/23 20:37:43 [INFO] generating a new CA key and certificate from CSR
2026/05/23 20:37:43 [INFO] generate received request
2026/05/23 20:37:43 [INFO] generating key: rsa-2048
2026/05/23 20:37:43 [INFO] signed certificate with serial number 6831289342662456212...
...

cfssljson -bare ca splits cfssl's JSON output into two files, ca.pem and ca-key.pem. After this step we have three CA pairs. These three private keys *-ca-key.pem are the most sensitive thing in the whole cluster — whoever has them can sign forged certs for any identity.

Step 2 — Certs for the control plane clients

The first four certs are all client certs, no SAN needed, just the right CN and O so the api-server reads the identity (recall Article 2: CN becomes user, O becomes group). Define a short function so we don't repeat the signing parameters:

GEN() { cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes "$@"; }

admin is the identity we use to administer the cluster, set with O=system:masters for full permissions:

cat > admin-csr.json <<'EOF'
{ "CN":"admin","key":{"algo":"rsa","size":2048},
  "names":[{"O":"system:masters","OU":"k8s-scratch","C":"VN"}] }
EOF
GEN admin-csr.json | cfssljson -bare admin

Three control plane components, each an identity with a built-in RBAC binding in Kubernetes — the CN must be exactly the system:kube-... string:

cat > kube-controller-manager-csr.json <<'EOF'
{ "CN":"system:kube-controller-manager","key":{"algo":"rsa","size":2048},
  "names":[{"O":"system:kube-controller-manager","C":"VN"}] }
EOF
GEN kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

cat > kube-scheduler-csr.json <<'EOF'
{ "CN":"system:kube-scheduler","key":{"algo":"rsa","size":2048},
  "names":[{"O":"system:kube-scheduler","C":"VN"}] }
EOF
GEN kube-scheduler-csr.json | cfssljson -bare kube-scheduler

cat > kube-proxy-csr.json <<'EOF'
{ "CN":"system:kube-proxy","key":{"algo":"rsa","size":2048},
  "names":[{"O":"system:node-proxier","C":"VN"}] }
EOF
GEN kube-proxy-csr.json | cfssljson -bare kube-proxy

When signing these certs, cfssl prints a warning This certificate lacks a "hosts" field. For a client cert that warning is harmless — the hosts field (SAN) is only needed for a server cert, while a client cert is identified by its CN/O, not by an address. Just ignore it.

Step 3 — A kubelet cert for each worker

This is where CN/O matters most, as said in Article 2. Each worker needs its own cert with CN=system:node:<node-name> and O=system:nodes. The system:node: prefix turns on the Node authorizer, limiting each kubelet to touch only resources belonging to its node. A kubelet cert is both a client (calling up to the api-server) and a server (the api-server calls down to fetch logs/exec), so it also needs a SAN with the node's hostname and IP.

A small function, called for each worker:

gen_kubelet() {  # $1=host  $2=ip
  local host=$1 ip=$2
  cat > ${host}-csr.json <<EOF
{ "CN":"system:node:${host}","key":{"algo":"rsa","size":2048},
  "hosts":["${host}","${ip}"],
  "names":[{"O":"system:nodes","C":"VN"}] }
EOF
  GEN -hostname=${host},${ip} ${host}-csr.json | cfssljson -bare ${host}
}
gen_kubelet worker-0 10.0.1.20
gen_kubelet worker-1 10.0.1.21

Double-check the CN and SAN are correct — this verify step is worth doing, because one wrong character in the CN means the kubelet is rejected when it joins:

openssl x509 -in worker-0.pem -noout -subject -ext subjectAltName

subject=C = VN, O = system:nodes, CN = system:node:worker-0
X509v3 Subject Alternative Name:
    DNS:worker-0, IP Address:10.0.1.20

Step 4 — The server cert for kube-apiserver

This cert needs the fullest SAN field, because the api-server is reached at many different addresses (recall Article 2). The SAN list includes:

• 127.0.0.1 and localhost — internal calls on the controller machine itself.
• The IP of each controller 10.0.1.11/12/13 — direct calls between nodes.
• The load balancer's private IP 10.0.1.10 and its Elastic IP — because kubectl from your machine goes through the load balancer.
• The DNS names kubernetes, kubernetes.default, ... .svc.cluster.local — the internal Service name pods use to call the api-server.
• 10.32.0.1 — the first ClusterIP of the Service range, which is the virtual address of the kubernetes Service.

Why the Elastic IP is here. kubectl on your laptop calls the api-server via the public IP of lb-0. EC2's auto-assigned public IP changes on every stop/start, but the SAN is fixed in the cert — if the IP jumps, kubectl reports x509: certificate is valid for ... not .... So here we assign lb-0 a fixed Elastic IP and put it in the SAN. If you didn't do this in Article 3, do it now: aws ec2 allocate-address --domain vpc then aws ec2 associate-address --instance-id <lb-0> --allocation-id <eip>. In this example the Elastic IP is 203.0.113.10.

APISERVER_SANS="10.32.0.1,10.0.1.10,203.0.113.10,10.0.1.11,10.0.1.12,10.0.1.13,127.0.0.1,localhost,lb-0,controller-0,controller-1,controller-2,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local"

cat > kube-apiserver-csr.json <<'EOF'
{ "CN":"kube-apiserver","key":{"algo":"rsa","size":2048},
  "names":[{"O":"Kubernetes","C":"VN"}] }
EOF
GEN -hostname="${APISERVER_SANS}" kube-apiserver-csr.json | cfssljson -bare kube-apiserver

Confirm the SAN is complete:

openssl x509 -in kube-apiserver.pem -noout -ext subjectAltName

X509v3 Subject Alternative Name:
    DNS:localhost, DNS:lb-0, DNS:controller-0, DNS:controller-1, DNS:controller-2,
    DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc,
    DNS:kubernetes.default.svc.cluster, DNS:kubernetes.default.svc.cluster.local,
    IP Address:10.32.0.1, IP Address:10.0.1.10, IP Address:203.0.113.10,
    IP Address:10.0.1.11, IP Address:10.0.1.12, IP Address:10.0.1.13, IP Address:127.0.0.1

Step 5 — The api-server's two client certs and the service-account key pair

The api-server is also a client in two conversations: it calls down to kubelet, and it reads/writes etcd. The cert for calling kubelet is set with O=system:masters so it's allowed to call any kubelet:

cat > apiserver-kubelet-client-csr.json <<'EOF'
{ "CN":"kube-apiserver-kubelet-client","key":{"algo":"rsa","size":2048},
  "names":[{"O":"system:masters","C":"VN"}] }
EOF
GEN apiserver-kubelet-client-csr.json | cfssljson -bare apiserver-kubelet-client

And the service-account key pair — as said in Article 2, it's used to sign and verify ServiceAccount tokens. Here we generate it as a CA-signed cert for convenience; what we actually use later is the key part: controller-manager uses the private key to sign tokens, the api-server uses the public key to verify them.

cat > service-account-csr.json <<'EOF'
{ "CN":"service-accounts","key":{"algo":"rsa","size":2048},
  "names":[{"O":"Kubernetes","C":"VN"}] }
EOF
GEN service-account-csr.json | cfssljson -bare service-account

Step 6 — Certs signed by the etcd CA and front-proxy CA

etcd has its own CA. The etcd server cert needs a SAN covering all three controllers (because etcd runs stacked on all three and they talk peer-to-peer) plus 127.0.0.1. We use one cert for both the server and peer roles to keep it simple:

ETCD_GEN() { cfssl gencert -ca=etcd-ca.pem -ca-key=etcd-ca-key.pem -config=ca-config.json -profile=kubernetes "$@"; }

ETCD_SANS="127.0.0.1,localhost,controller-0,controller-1,controller-2,10.0.1.11,10.0.1.12,10.0.1.13"
cat > etcd-csr.json <<'EOF'
{ "CN":"etcd","key":{"algo":"rsa","size":2048}, "names":[{"O":"etcd","C":"VN"}] }
EOF
ETCD_GEN -hostname="${ETCD_SANS}" etcd-csr.json | cfssljson -bare etcd

# api-server → etcd (client)
cat > apiserver-etcd-client-csr.json <<'EOF'
{ "CN":"kube-apiserver-etcd-client","key":{"algo":"rsa","size":2048}, "names":[{"O":"etcd","C":"VN"}] }
EOF
ETCD_GEN apiserver-etcd-client-csr.json | cfssljson -bare apiserver-etcd-client

The front-proxy CA signs a single client cert, used for the aggregation layer (we configure the matching flag in Article 7, used later when extending the API):

cat > front-proxy-client-csr.json <<'EOF'
{ "CN":"front-proxy-client","key":{"algo":"rsa","size":2048}, "names":[{"O":"front-proxy","C":"VN"}] }
EOF
cfssl gencert -ca=front-proxy-ca.pem -ca-key=front-proxy-ca-key.pem \
  -config=ca-config.json -profile=kubernetes front-proxy-client-csr.json | cfssljson -bare front-proxy-client

Verify the trust chain

Before wrapping up, check that every cert really is signed by its proper CA. openssl verify reporting OK means the trust chain matches:

openssl verify -CAfile etcd-ca.pem etcd.pem apiserver-etcd-client.pem
openssl verify -CAfile front-proxy-ca.pem front-proxy-client.pem
openssl verify -CAfile ca.pem kube-apiserver.pem admin.pem worker-0.pem kube-proxy.pem

etcd.pem: OK
apiserver-etcd-client.pem: OK
front-proxy-client.pem: OK
kube-apiserver.pem: OK
admin.pem: OK
worker-0.pem: OK
kube-proxy.pem: OK

The full inventory of the identities we just created:

ca                         CN=kubernetes-ca           (CA)
etcd-ca                    CN=etcd-ca                 (CA)
front-proxy-ca             CN=front-proxy-ca          (CA)
kube-apiserver             CN=kube-apiserver          O=Kubernetes
apiserver-kubelet-client   CN=kube-apiserver-kubelet-client  O=system:masters
apiserver-etcd-client      CN=kube-apiserver-etcd-client     O=etcd
service-account            CN=service-accounts        O=Kubernetes
etcd                       CN=etcd                    O=etcd
admin                      CN=admin                   O=system:masters
kube-controller-manager    CN=system:kube-controller-manager
kube-scheduler             CN=system:kube-scheduler
kube-proxy                 CN=system:kube-proxy       O=system:node-proxier
worker-0                   CN=system:node:worker-0    O=system:nodes
worker-1                   CN=system:node:worker-1    O=system:nodes
front-proxy-client         CN=front-proxy-client      O=front-proxy

Fifteen cert/key pairs, all sitting on the workstation in ~/k8s-scratch/pki. We haven't pushed them to any node yet — that's done gradually in the bootstrap articles, when each component needs its own cert.

🧹 A security note, not cleanup

This article creates no cloud resources, so there's nothing to clean up. But the pki directory now holds private keys — especially ca-key.pem, etcd-ca-key.pem, front-proxy-ca-key.pem. Keep them on the workstation, don't commit them to Git, don't share them. In this series we keep them in one directory for convenient learning; in practice, a CA's private key should be stored somewhere separate with restricted access.

Wrap-up

We now have ID cards for every component in the cluster: three CAs and twelve leaf certs, each with the right CN/O/SAN as in Article 2's PKI diagram, and we've verified the trust chain. This is the most laborious part but also the part that makes you understand most clearly why, later, each component can trust the others.

We have the certs but no component can use them yet, because they need to be bundled together with the api-server address and the CA into a format the Kubernetes binaries read: kubeconfig. Article 5 will generate kubeconfigs for admin, controller-manager, scheduler, kube-proxy and each kubelet, and also create the Secret encryption config (encryption at rest) — finishing the config preparation before we switch on etcd in Article 6.