Operator: CRD Plus a Reconcile Loop
Article 57 built the Widget CRD, but once you create a Widget, nothing happens — a CRD is only a data structure, no one reacts to it. The operator is the missing piece: a controller running a loop, watching a custom resource and acting to bring reality in line with the spec. This is how Kubernetes runs everything itself (Deployment creates ReplicaSet, ReplicaSet creates Pod in Article 24), now applied to our own kind. This article builds a real operator from scratch, no framework, to see that loop clearly.
What an operator is
An operator wraps two things: a CRD defining the kind, and a controller watching that kind:
user creates/edits/deletes Echo (custom resource)
│
▼ watch
┌──────────────── controller (running in a pod) ─────────────────┐
│ reconcile loop: │
│ read desired spec ──► compare to reality ──► act │
│ (message, replicas) (existing Deployment) (create/edit)│
│ then report status back │
└────────────────────────────────────────────────────────────────┘
│ create/edit
▼
Deployment (a real resource, managed by the operator)
The controller is a client of the API server, no different from kubectl — it reads the custom resource through the API, then creates/edits/deletes other resources to match. We'll make an operator managing an Echo kind: each Echo produces a Deployment running http-echo with exactly the message and replicas declared in the spec.
The Echo CRD
Reuse the pattern from Article 57. Echo has spec.message, spec.replicas, and status.ready for the operator to report back:
spec:
group: kkloud.io
names: {plural: echoes, singular: echo, kind: Echo, shortNames: [ec]}
versions:
- name: v1
served: true
storage: true
subresources: {status: {}}
schema:
openAPIV3Schema:
type: object
properties:
spec:
required: [message]
properties:
message: {type: string}
replicas: {type: integer, minimum: 1, maximum: 5, default: 1}
status:
properties: {ready: {type: string}}
A controller running in a pod
The controller here is a shell loop using kubectl — enough to show the essence of reconcile without a framework. For each Echo, it applys a Deployment carrying ownerReferences pointing back to that Echo, then updates status.ready:
while true; do
for name in $(kubectl get echoes -n $NS -o jsonpath='{.items[*].metadata.name}'); do
msg=$(kubectl get echo $name -n $NS -o jsonpath='{.spec.message}')
reps=$(kubectl get echo $name -n $NS -o jsonpath='{.spec.replicas}')
uid=$(kubectl get echo $name -n $NS -o jsonpath='{.metadata.uid}')
kubectl apply -f - <<YAML
# Deployment echo-$name, replicas=$reps, http-echo "-text=$msg"
# ownerReferences: {kind: Echo, name: $name, uid: $uid, controller: true}
YAML
ready=$(kubectl get deploy echo-$name -n $NS -o jsonpath='{.status.readyReplicas}')
kubectl patch echo $name -n $NS --subresource=status -p "{\"status\":{\"ready\":\"$ready/$reps\"}}"
done
sleep 5
done
Run it inside the cluster like a real operator: a ServiceAccount with just-enough RBAC (watch echoes, write echoes/status, manage deployments), and a Deployment using that SA. kubectl in the pod automatically uses the SA's token (Article 53), so no extra configuration is needed:
rules:
- {apiGroups: ["kkloud.io"], resources: ["echoes"], verbs: ["get","list","watch"]}
- {apiGroups: ["kkloud.io"], resources: ["echoes/status"], verbs: ["get","update","patch"]}
- {apiGroups: ["apps"], resources: ["deployments"], verbs: ["get","list","create","update","patch","delete"]}
This minimal RBAC (Article 52) is a point in itself: the operator can only touch the exact resources it manages, no more.
Reconcile when an Echo is created
Create an Echo and watch the operator react:
kubectl -n op-demo apply -f - <<'EOF'
apiVersion: kkloud.io/v1
kind: Echo
metadata: {name: hello}
spec: {message: "xin chao operator", replicas: 2}
EOF
# wait ~10s
kubectl -n op-demo get ec
kubectl -n op-demo get deploy echo-hello
NAME MESSAGE REPLICAS READY
hello xin chao operator 2 2/2
NAME READY UP-TO-DATE AVAILABLE AGE
echo-hello 2/2 2 2 8s
We only created one Echo; the operator stood up the Deployment echo-hello with 2 replicas and wrote status.ready=2/2 onto the Echo. The Deployment carries ownerReferences pointing back to the Echo:
kubectl -n op-demo get deploy echo-hello \
-o jsonpath='{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name} controller={.metadata.ownerReferences[0].controller}'
Echo/hello controller=true
Reconcile when edited, and GC when deleted
A reconcile loop means editing the spec makes the operator bring reality along. Change replicas 2→4:
kubectl -n op-demo patch echo hello --type=merge -p '{"spec":{"replicas":4}}'
# wait for the next reconcile cycle
kubectl -n op-demo get deploy echo-hello
NAME READY UP-TO-DATE AVAILABLE AGE
echo-hello 4/4 4 4 29s
The operator sees the spec change and scales the Deployment to 4. Delete the Echo and the Deployment goes with it — not because the operator cleans up, but thanks to ownerReferences + garbage collection (Article 29):
kubectl -n op-demo delete echo hello
kubectl -n op-demo get deploy echo-hello
echo.kkloud.io "hello" deleted
Error from server (NotFound): deployments.apps "echo-hello" not found
The Deployment disappears the moment the Echo is deleted, because it's a child of the Echo via ownerReferences — GC cleans up children when the parent is gone, exactly the mechanism from Article 29. The operator needs no delete-handling code; it lets the ownership relationship take care of it.
🧹 Cleanup
kubectl delete crd echoes.kkloud.io # deletes the CRD + every remaining Echo
kubectl delete namespace op-demo # deletes the operator, SA, RBAC
The manifests and reconcile script are at github.com/nghiadaulau/kubernetes-from-scratch, directory 59-operator.
Wrap-up
An operator joins a CRD (Article 57) with a controller running a reconcile loop: read the desired spec, compare to reality, act to match, report status back. We built a real operator — an Echo CRD and a shell controller running in a pod with a minimal ServiceAccount + RBAC — and saw it create a Deployment automatically when we created an Echo (status.ready=2/2), scale automatically when we edited replicas to 4, and let the Deployment be GC'd when we deleted the Echo thanks to ownerReferences (Article 29). The controller is just an API client like kubectl, running in the cluster with the SA's token (Article 53), bounded by RBAC to exactly the resources it manages (Article 52). This is the model every native Kubernetes controller uses, applied to your own kind — and it's how real operators (databases, message queues, cert-manager) package operational knowledge.
CRDs and operators extend the API server by adding data and controllers. Article 60 extends in a deeper way: API aggregation — bolting a second API server right behind the main one, serving an API group that the separate server provides. The cluster already has a running example since Article 39: metrics-server.