Smoke Test: The Whole Cluster Running Together
The last fifteen articles built each piece: control plane, workers, container runtime, pod networking, Services, DNS. Each piece was verified on its own as it was installed. This article does something different — it runs a real application the way an end user would, and observes all the pieces working together. This is both the reward (finally kubectl apply an app) and a systematic test: each operation below, if it runs correctly, proves a specific component works.
We'll deploy a Deployment, expose it via a Service, then in turn: call it by name, watch traffic spread, read logs, get into a container, forward a port back to the laptop, delete a pod to see self-healing, and scale. The article ends with a table mapping each observation back to the article that built it.
kubectl apply ─► api-server ─► etcd (store desired state)
│
controller-mgr ┤ Deployment→ReplicaSet→Pod
scheduler ──────┤ assign Pod to a node
▼
kubelet ─► containerd ─► pod (CNI assigns IP)
▲
client pod ─DNS(CoreDNS)─► ClusterIP ─kube-proxy─► one web pod
Step 1 — Deploy a real application
Use agnhost — a Kubernetes test image whose netexec mode stands up an HTTP server that returns the pod name at the /hostname path. The pod name in the answer is exactly what lets us see which replica the traffic landed on.
cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
replicas: 3
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
containers:
- name: web
image: registry.k8s.io/e2e-test-images/agnhost:2.52
args: ["netexec", "--http-port=8080"]
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: web
spec:
selector:
app: web
ports:
- port: 80
targetPort: 8080
EOF
kubectl apply only sends the desired state to the api-server (3 replicas of this image). The part that turns desire into reality is the controller chain we built in Article 8: the controller-manager sees the Deployment, creates a ReplicaSet; the ReplicaSet creates 3 Pods; the scheduler assigns each Pod to a node; kubelet on that node calls containerd to run the container; CNI assigns the IP. See the result:
kubectl rollout status deploy/web
kubectl get pods -l app=web -o wide
deployment "web" successfully rolled out
NAME READY STATUS RESTARTS AGE IP NODE
web-578474b6fc-8rsb8 1/1 Running 0 4s 10.200.0.6 worker-0
web-578474b6fc-b67xb 1/1 Running 0 4s 10.200.1.5 worker-1
web-578474b6fc-hp4jj 1/1 Running 0 4s 10.200.0.5 worker-0
Three pods Running, spread across both nodes, each pod with an IP in its node's range. The web Service gathers them:
kubectl get svc web
kubectl get endpointslices -l kubernetes.io/service-name=web
NAME TYPE CLUSTER-IP PORT(S) AGE
web ClusterIP 10.32.0.93 80/TCP 5s
NAME ADDRESSTYPE PORTS ENDPOINTS
web-jv5dr IPv4 8080 10.200.0.5,10.200.0.6,10.200.1.5
The Service got a ClusterIP (10.32.0.93, this time letting Kubernetes assign it), and the EndpointSlice gathered exactly the three pod IPs on its own — this is the endpoint controller watching the app=web label and updating the list. Every piece is in place; now let's use it.
Step 2 — Call by name, and watch the load spread
Create a client pod, then call the Service by name web (not by IP):
kubectl run client --image=busybox:1.36 --restart=Never --command -- sleep 3600
kubectl exec client -- sh -c 'for i in 1 2 3 4 5 6; do wget -qO- http://web/hostname; echo; done'
web-578474b6fc-8rsb8
web-578474b6fc-b67xb
web-578474b6fc-8rsb8
web-578474b6fc-b67xb
web-578474b6fc-8rsb8
web-578474b6fc-b67xb
A tidy single wget http://web/... call actually drags along a whole chain: busybox asks CoreDNS (Article 15) to resolve web into 10.32.0.93; a packet to that ClusterIP gets DNAT'd by kube-proxy (Article 12) to a pod endpoint; the packet reaches the destination pod over the Article 14 network. The pod name changing between calls shows load balancing at work — each new connection lands on a different endpoint per iptables' probabilities. Name resolution confirms it too:
kubectl exec client -- nslookup web.default.svc.cluster.local
Name: web.default.svc.cluster.local
Address: 10.32.0.93
Step 3 — logs and exec
The two most reflexive commands in operations, and both go via the api-server → kubelet path we granted RBAC for in Article 9 and enabled webhook authz for in Article 11:
POD=$(kubectl get pod -l app=web -o jsonpath='{.items[0].metadata.name}')
kubectl logs $POD --tail=3
kubectl exec $POD -- hostname
I0523 14:55:35.269860 1 log.go:245] hostname: web-578474b6fc-8rsb8
I0523 14:55:35.277355 1 log.go:245] GET /hostname
I0523 14:55:35.277394 1 log.go:245] hostname: web-578474b6fc-8rsb8
web-578474b6fc-8rsb8
The log shows exactly the /hostname calls we just fired in Step 2. exec runs a command inside the container and returns the result. If the apiserver-to-kubelet RBAC or kubelet's authentication were wrong, these two commands would return a permission error — that they run means that path is open.
Step 4 — port-forward back to the laptop
kubectl port-forward opens a tunnel from your machine through the api-server down to the pod — a way to reach an internal Service without exposing it externally. It uses kubelet's portforward subresource, again the api-server → kubelet path:
kubectl port-forward svc/web 18080:80 &
curl -s http://127.0.0.1:18080/hostname; echo
web-578474b6fc-hp4jj
From the laptop, curl localhost:18080 reaches a web pod buried deep in the VPC. This is the handiest way to try an in-cluster service during development.
Step 5 — Delete a pod, watch the cluster self-heal
This is the test for the control loop — the thing that distinguishes Kubernetes from running containers by hand. The desired state is 3 replicas; if reality drifts, the controller pulls it back. Delete a pod outright:
kubectl delete pod $POD
sleep 4
kubectl get pods -l app=web
pod "web-578474b6fc-8rsb8" deleted
NAME READY STATUS RESTARTS AGE
web-578474b6fc-b67xb 1/1 Running 0 33s
web-578474b6fc-hp4jj 1/1 Running 0 33s
web-578474b6fc-n7npb 1/1 Running 0 5s
Pod 8rsb8 is gone, but a new pod n7npb (5 seconds old) has taken its place — the ReplicaSet saw only 2/3 replicas left and immediately created a replacement. No one intervened; that's the controller-manager doing its job. The EndpointSlice and kube-proxy rules updated along with it, so the Service never broke during the pod swap.
Step 6 — Scale declaratively
Finally, change the desired replica count from 3 to 4:
kubectl scale deploy/web --replicas=4
kubectl rollout status deploy/web
kubectl get deploy web
deployment.apps/web scaled
deployment "web" successfully rolled out
NAME READY UP-TO-DATE AVAILABLE AGE
web 4/4 4 4 51s
The same control loop as Step 5, this time running in the increasing direction: we declare 4, the controller creates one more pod, the scheduler finds a node for it, and 4/4 is ready. We never said how, only what we want — the rest is the controllers' job.
Which piece each test shines a light on
Taken together, this smoke test isn't one operation but a cross-section of the whole system:
| Observation | Component confirmed | Built in article |
|---|---|---|
| Pods spread across two nodes | scheduler | 8 |
| Containers run, pods have IPs | kubelet + containerd + CNI | 10, 11, 14 |
| EndpointSlice gathers pod IPs on its own | endpoint controller | 8 |
http://web is callable |
CoreDNS | 15 |
| Pod name changes between calls | kube-proxy (load balancing) | 12 |
logs / exec / port-forward |
api-server → kubelet + RBAC | 9, 11 |
| Deleting a pod recreates it | controller-manager (control loop) | 8 |
scale to the right count |
Deployment/ReplicaSet controller | 8 |
If any row breaks, you immediately know which article to go back and examine. All eight green means this self-built cluster doesn't just "come up" but really works like a full Kubernetes.
🧹 Cleanup
Delete the test application and the client pod; keep CoreDNS and the system components:
kubectl delete deploy web
kubectl delete svc web
kubectl delete pod client
There's nothing to clean up on the node or in the VPC — every resource in this step is an in-cluster object, deleting it is clean. The manifests are at github.com/nghiadaulau/kubernetes-from-scratch, directory 16-smoke-test.
Wrap-up
The cluster built by hand from nothing now runs a real application, balances load, self-heals, and scales — exactly what people expect from Kubernetes. More important than the fact that it runs is that we know why it runs: each operation in this article maps to a specific component we built ourselves, so when something breaks, troubleshooting is no longer guesswork.
This closes the "build" part. The next two articles shift to understanding more deeply what we've built. Article 17 traces the lifecycle of a request from kubectl apply until the pod runs — passing through exactly those components above, but this time in the chronological order of an object, to see how they hand off to one another. That's the bridge before we replace the networking layer with Cilium eBPF in later articles.