Probes: liveness, readiness and startup
Article 18 built a checklist of pod conditions and still owed one item: Ready. Article 19 paid back Initialized; this article pays back Ready, and the thing behind it is the probe. A probe is how the kubelet periodically "examines" a container to answer three different questions, each matched to one kind of probe:
- liveness: is the container still alive? If it's hung, kill it and restart.
- readiness: is the container ready to receive traffic? If not, remove it from the Service.
- startup: has the app inside the container finished starting? If not, leave the other two probes alone.
These three are often lumped into one ("is it running?") but the consequence of each is entirely different. This whole article is about separating them, and verifying each on a real cluster.
Four ways to "examine" and three outcomes
Before getting into each kind, you need to know how the kubelet examines. The docs list four mechanisms (check mechanisms), i.e. the exec/httpGet/tcpSocket/grpc part of a probe declaration:
exec: runs a command inside the container; exit code 0 is success, non-zero is failure.httpGet: sends an HTTP GET to the container; a status code from 200 to below 400 is success.tcpSocket: tries to open a TCP socket to a port; if it opens, that's success.grpc: calls a health check using the gRPC standard.
Each examination yields one of three outcomes: Success, Failure, or Unknown (the kubelet couldn't examine, e.g. the container isn't up yet). The three kinds of probe above share these same four mechanisms; they differ only in what the kubelet does on a Failure outcome. That's the part worth learning.
All three also share one set of fields tuning the examination cadence (defaults quoted from the docs):
| Field | Default | Meaning |
|---|---|---|
initialDelaySeconds |
0 | How long to wait before the first examination |
periodSeconds |
10 | Interval between examinations |
timeoutSeconds |
1 | Max time to wait for a reply before treating it as failure |
successThreshold |
1 | How many consecutive successes to be considered healthy again |
failureThreshold |
3 | How many consecutive failures before acting |
In the tests below I'll tune these fields for speed and easy observation, but the defaults above are what applies if you declare nothing.
Liveness: still alive, kill if hung
The definition in the docs is one line: "livenessProbe: Indicates whether the container is running. If the liveness probe fails, the kubelet kills the container, and the container is subjected to its restart policy." That is, this probe doesn't ask "is the process still running" (the kubelet already knows that) but "is the process still responsive," catching exactly the disease of an app frozen solid while its process hasn't yet died. The docs say clearly when it's useful: "useful for catching bugs where an application gets stuck in an unresponsive state and cannot recover except by being restarted", and also say plainly when it's not needed: "a liveness probe is not needed if the container will simply exit if the process crashes", because then restartPolicy (Article 18) already handles the restart.
Try an exec-style liveness probe. The container below creates the file /tmp/healthy, stays healthy for 30 seconds, then deletes the file, simulating an app that runs fine then keels over and hangs:
apiVersion: v1
kind: Pod
metadata:
name: liveness-exec
labels: {test: liveness}
spec:
containers:
- name: liveness
image: busybox:1.36
args: ["/bin/sh","-c","touch /tmp/healthy; sleep 30; rm -f /tmp/healthy; sleep 600"]
livenessProbe:
exec:
command: ["cat","/tmp/healthy"] # cat returns 0 = alive; file gone => non-zero = dead
initialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 3
For the first 30 seconds, cat /tmp/healthy exits code 0 → probe Success, the pod runs normally. After second 30 the file disappears and the probe starts failing. With periodSeconds: 5 and failureThreshold: 3, the kubelet needs three consecutive failures (≈15 seconds) before acting. Catch the pod after about a minute and a half:
kubectl get pod liveness-exec
kubectl get pod liveness-exec -o jsonpath='restartCount={.status.containerStatuses[0].restartCount}{"\n"}lastState={.status.containerStatuses[0].lastState.terminated.reason} exitCode={.status.containerStatuses[0].lastState.terminated.exitCode}{"\n"}'
NAME READY STATUS RESTARTS AGE
liveness-exec 1/1 Running 1 (29s ago) 105s
restartCount=1
lastState=Error exitCode=137
The container was restarted once (RESTARTS 1), and lastState records the previous death: Error, exitCode 137. The number 137 is 128 + 9, i.e. SIGKILL. It's KILL rather than a clean shutdown because PID 1 in the container (busybox's sh) installs no SIGTERM handler so it ignores the polite shutdown signal; the kubelet waits out the grace period then SIGKILLs. A real app should catch SIGTERM to shut down cleanly, but here the 137 conveniently shows us the kubelet forcibly killing a broken container.
The truth lives in the events:
kubectl events --for pod/liveness-exec
LAST SEEN TYPE REASON MESSAGE
60s (x3 over 70s) Warning Unhealthy Liveness probe failed: cat: can't open '/tmp/healthy': No such file or directory
60s Normal Killing Container liveness failed liveness probe, will be restarted
30s (x2 over 105s) Normal Pulled Container image "busybox:1.36" already present on machine ...
30s (x2 over 105s) Normal Created Container created
30s (x2 over 105s) Normal Started Container started
Read top to bottom and it's the whole story: Liveness probe failed ... (x3), exactly three times as failureThreshold set; then Killing ... will be restarted; then Created/Started a second time (x2). That's the restart lifecycle of Article 18, but this time the one pressing the restart button is the liveness probe, not the process dying on its own. That's the core difference: liveness turns "hung but not dead" into "dead for real, then restarted."
Readiness: ready to receive traffic yet
Liveness asks are you still alive. Readiness asks something quite different: are you ready to receive requests yet. A container can be perfectly alive but not have finished loading its cache, not yet connected to the database, restarting a pool: alive, but not yet ready to receive traffic. The docs: "readinessProbe: Indicates whether the container is ready to receive traffic. If the readiness probe fails, the endpoints controller removes the Pod's IP address from the endpoints of all Services that match the Pod."
Note the consequence is quite different from liveness: a readiness failure does not kill the container. It only removes the pod's IP from the Service's endpoints; the pod keeps running, it just no longer receives requests through the Service. This is the mechanism behind Article 18's Ready condition, and the gate into the EndpointSlice we saw in Article 16. Pair it with a Service to see the full path:
apiVersion: v1
kind: Pod
metadata:
name: ready-demo
labels: {app: ready-demo}
spec:
containers:
- name: app
image: registry.k8s.io/e2e-test-images/agnhost:2.52
args: ["netexec","--http-port=8080"]
readinessProbe:
exec:
command: ["cat","/tmp/ready"] # /tmp/ready doesn't exist yet => not ready
initialDelaySeconds: 2
periodSeconds: 3
failureThreshold: 1
---
apiVersion: v1
kind: Service
metadata:
name: ready-demo
spec:
selector: {app: ready-demo}
ports:
- {port: 80, targetPort: 8080}
The file /tmp/ready is deliberately not there, so readiness fails immediately. The pod will be Running (the container is alive) but 0/1 (not ready):
kubectl get pod ready-demo
kubectl get pod ready-demo -o jsonpath='Ready={..conditions[?(@.type=="Ready")].status} ContainersReady={..conditions[?(@.type=="ContainersReady")].status}{"\n"}'
kubectl get endpointslices -l kubernetes.io/service-name=ready-demo \
-o jsonpath='{range .items[*]}endpoints={.endpoints[*].addresses} ready={.endpoints[*].conditions.ready}{"\n"}{end}'
kubectl get endpoints ready-demo
NAME READY STATUS RESTARTS AGE
ready-demo 0/1 Running 0 22s
Ready=False ContainersReady=False
endpoints=["10.200.1.13"] ready=false
NAME ENDPOINTS AGE
ready-demo 23s
Three observations agree: the pod is 0/1 Running; the conditions Ready=False and ContainersReady=False; and most importantly, the EndpointSlice does record the pod's address (10.200.1.13) but flags it ready=false, while the legacy Endpoints object has an empty ENDPOINTS column. In other words, kube-proxy will not send Service traffic to this pod. This is what the readiness probe guards: a pod that isn't ready is not in the list of request recipients, even though it's running.
Now make it "ready" by creating the file the probe is looking for, then watch the endpoint flip:
kubectl exec ready-demo -- touch /tmp/ready
sleep 6
kubectl get pod ready-demo
kubectl get endpointslices -l kubernetes.io/service-name=ready-demo \
-o jsonpath='{range .items[*]}endpoints={.endpoints[*].addresses} ready={.endpoints[*].conditions.ready}{"\n"}{end}'
kubectl get endpoints ready-demo
NAME READY STATUS RESTARTS AGE
ready-demo 1/1 Running 0 43s
endpoints=["10.200.1.13"] ready=true
NAME ENDPOINTS AGE
ready-demo 10.200.1.13:8080 44s
Probe Success → the Ready condition flips True → the pod becomes 1/1 → the EndpointSlice changes to ready=true → the Endpoints object now lists 10.200.1.13:8080. From this second, kube-proxy routes Service traffic into the pod. Same pod address, same running container, only the readiness outcome decides whether it's in the traffic path.
Compare the two probes directly to drive it home: on the same Failure outcome, liveness kills the container, while readiness only removes it from the endpoints. Mixing them up has serious consequences: putting a "can I connect to the database" check into liveness would cause apps to be restarted en masse when the database flickers (and restarting helps nothing); putting it into readiness means the app just temporarily leaves the endpoints and returns on its own when the database recovers. That's why you must separate them.
Startup: gatekeeper for slow-starting apps
There's one more situation: an app that starts slowly, like a JVM loading, a cache warming up, a migration taking a full minute. If you plug in liveness right away, the probe fails while the app is still coming up and the kubelet wrongfully kills it before it's ready. We could widen liveness's initialDelaySeconds to something large, but then once the app is running steadily, liveness reacts slowly to a real failure. The startup probe exists to resolve exactly this conflict.
The definition in the docs is extremely short but carries a lot: "startupProbe: Indicates whether the application within the container has started. All other probes are disabled if a startup probe is provided, until it succeeds." That is, as long as the startup probe hasn't succeeded, liveness and readiness are fully disabled — the app is left alone to start. Once the startup probe succeeds for the first time, it retires permanently and hands off to liveness/readiness.
To prove "the other two probes are disabled," I set a trap: the container takes 25 seconds to create /tmp/started, with a startup probe watching that file, and a liveness probe also watching that file but configured extremely aggressively at periodSeconds: 2, failureThreshold: 1. If liveness ran while the app was starting up, it would kill the container after just 2 seconds (file not there yet). Whether the container survives past 25 seconds tells us whether liveness was really disabled:
apiVersion: v1
kind: Pod
metadata:
name: startup-demo
spec:
containers:
- name: app
image: busybox:1.36
args: ["/bin/sh","-c","sleep 25; touch /tmp/started; sleep 600"]
startupProbe:
exec:
command: ["cat","/tmp/started"]
periodSeconds: 3
failureThreshold: 20 # allow up to 20 x 3 = 60s to start
livenessProbe:
exec:
command: ["cat","/tmp/started"]
periodSeconds: 2
failureThreshold: 1 # aggressive: 1 fail kills — BUT disabled by startup
The fields failureThreshold: 20 × periodSeconds: 3 mean the startup probe tolerates the app taking up to 60 seconds to come up before giving up (the formula failureThreshold × periodSeconds = max startup time). Our app comes up at second 25 so it's within the limit. Observe:
kubectl get pod startup-demo
kubectl get pod startup-demo -o jsonpath='restartCount={.status.containerStatuses[0].restartCount} started={.status.containerStatuses[0].started}{"\n"}'
kubectl events --for pod/startup-demo
NAME READY STATUS RESTARTS AGE
startup-demo 1/1 Running 0 118s
restartCount=0 started=true
LAST SEEN TYPE REASON MESSAGE
119s Normal Started Container started
118s Normal Scheduled Successfully assigned default/startup-demo to worker-0
95s (x8 over 116s) Warning Unhealthy Startup probe failed: cat: can't open '/tmp/started': No such file or directory
The evidence is here. restartCount=0: the container was not restarted once, even though liveness was aggressive to the point of 1-fail-kills. The events only record Startup probe failed (x8) over the first ~25 seconds, with not a single Liveness probe failed or Killing line. Liveness was disabled by the startup probe, exactly as the docs say. When /tmp/started appears at second 25, the startup probe is Success, started=true, and from then on liveness takes over (and now it passes too, since the file exists). Without the startup probe, that failureThreshold: 1 liveness would have killed the container at second 2 and the pod would be stuck in an endless CrashLoopBackOff, never getting time to start.
An operational note from the docs: don't have multiple containers in the same pod share one endpoint for the startup probe — "this can cause the startup sequence to fail if the endpoint does not return success".
The three probes side by side
time ───────────────────────────────────────────────►
|== container starting up ==|========= running steadily =========>
STARTUP [examine... examine... ✓] (done, then retires for good)
└ blocks liveness/readiness throughout this phase
LIVENESS (disabled) [examine regularly... ✗✗✗ → KILL + restart]
still alive? kill if hung
READINESS (disabled) [examine regularly... ✗ → remove from Service]
ready yet? if not, leave the endpoints
(container STILL alive)
Three questions, three consequences: startup gatekeeps until the app is up; liveness kills and restarts a hung container; readiness removes from endpoints a not-yet-ready container without touching it. They share the same exec/httpGet/tcpSocket/grpc and the same cadence fields, differing only in what the kubelet does on a probe Failure.
🧹 Cleanup
kubectl delete pod liveness-exec ready-demo startup-demo --now
kubectl delete svc ready-demo
All are objects in the cluster — deleting them is clean, and the cluster returns to just the two resident CoreDNS pods. Manifests at github.com/nghiadaulau/kubernetes-from-scratch, directory 20-probes.
Wrap-up
A probe is how the kubelet periodically examines a container, and the key is that the three kinds of probe answer three different questions with three different consequences on failure. Liveness asks are you still alive; on failure the kubelet kills the container and lets restartPolicy restart it (we saw exitCode 137 and the Killing event). Readiness asks are you ready to receive traffic; on failure the endpoints controller removes the pod's IP from the Service without killing the container (we saw the Ready condition flip and the EndpointSlice toggle ready true/false, the exact mechanism behind Article 18's Ready condition and Article 16's EndpointSlice). Startup asks has the app finished starting; it disables the other two probes until it succeeds, giving slow apps time to come up without being wrongfully killed (we saw restartCount=0 through the startup window even with an extremely aggressive liveness config). The four check mechanisms and the periodSeconds/failureThreshold/... fields are shared; choosing the right kind of probe for the right intent is the hard part.
With this, Article 18's condition checklist is fully explained: Initialized (Article 19) and Ready (this article). Article 21 moves on to debugging pods when there's no shell available: ephemeral containers and kubectl debug, how to slip a tooling container into a running pod without restarting it or modifying its image.