spring-boot-demo/kubernetes-deployment/docs/02-probes.md

2. Probes: which checks belong in which probe

← 1. The lab · Index · Next: 3. JVM ergonomics →

Spring Boot exposes two health groups for Kubernetes, /actuator/health/liveness and /actuator/health/readiness. They are added automatically when Boot detects Kubernetes (the *_SERVICE_HOST / *_SERVICE_PORT variables), and explicitly here with management.endpoint.health.probes.enabled=true so they also exist on a laptop. By default each contains exactly one thing - the application's LivenessState and ReadinessState - and nothing else, not the database, not diskSpace, not your own indicators.

The three probes answer different questions, and the kubelet does something different on failure:

Probe	Question	On failure
startup	has it finished starting?	kill after failureThreshold × periodSeconds; liveness/readiness wait until it passes
liveness	is this process broken beyond repair?	kill and restart the container
readiness	should it get traffic right now?	remove from the Service; no restart

One dependency outage, three configurations

DownstreamHealthIndicator calls a downstream service. demo-probes.sh scales the downstream to zero for 60 s and watches the two orders pods, the ready endpoints, and whether GET /work - an endpoint that does not use the downstream at all - still works through the Service.

In the liveness group (probes-liveness-includes-downstream.txt):

t= 17s  42sgx ready=true restarts=0 running | x8bjd ready=true restarts=0 running |  serving endpoints=2  GET /work via Service: ok
t= 23s  42sgx ready=false restarts=1 running | x8bjd ready=false restarts=1 running |  serving endpoints=0  GET /work via Service: FAIL
-------- downstream restored --------
t= 69s  42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running |  serving endpoints=2  GET /work via Service: ok

Both replicas were killed and restarted together - three failed liveness probes, 15 s - and the restarted containers could not pass their startup probe (it checks the same group) until the downstream came back. A dependency outage became a full outage of a service that did not need the dependency, and the JVMs lost their warm caches and JIT state on top. Had the outage outlasted the startup probe's 120 s budget, the pods would have been restarted again, into CrashLoopBackOff.

In the readiness group (probes-readiness-includes-downstream.txt):

t=  6s  8mf9m ready=false restarts=0 running | btcl4 ready=false restarts=0 running |  serving endpoints=0  GET /work via Service: FAIL
-------- downstream restored --------
t= 69s  8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running |  serving endpoints=2  GET /work via Service: ok

No restarts, and recovery within seconds of the downstream returning. But still zero serving endpoints within 6 s - every replica checks the same shared dependency, so every replica leaves the Service at once. Clients get connection refused instead of a meaningful error, including for the requests that never needed the downstream.

In neither group - Spring Boot's default (probes-default-groups.txt):

t= 23s  7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running |  serving endpoints=2  GET /work via Service: ok

The pods stay in service. The requests that need the downstream fail - with whatever error your code returns, which can be a precise 503 - and the rest keep working.

The rule this gives

• Liveness: only things a restart would fix. Boot's default livenessState is usually exactly right. A shared dependency in liveness restarts every replica at once.
• Readiness: only things specific to this instance - warming a local cache, a full local queue. A shared dependency in readiness takes every replica out at once.
• Shared dependencies belong in the request path: timeouts, a circuit breaker, a fast 503. Keep them in /actuator/health for dashboards - just not in a probe group.

Slow starts: the startup probe

demo-startup.sh makes the context take ~45 s to refresh (DEMO_STARTUP_DELAY=40s, a sleep in a @PostConstruct) and rolls it out to one replica, first without a startup probe (startup-probe.txt):

t= 44s  [gqllk ready=false restarts=0] [7ltj7 ready=true restarts=0] 
t= 50s  [gqllk ready=false restarts=1] [7ltj7 ready=true restarts=0] 
t= 88s  [gqllk ready=false restarts=2] [7ltj7 ready=true restarts=0] 
t=118s  [gqllk ready=false restarts=2] [7ltj7 ready=true restarts=0] 

Liveness (every 5 s, three failures) starts failing the moment the container starts, because Tomcat is not listening yet (connect: connection refused in the events). The new pod is killed before it can ever finish starting, over and over. The first kill only shows as a restart at t=50 s because the kill runs the 5 s preStop sleep and then waits for the JVM to exit.

The old pod 7ltj7 keeps serving throughout: with maxUnavailable: 0 the rollout simply stalls until progressDeadlineSeconds (600 s) marks it failed. With more replicas and a maxUnavailable above zero, the same change takes capacity away with every attempt.

The same rollout with a startup probe:

t= 45s  [7ltj7 ready=true restarts=0] [jthjg ready=false restarts=0] 
t= 51s  [jthjg ready=true restarts=0] 

Ready at ~51 s, no restarts, and liveness only begins once the startup probe has passed. Size the startup budget (failureThreshold × periodSeconds, 120 s here) for your slowest start - a cold node, a slow config server - and keep liveness tight.