# 2. Probes: which checks belong in which probe

[← 1. The lab](01-the-lab.md) · [Index](../README.md) · Next: [3. JVM ergonomics →](03-jvm-ergonomics.md)

Spring Boot exposes two health groups for Kubernetes, `/actuator/health/liveness` and
`/actuator/health/readiness`. They are added automatically when Boot detects Kubernetes (the
`*_SERVICE_HOST` / `*_SERVICE_PORT` variables), and explicitly here with
`management.endpoint.health.probes.enabled=true` so they also exist on a laptop. By default each
contains exactly one thing - the application's `LivenessState` and `ReadinessState` - and **nothing
else**, not the database, not `diskSpace`, not your own indicators.

The three probes answer different questions, and the kubelet does something different on failure:

| Probe | Question | On failure |
|---|---|---|
| startup | has it finished starting? | kill after `failureThreshold × periodSeconds`; liveness/readiness wait until it passes |
| liveness | is this process broken beyond repair? | **kill and restart the container** |
| readiness | should it get traffic right now? | remove from the Service; no restart |

## One dependency outage, three configurations

[`DownstreamHealthIndicator`](../src/main/java/com/ankurm/k8s/health/DownstreamHealthIndicator.java)
calls a `downstream` service. [`demo-probes.sh`](../scripts/demo-probes.sh) scales the downstream to
zero for 60 s and watches the two `orders` pods, the ready endpoints, and whether `GET /work` - an
endpoint that **does not use the downstream at all** - still works through the Service.

**In the liveness group** ([`probes-liveness-includes-downstream.txt`](output/probes-liveness-includes-downstream.txt)):

```
t= 17s  42sgx ready=true restarts=0 running | x8bjd ready=true restarts=0 running |  serving endpoints=2  GET /work via Service: ok
t= 23s  42sgx ready=false restarts=1 running | x8bjd ready=false restarts=1 running |  serving endpoints=0  GET /work via Service: FAIL
-------- downstream restored --------
t= 69s  42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running |  serving endpoints=2  GET /work via Service: ok
```

Both replicas were killed and restarted together - three failed liveness probes, 15 s - and the
restarted containers could not pass their startup probe (it checks the same group) until the
downstream came back. A dependency outage became a full outage of a service that did not need the
dependency, and the JVMs lost their warm caches and JIT state on top. Had the outage outlasted the
startup probe's 120 s budget, the pods would have been restarted again, into `CrashLoopBackOff`.

**In the readiness group** ([`probes-readiness-includes-downstream.txt`](output/probes-readiness-includes-downstream.txt)):

```
t=  6s  8mf9m ready=false restarts=0 running | btcl4 ready=false restarts=0 running |  serving endpoints=0  GET /work via Service: FAIL
-------- downstream restored --------
t= 69s  8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running |  serving endpoints=2  GET /work via Service: ok
```

No restarts, and recovery within seconds of the downstream returning. **But still zero serving
endpoints within 6 s** - every replica checks the same shared dependency, so every replica leaves
the Service at once. Clients get connection refused instead of a meaningful error, including for
the requests that never needed the downstream.

**In neither group** - Spring Boot's default ([`probes-default-groups.txt`](output/probes-default-groups.txt)):

```
t= 23s  7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running |  serving endpoints=2  GET /work via Service: ok
```

The pods stay in service. The requests that need the downstream fail - with whatever error your
code returns, which can be a precise 503 - and the rest keep working.

## The rule this gives

- **Liveness**: only things a restart would fix. Boot's default `livenessState` is usually exactly
  right. A shared dependency in liveness restarts every replica at once.
- **Readiness**: only things specific to *this* instance - warming a local cache, a full local
  queue. A shared dependency in readiness takes every replica out at once.
- **Shared dependencies** belong in the request path: timeouts, a circuit breaker, a fast 503.
  Keep them in `/actuator/health` for dashboards - just not in a probe group.

## Slow starts: the startup probe

[`demo-startup.sh`](../scripts/demo-startup.sh) makes the context take ~45 s to refresh
(`DEMO_STARTUP_DELAY=40s`, a sleep in a `@PostConstruct`) and rolls it out to one replica, first
without a startup probe ([`startup-probe.txt`](output/startup-probe.txt)):

```
t= 44s  [gqllk ready=false restarts=0] [7ltj7 ready=true restarts=0] 
t= 50s  [gqllk ready=false restarts=1] [7ltj7 ready=true restarts=0] 
t= 88s  [gqllk ready=false restarts=2] [7ltj7 ready=true restarts=0] 
t=118s  [gqllk ready=false restarts=2] [7ltj7 ready=true restarts=0] 
```

Liveness (every 5 s, three failures) starts failing the moment the container starts, because
Tomcat is not listening yet (`connect: connection refused` in the events). The new pod is killed
before it can ever finish starting, over and over. The first kill only *shows* as a restart at
t=50 s because the kill runs the 5 s `preStop` sleep and then waits for the JVM to exit.

The old pod `7ltj7` keeps serving throughout: with `maxUnavailable: 0` the rollout simply stalls
until `progressDeadlineSeconds` (600 s) marks it failed. With more replicas and a
`maxUnavailable` above zero, the same change takes capacity away with every attempt.

The same rollout with a startup probe:

```
t= 45s  [7ltj7 ready=true restarts=0] [jthjg ready=false restarts=0] 
t= 51s  [jthjg ready=true restarts=0] 
```

Ready at ~51 s, no restarts, and liveness only begins once the startup probe has passed. Size the
startup budget (`failureThreshold × periodSeconds`, 120 s here) for your slowest start - a cold
node, a slow config server - and keep liveness tight.