Add kubernetes-deployment: probes, shutdown, JVM ergonomics, HPA
Companion code for "Deploying Spring Boot 4 on Kubernetes: Probes, Graceful Shutdown, Limits and JVM Ergonomics". A dependency outage under three probe-group setups, a rolling restart under load four ways (three runs each), the JVM's ergonomic choices for nine pod shapes, one GC-heavy load under five CPU limits with throttling counters, and an HPA driven by a Micrometer gauge through prometheus-adapter. Measured on k3s v1.36.4. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3TETMrqVUWeFkNtz3Jbo3
This commit is contained in:
Claude Opus 5
@@ -0,0 +1,104 @@
|
||||
# 2. Probes: which checks belong in which probe
|
||||
|
||||
[← 1. The lab](01-the-lab.md) · [Index](../README.md) · Next: [3. JVM ergonomics →](03-jvm-ergonomics.md)
|
||||
|
||||
Spring Boot exposes two health groups for Kubernetes, `/actuator/health/liveness` and
|
||||
`/actuator/health/readiness`. They are added automatically when Boot detects Kubernetes (the
|
||||
`*_SERVICE_HOST` / `*_SERVICE_PORT` variables), and explicitly here with
|
||||
`management.endpoint.health.probes.enabled=true` so they also exist on a laptop. By default each
|
||||
contains exactly one thing - the application's `LivenessState` and `ReadinessState` - and **nothing
|
||||
else**, not the database, not `diskSpace`, not your own indicators.
|
||||
|
||||
The three probes answer different questions, and the kubelet does something different on failure:
|
||||
|
||||
| Probe | Question | On failure |
|
||||
|---|---|---|
|
||||
| startup | has it finished starting? | kill after `failureThreshold × periodSeconds`; liveness/readiness wait until it passes |
|
||||
| liveness | is this process broken beyond repair? | **kill and restart the container** |
|
||||
| readiness | should it get traffic right now? | remove from the Service; no restart |
|
||||
|
||||
## One dependency outage, three configurations
|
||||
|
||||
[`DownstreamHealthIndicator`](../src/main/java/com/ankurm/k8s/health/DownstreamHealthIndicator.java)
|
||||
calls a `downstream` service. [`demo-probes.sh`](../scripts/demo-probes.sh) scales the downstream to
|
||||
zero for 60 s and watches the two `orders` pods, the ready endpoints, and whether `GET /work` - an
|
||||
endpoint that **does not use the downstream at all** - still works through the Service.
|
||||
|
||||
**In the liveness group** ([`probes-liveness-includes-downstream.txt`](output/probes-liveness-includes-downstream.txt)):
|
||||
|
||||
```
|
||||
t= 17s 42sgx ready=true restarts=0 running | x8bjd ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
t= 23s 42sgx ready=false restarts=1 running | x8bjd ready=false restarts=1 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
-------- downstream restored --------
|
||||
t= 69s 42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running | serving endpoints=2 GET /work via Service: ok
|
||||
```
|
||||
|
||||
Both replicas were killed and restarted together - three failed liveness probes, 15 s - and the
|
||||
restarted containers could not pass their startup probe (it checks the same group) until the
|
||||
downstream came back. A dependency outage became a full outage of a service that did not need the
|
||||
dependency, and the JVMs lost their warm caches and JIT state on top. Had the outage outlasted the
|
||||
startup probe's 120 s budget, the pods would have been restarted again, into `CrashLoopBackOff`.
|
||||
|
||||
**In the readiness group** ([`probes-readiness-includes-downstream.txt`](output/probes-readiness-includes-downstream.txt)):
|
||||
|
||||
```
|
||||
t= 6s 8mf9m ready=false restarts=0 running | btcl4 ready=false restarts=0 running | serving endpoints=0 GET /work via Service: FAIL
|
||||
-------- downstream restored --------
|
||||
t= 69s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
```
|
||||
|
||||
No restarts, and recovery within seconds of the downstream returning. **But still zero serving
|
||||
endpoints within 6 s** - every replica checks the same shared dependency, so every replica leaves
|
||||
the Service at once. Clients get connection refused instead of a meaningful error, including for
|
||||
the requests that never needed the downstream.
|
||||
|
||||
**In neither group** - Spring Boot's default ([`probes-default-groups.txt`](output/probes-default-groups.txt)):
|
||||
|
||||
```
|
||||
t= 23s 7ltj7 ready=true restarts=0 running | cpnkk ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
|
||||
```
|
||||
|
||||
The pods stay in service. The requests that need the downstream fail - with whatever error your
|
||||
code returns, which can be a precise 503 - and the rest keep working.
|
||||
|
||||
## The rule this gives
|
||||
|
||||
- **Liveness**: only things a restart would fix. Boot's default `livenessState` is usually exactly
|
||||
right. A shared dependency in liveness restarts every replica at once.
|
||||
- **Readiness**: only things specific to *this* instance - warming a local cache, a full local
|
||||
queue. A shared dependency in readiness takes every replica out at once.
|
||||
- **Shared dependencies** belong in the request path: timeouts, a circuit breaker, a fast 503.
|
||||
Keep them in `/actuator/health` for dashboards - just not in a probe group.
|
||||
|
||||
## Slow starts: the startup probe
|
||||
|
||||
[`demo-startup.sh`](../scripts/demo-startup.sh) makes the context take ~45 s to refresh
|
||||
(`DEMO_STARTUP_DELAY=40s`, a sleep in a `@PostConstruct`) and rolls it out to one replica, first
|
||||
without a startup probe ([`startup-probe.txt`](output/startup-probe.txt)):
|
||||
|
||||
```
|
||||
t= 44s [gqllk ready=false restarts=0] [7ltj7 ready=true restarts=0]
|
||||
t= 50s [gqllk ready=false restarts=1] [7ltj7 ready=true restarts=0]
|
||||
t= 88s [gqllk ready=false restarts=2] [7ltj7 ready=true restarts=0]
|
||||
t=118s [gqllk ready=false restarts=2] [7ltj7 ready=true restarts=0]
|
||||
```
|
||||
|
||||
Liveness (every 5 s, three failures) starts failing the moment the container starts, because
|
||||
Tomcat is not listening yet (`connect: connection refused` in the events). The new pod is killed
|
||||
before it can ever finish starting, over and over. The first kill only *shows* as a restart at
|
||||
t=50 s because the kill runs the 5 s `preStop` sleep and then waits for the JVM to exit.
|
||||
|
||||
The old pod `7ltj7` keeps serving throughout: with `maxUnavailable: 0` the rollout simply stalls
|
||||
until `progressDeadlineSeconds` (600 s) marks it failed. With more replicas and a
|
||||
`maxUnavailable` above zero, the same change takes capacity away with every attempt.
|
||||
|
||||
The same rollout with a startup probe:
|
||||
|
||||
```
|
||||
t= 45s [7ltj7 ready=true restarts=0] [jthjg ready=false restarts=0]
|
||||
t= 51s [jthjg ready=true restarts=0]
|
||||
```
|
||||
|
||||
Ready at ~51 s, no restarts, and liveness only begins once the startup probe has passed. Size the
|
||||
startup budget (`failureThreshold × periodSeconds`, 120 s here) for your slowest start - a cold
|
||||
node, a slow config server - and keep liveness tight.
|
||||
Reference in New Issue
Block a user