asmhatreandClaude Opus 5
a065696478
Companion code for "Deploying Spring Boot 4 on Kubernetes: Probes, Graceful Shutdown, Limits and JVM Ergonomics". A dependency outage under three probe-group setups, a rolling restart under load four ways (three runs each), the JVM's ergonomic choices for nine pod shapes, one GC-heavy load under five CPU limits with throttling counters, and an HPA driven by a Micrometer gauge through prometheus-adapter. Measured on k3s v1.36.4. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3TETMrqVUWeFkNtz3Jbo3
65 lines
3.9 KiB
Markdown
65 lines
3.9 KiB
Markdown
# 5. Graceful shutdown and the rolling-update race
|
|
|
|
[← 4. CPU limits and GC](04-cpu-limits-and-gc.md) · [Index](../README.md) · Next: [6. HPA on a custom metric →](06-hpa-custom-metrics.md)
|
|
|
|
When Kubernetes deletes a pod, two things start **at the same time**: the kubelet runs the
|
|
`preStop` hook and then sends SIGTERM, and the endpoints controller removes the pod from the
|
|
Service, after which every node's kube-proxy rewrites its rules. Nothing orders them. For a short
|
|
window a pod that is shutting down still receives new connections.
|
|
|
|
Spring Boot's graceful shutdown (on by default in Boot 4.1.1 - `server.shutdown=graceful`) handles
|
|
the other half: on SIGTERM, Tomcat stops accepting connections and lets in-flight requests finish,
|
|
for up to `spring.lifecycle.timeout-per-shutdown-phase` (30 s).
|
|
|
|
## The experiment
|
|
|
|
[`demo-shutdown.sh`](../scripts/demo-shutdown.sh): 20 closed-loop clients send `POST /work?ms=300`
|
|
through the Service for 45 s; at t≈8 s, `kubectl rollout restart` replaces both replicas. Each
|
|
variant three times ([`shutdown-post-summary.txt`](output/shutdown-post-summary.txt)):
|
|
|
|
```
|
|
1-immediate-no-prestop | run 1: IOException=26 [2909] | run 2: ConnectException=6 IOException=26 [2909] | run 3: ConnectException=5 IOException=24 [2915]
|
|
2-graceful-no-prestop | run 1: IOException=3 [2924] | run 2: ConnectException=8 IOException=11 [2932] | run 3: ConnectException=6 IOException=5 [2929]
|
|
3-graceful-prestop-sleep | run 1: IOException=2 [2927] | run 2: 0 failures [2942] | run 3: IOException=1 [2939]
|
|
4-graceful-prestop-exec-sh | run 1: ConnectException=4 IOException=3 [2921] | run 2: ConnectException=5 IOException=2 [2926] | run 3: ConnectException=7 IOException=1 [2930]
|
|
```
|
|
|
|
Reading it by failure type:
|
|
|
|
- **`IOException`** is a request cut off mid-flight, or sent on a keep-alive connection the server
|
|
closed. `server.shutdown=immediate` produces 24-26 per restart; graceful shutdown cuts that to a
|
|
handful.
|
|
- **`ConnectException`** is a new connection to a pod that has already stopped listening but is
|
|
still in the Service - the race above. Graceful shutdown alone does nothing for it (0, 8, 6).
|
|
**A 5 s `preStop: sleep` removed it in all three runs**: the pod keeps serving while the endpoint
|
|
removal propagates, and only then gets SIGTERM.
|
|
- **Variant 4 is variant 2 with extra steps.** An `exec` hook running `sh -c "sleep 5"` fails on
|
|
the distroless image because there is no `sh`; Kubernetes records `FailedPreStopHook`, sends
|
|
SIGTERM immediately, and the connect errors come back (4, 5, 7).
|
|
|
|
```
|
|
1 Warning FailedPreStopHook pod/orders-68fd794d97-sjq65 PreStopHook failed
|
|
1 Warning FailedPreStopHook pod/orders-68fd794d97-zsznq PreStopHook failed
|
|
```
|
|
|
|
## Why the experiment uses POST
|
|
|
|
The first version used GET and reported far fewer failures - 7 for `immediate`
|
|
([`shutdown-get-1-immediate-no-prestop.txt`](output/shutdown-get-1-immediate-no-prestop.txt)), all
|
|
`ConnectException`, against 26-32 with POST. The JDK `HttpClient` quietly retries an idempotent
|
|
request whose connection was closed under it, so a GET load test hides exactly the failures this
|
|
experiment is looking for. Your clients may or may not retry; a POST shows what the server did.
|
|
|
|
## The residue
|
|
|
|
Variant 3 still lost 0-2 requests out of ~2,930 per run, all `IOException`: a client reusing an idle
|
|
keep-alive connection at the moment Tomcat closes it. No server-side setting removes that race; it
|
|
is why non-idempotent calls between services need retries with idempotency keys, whatever the
|
|
deployment does.
|
|
|
|
## Sizing the grace period
|
|
|
|
`terminationGracePeriodSeconds` (default 30 s) covers the `preStop` hook **and** the shutdown
|
|
after it. With a 5 s sleep and Spring's 30 s phase timeout, a slow request can be SIGKILLed at 30 s
|
|
total. Keep `preStop` + `timeout-per-shutdown-phase` below the grace period, or raise it.
|