Add kubernetes-deployment: probes, shutdown, JVM ergonomics, HPA

Companion code for "Deploying Spring Boot 4 on Kubernetes: Probes, Graceful
Shutdown, Limits and JVM Ergonomics". A dependency outage under three
probe-group setups, a rolling restart under load four ways (three runs
each), the JVM's ergonomic choices for nine pod shapes, one GC-heavy load
under five CPU limits with throttling counters, and an HPA driven by a
Micrometer gauge through prometheus-adapter. Measured on k3s v1.36.4.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01C3TETMrqVUWeFkNtz3Jbo3

kubernetes-deployment/docs/05-graceful-shutdown.md

@@ -0,0 +1,64 @@
+# 5. Graceful shutdown and the rolling-update race
+
+[← 4. CPU limits and GC](04-cpu-limits-and-gc.md) · [Index](../README.md) · Next: [6. HPA on a custom metric →](06-hpa-custom-metrics.md)
+
+When Kubernetes deletes a pod, two things start **at the same time**: the kubelet runs the
+`preStop` hook and then sends SIGTERM, and the endpoints controller removes the pod from the
+Service, after which every node's kube-proxy rewrites its rules. Nothing orders them. For a short
+window a pod that is shutting down still receives new connections.
+
+Spring Boot's graceful shutdown (on by default in Boot 4.1.1 - `server.shutdown=graceful`) handles
+the other half: on SIGTERM, Tomcat stops accepting connections and lets in-flight requests finish,
+for up to `spring.lifecycle.timeout-per-shutdown-phase` (30 s).
+
+## The experiment
+
+[`demo-shutdown.sh`](../scripts/demo-shutdown.sh): 20 closed-loop clients send `POST /work?ms=300`
+through the Service for 45 s; at t≈8 s, `kubectl rollout restart` replaces both replicas. Each
+variant three times ([`shutdown-post-summary.txt`](output/shutdown-post-summary.txt)):
+
+```
+1-immediate-no-prestop           | run 1: IOException=26  [2909]                 | run 2: ConnectException=6 IOException=26  [2909] | run 3: ConnectException=5 IOException=24  [2915]
+2-graceful-no-prestop            | run 1: IOException=3  [2924]                  | run 2: ConnectException=8 IOException=11  [2932] | run 3: ConnectException=6 IOException=5  [2929]
+3-graceful-prestop-sleep         | run 1: IOException=2  [2927]                  | run 2: 0 failures [2942]                      | run 3: IOException=1  [2939]                 
+4-graceful-prestop-exec-sh       | run 1: ConnectException=4 IOException=3  [2921] | run 2: ConnectException=5 IOException=2  [2926] | run 3: ConnectException=7 IOException=1  [2930]
+```
+
+Reading it by failure type:
+
+- **`IOException`** is a request cut off mid-flight, or sent on a keep-alive connection the server
+  closed. `server.shutdown=immediate` produces 24-26 per restart; graceful shutdown cuts that to a
+  handful.
+- **`ConnectException`** is a new connection to a pod that has already stopped listening but is
+  still in the Service - the race above. Graceful shutdown alone does nothing for it (0, 8, 6).
+  **A 5 s `preStop: sleep` removed it in all three runs**: the pod keeps serving while the endpoint
+  removal propagates, and only then gets SIGTERM.
+- **Variant 4 is variant 2 with extra steps.** An `exec` hook running `sh -c "sleep 5"` fails on
+  the distroless image because there is no `sh`; Kubernetes records `FailedPreStopHook`, sends
+  SIGTERM immediately, and the connect errors come back (4, 5, 7).
+
+```
+      1 Warning FailedPreStopHook pod/orders-68fd794d97-sjq65 PreStopHook failed
+      1 Warning FailedPreStopHook pod/orders-68fd794d97-zsznq PreStopHook failed
+```
+
+## Why the experiment uses POST
+
+The first version used GET and reported far fewer failures - 7 for `immediate`
+([`shutdown-get-1-immediate-no-prestop.txt`](output/shutdown-get-1-immediate-no-prestop.txt)), all
+`ConnectException`, against 26-32 with POST. The JDK `HttpClient` quietly retries an idempotent
+request whose connection was closed under it, so a GET load test hides exactly the failures this
+experiment is looking for. Your clients may or may not retry; a POST shows what the server did.
+
+## The residue
+
+Variant 3 still lost 0-2 requests out of ~2,930 per run, all `IOException`: a client reusing an idle
+keep-alive connection at the moment Tomcat closes it. No server-side setting removes that race; it
+is why non-idempotent calls between services need retries with idempotency keys, whatever the
+deployment does.
+
+## Sizing the grace period
+
+`terminationGracePeriodSeconds` (default 30 s) covers the `preStop` hook **and** the shutdown
+after it. With a 5 s sleep and Spring's 30 s phase timeout, a slow request can be SIGKILLed at 30 s
+total. Keep `preStop` + `timeout-per-shutdown-phase` below the grace period, or raise it.