# 5. Graceful shutdown and the rolling-update race

[← 4. CPU limits and GC](04-cpu-limits-and-gc.md) · [Index](../README.md) · Next: [6. HPA on a custom metric →](06-hpa-custom-metrics.md)

When Kubernetes deletes a pod, two things start **at the same time**: the kubelet runs the
`preStop` hook and then sends SIGTERM, and the endpoints controller removes the pod from the
Service, after which every node's kube-proxy rewrites its rules. Nothing orders them. For a short
window a pod that is shutting down still receives new connections.

Spring Boot's graceful shutdown (on by default in Boot 4.1.1 - `server.shutdown=graceful`) handles
the other half: on SIGTERM, Tomcat stops accepting connections and lets in-flight requests finish,
for up to `spring.lifecycle.timeout-per-shutdown-phase` (30 s).

## The experiment

[`demo-shutdown.sh`](../scripts/demo-shutdown.sh): 20 closed-loop clients send `POST /work?ms=300`
through the Service for 45 s; at t≈8 s, `kubectl rollout restart` replaces both replicas. Each
variant three times ([`shutdown-post-summary.txt`](output/shutdown-post-summary.txt)):

```
1-immediate-no-prestop           | run 1: IOException=26  [2909]                 | run 2: ConnectException=6 IOException=26  [2909] | run 3: ConnectException=5 IOException=24  [2915]
2-graceful-no-prestop            | run 1: IOException=3  [2924]                  | run 2: ConnectException=8 IOException=11  [2932] | run 3: ConnectException=6 IOException=5  [2929]
3-graceful-prestop-sleep         | run 1: IOException=2  [2927]                  | run 2: 0 failures [2942]                      | run 3: IOException=1  [2939]                 
4-graceful-prestop-exec-sh       | run 1: ConnectException=4 IOException=3  [2921] | run 2: ConnectException=5 IOException=2  [2926] | run 3: ConnectException=7 IOException=1  [2930]
```

Reading it by failure type:

- **`IOException`** is a request cut off mid-flight, or sent on a keep-alive connection the server
  closed. `server.shutdown=immediate` produces 24-26 per restart; graceful shutdown cuts that to a
  handful.
- **`ConnectException`** is a new connection to a pod that has already stopped listening but is
  still in the Service - the race above. Graceful shutdown alone does nothing for it (0, 8, 6).
  **A 5 s `preStop: sleep` removed it in all three runs**: the pod keeps serving while the endpoint
  removal propagates, and only then gets SIGTERM.
- **Variant 4 is variant 2 with extra steps.** An `exec` hook running `sh -c "sleep 5"` fails on
  the distroless image because there is no `sh`; Kubernetes records `FailedPreStopHook`, sends
  SIGTERM immediately, and the connect errors come back (4, 5, 7).

```
      1 Warning FailedPreStopHook pod/orders-68fd794d97-sjq65 PreStopHook failed
      1 Warning FailedPreStopHook pod/orders-68fd794d97-zsznq PreStopHook failed
```

## Why the experiment uses POST

The first version used GET and reported far fewer failures - 7 for `immediate`
([`shutdown-get-1-immediate-no-prestop.txt`](output/shutdown-get-1-immediate-no-prestop.txt)), all
`ConnectException`, against 26-32 with POST. The JDK `HttpClient` quietly retries an idempotent
request whose connection was closed under it, so a GET load test hides exactly the failures this
experiment is looking for. Your clients may or may not retry; a POST shows what the server did.

## The residue

Variant 3 still lost 0-2 requests out of ~2,930 per run, all `IOException`: a client reusing an idle
keep-alive connection at the moment Tomcat closes it. No server-side setting removes that race; it
is why non-idempotent calls between services need retries with idempotency keys, whatever the
deployment does.

## Sizing the grace period

`terminationGracePeriodSeconds` (default 30 s) covers the `preStop` hook **and** the shutdown
after it. With a 5 s sleep and Spring's 30 s phase timeout, a slow request can be SIGKILLed at 30 s
total. Keep `preStop` + `timeout-per-shutdown-phase` below the grace period, or raise it.