spring-boot-demo/kubernetes-deployment/docs/05-graceful-shutdown.md

5. Graceful shutdown and the rolling-update race

← 4. CPU limits and GC · Index · Next: 6. HPA on a custom metric →

When Kubernetes deletes a pod, two things start at the same time: the kubelet runs the preStop hook and then sends SIGTERM, and the endpoints controller removes the pod from the Service, after which every node's kube-proxy rewrites its rules. Nothing orders them. For a short window a pod that is shutting down still receives new connections.

Spring Boot's graceful shutdown (on by default in Boot 4.1.1 - server.shutdown=graceful) handles the other half: on SIGTERM, Tomcat stops accepting connections and lets in-flight requests finish, for up to spring.lifecycle.timeout-per-shutdown-phase (30 s).

The experiment

demo-shutdown.sh: 20 closed-loop clients send POST /work?ms=300 through the Service for 45 s; at t≈8 s, kubectl rollout restart replaces both replicas. Each variant three times (shutdown-post-summary.txt):

1-immediate-no-prestop           | run 1: IOException=26  [2909]                 | run 2: ConnectException=6 IOException=26  [2909] | run 3: ConnectException=5 IOException=24  [2915]
2-graceful-no-prestop            | run 1: IOException=3  [2924]                  | run 2: ConnectException=8 IOException=11  [2932] | run 3: ConnectException=6 IOException=5  [2929]
3-graceful-prestop-sleep         | run 1: IOException=2  [2927]                  | run 2: 0 failures [2942]                      | run 3: IOException=1  [2939]                 
4-graceful-prestop-exec-sh       | run 1: ConnectException=4 IOException=3  [2921] | run 2: ConnectException=5 IOException=2  [2926] | run 3: ConnectException=7 IOException=1  [2930]

Reading it by failure type:

• IOException is a request cut off mid-flight, or sent on a keep-alive connection the server closed. server.shutdown=immediate produces 24-26 per restart; graceful shutdown cuts that to a handful.
• ConnectException is a new connection to a pod that has already stopped listening but is still in the Service - the race above. Graceful shutdown alone does nothing for it (0, 8, 6). A 5 s preStop: sleep removed it in all three runs: the pod keeps serving while the endpoint removal propagates, and only then gets SIGTERM.
• Variant 4 is variant 2 with extra steps. An exec hook running sh -c "sleep 5" fails on the distroless image because there is no sh; Kubernetes records FailedPreStopHook, sends SIGTERM immediately, and the connect errors come back (4, 5, 7).

      1 Warning FailedPreStopHook pod/orders-68fd794d97-sjq65 PreStopHook failed
      1 Warning FailedPreStopHook pod/orders-68fd794d97-zsznq PreStopHook failed

Why the experiment uses POST

The first version used GET and reported far fewer failures - 7 for immediate (shutdown-get-1-immediate-no-prestop.txt), all ConnectException, against 26-32 with POST. The JDK HttpClient quietly retries an idempotent request whose connection was closed under it, so a GET load test hides exactly the failures this experiment is looking for. Your clients may or may not retry; a POST shows what the server did.

The residue

Variant 3 still lost 0-2 requests out of ~2,930 per run, all IOException: a client reusing an idle keep-alive connection at the moment Tomcat closes it. No server-side setting removes that race; it is why non-idempotent calls between services need retries with idempotency keys, whatever the deployment does.

Sizing the grace period

terminationGracePeriodSeconds (default 30 s) covers the preStop hook and the shutdown after it. With a 5 s sleep and Spring's 30 s phase timeout, a slow request can be SIGKILLed at 30 s total. Keep preStop + timeout-per-shutdown-phase below the grace period, or raise it.