Deploying Spring Boot 4 on Kubernetes: Probes, Graceful Shutdown, Limits and JVM Ergonomics
Probes, graceful shutdown, resource limits and JVM ergonomics for Spring Boot 4, measured on a real cluster: a dependency outage under three probe setups, a rolling restart under load four ways, what the JVM decides for nine pod shapes, why CPU limits throttle your garbage collector, and an HPA scaling on a custom Micrometer metric.
Kubernetes knows three things about your Spring Boot service: whether it is alive, whether it is ready, and how much CPU and memory it may use. Each is one small block of YAML, and each one quietly decides something bigger — whether a database blip restarts every replica at once, whether a deploy drops requests, which garbage collector the JVM picks and how often the kernel freezes it mid-pause.
This article measures those decisions on a real cluster rather than describing them: a dependency outage under three probe configurations, a rolling restart under load four ways, the JVM’s ergonomic choices for nine pod shapes, the same garbage-heavy workload under five CPU limits, and a HorizontalPodAutoscaler driven by a Micrometer gauge instead of CPU.
Part
For you if
Covers
1 — Beginner
you are writing your first Deployment for a Boot app
liveness vs readiness vs startup, Boot’s probe groups, what belongs in which probe, measured
2 — Intermediate
your deploys drop a few requests
the termination race, graceful shutdown, preStop, why distroless breaks the usual hook
3 — Advanced
you set CPU limits and scale on CPU
JVM ergonomics in containers, CPU limits throttling the collector, HPA on a custom Micrometer metric
Versions this was verified against. Spring Boot 4.1.1 (GA, published to Maven Central on 20 August 2026), Spring Framework 7.0.9, Temurin JDK 25.0.4.1 in the gcr.io/distroless/java25-debian13:nonroot image from the Docker article, Kubernetes 1.36.4 (k3s) on a single 2-vCPU node, Prometheus 3.14.0, prometheus-adapter 0.12.0. The node runs cgroup v1, which Kubernetes 1.35+ only accepts with failCgroupV1=false; the CPU quota mechanics measured here are the same on cgroup v2, and the companion docs list the file names that differ. Small node on purpose — CPU contention is easier to see on two cores than on sixty-four — so treat the exact numbers as indicative and the directions as the result.
Companion code: spring-boot-demo, directory kubernetes-deployment/. The service, the manifests, one script per experiment, and every transcript below under docs/output/.
Part 1 — Probes: which check belongs where
What Spring Boot gives you
Spring Boot exposes two health groups for Kubernetes: /actuator/health/liveness and /actuator/health/readiness. Boot adds them automatically when it detects Kubernetes (the *_SERVICE_HOST and *_SERVICE_PORT variables every pod gets) and you can force them with management.endpoint.health.probes.enabled=true. By default each group holds exactly one thing, the application’s own LivenessState or ReadinessState — not the database, not disk space, not your custom indicators.
The three probes ask different questions, and the kubelet does very different things when they fail:
Probe
Question
On failure
startup
has it finished starting?
kill after failureThreshold × periodSeconds; the other two wait for it
liveness
is this process broken beyond repair?
kill and restart the container
readiness
should it receive traffic right now?
remove it from the Service; no restart
The baseline Deployment every experiment starts from:
Every line of that is justified by a measurement below — including the missing CPU limit.
One dependency outage, three configurations
The most common probe mistake is putting a shared dependency — the database, a downstream API — into a probe, because “if the database is down, the app is not healthy”. The companion project has a health indicator for a downstream service and takes that service away for 60 seconds, watching the two replicas and whether GET /work — an endpoint that does not use the downstream at all — still answers through the Service.
With the indicator in the liveness group:
t= 17s 42sgx ready=true restarts=0 running | x8bjd ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
t= 23s 42sgx ready=false restarts=1 running | x8bjd ready=false restarts=1 running | serving endpoints=0 GET /work via Service: FAIL
-------- downstream restored --------
t= 69s 42sgx ready=true restarts=1 running | x8bjd ready=true restarts=1 running | serving endpoints=2 GET /work via Service: ok
Both replicas were killed and restarted together, and the new containers could not pass their startup probe — it checks the same group — until the downstream came back. A dependency outage became a total outage of a service that did not need the dependency, and the JVMs threw away their warm caches and JIT state on the way.
With the indicator in the readiness group:
t= 6s 8mf9m ready=false restarts=0 running | btcl4 ready=false restarts=0 running | serving endpoints=0 GET /work via Service: FAIL
-------- downstream restored --------
t= 69s 8mf9m ready=true restarts=0 running | btcl4 ready=true restarts=0 running | serving endpoints=2 GET /work via Service: ok
No restarts, fast recovery — and still zero serving endpoints within six seconds. Every replica checks the same dependency, so every replica leaves the Service at the same moment, and clients get connection refused instead of an error they could act on.
With the indicator in neither — Spring Boot’s default — both pods stay in service throughout. Requests that need the downstream fail with whatever your code returns, which can be a precise 503; everything else keeps working.
A shared dependency in a probe turns its outage into yours. Liveness should contain only what a restart would fix — Boot’s default is usually exactly right. Readiness should contain only what is specific to this instance: a local cache still warming, a full local queue. Shared dependencies belong in the request path — a timeout, a circuit breaker, a fast 503 — and in /actuator/health for dashboards, just not in a probe group.
Slow starts need a startup probe, not a longer liveness delay
The same service, made to take about 45 seconds to start, rolled out to one replica without a startup probe:
Liveness starts probing the moment the container starts, Tomcat is not listening yet, and three refused connections later the new pod is killed — again and again, never getting to finish. The old pod 7ltj7 keeps serving only because maxUnavailable: 0 makes the rollout stall rather than proceed; with more replicas and a looser strategy, every attempt takes capacity away. With the startup probe from the baseline, the same rollout is ready at 51 seconds with no restarts. The probe’s budget, failureThreshold × periodSeconds (120 s here), is sized for the slowest start you expect; liveness stays tight because it only begins once startup has passed.
Part 2 — Shutting down without dropping requests
Two things start at once
When Kubernetes deletes a pod — a rolling update, a scale-down, a node drain — it does two things in parallel. The kubelet runs the container’s preStop hook and then sends SIGTERM. Meanwhile the endpoints controller removes the pod from the Service, and every node’s kube-proxy has to notice and rewrite its rules. Nothing orders them. So for a short window, a pod that has received SIGTERM is still being sent new connections.
Spring Boot 4.1.1 ships with graceful shutdown on (server.shutdown defaults to graceful in its configuration metadata): on SIGTERM, Tomcat stops accepting connections and lets in-flight requests finish, for up to spring.lifecycle.timeout-per-shutdown-phase, 30 s by default. That covers the requests already inside the pod. The preStop hook covers the ones still on their way:
lifecycle:
preStop:
sleep:
seconds: 5
The native sleep action is on by default since Kubernetes 1.30 (beta) and stable since 1.34, per KEP-3960; the Spring Boot reference suggests it from 1.32. Before it, everyone wrote exec: ["sh", "-c", "sleep 5"] — hold that thought.
Measured: a rolling restart under load
Twenty clients send POST /work?ms=300 through the Service for 45 seconds; eight seconds in, kubectl rollout restart replaces both replicas. Four configurations, three runs each, failed requests per run with successful ones in brackets:
1-immediate-no-prestop | run 1: IOException=26 [2909] | run 2: ConnectException=6 IOException=26 [2909] | run 3: ConnectException=5 IOException=24 [2915]
2-graceful-no-prestop | run 1: IOException=3 [2924] | run 2: ConnectException=8 IOException=11 [2932] | run 3: ConnectException=6 IOException=5 [2929]
3-graceful-prestop-sleep | run 1: IOException=2 [2927] | run 2: 0 failures [2942] | run 3: IOException=1 [2939]
4-graceful-prestop-exec-sh | run 1: ConnectException=4 IOException=3 [2921] | run 2: ConnectException=5 IOException=2 [2926] | run 3: ConnectException=7 IOException=1 [2930]
The two exception types are the two halves of the problem. IOException is a request cut off in flight, or sent on a keep-alive connection the server just closed: 24–26 per restart with server.shutdown=immediate, a handful with graceful shutdown. ConnectException is a new connection to a pod that has stopped listening but is still in the Service — the race in the diagram. Graceful shutdown alone does nothing for it. The five-second sleep removed it in all three runs.
The first version of this test used GET, and it lied. Seven failures for server.shutdown=immediate, all ConnectException — against 26 to 32 with POST. The JDK HttpClient quietly retries an idempotent request whose connection was closed under it (jdk.httpclient.enableAllMethodRetry is the switch that extends this to other methods), so a GET load test hides exactly the dropped requests you are trying to count. Your clients may retry and may not. Measure with a method nobody retries.
The preStop hook that is not there
Row 4 is the exec hook everyone wrote before the native action existed — on the distroless image from the Docker article, which has no shell:
The hook fails instantly, SIGTERM follows immediately, and the connect errors are back — 4, 5 and 7, the same as having no hook at all. It is a warning event on a pod that is about to disappear, which nobody reads. Moving an existing Deployment to a distroless image silently turns its preStop into a no-op. Use the native sleep action.
What remains in row 3 — zero to two failures out of about 2,930 — is a client reusing an idle keep-alive connection at the instant Tomcat closes it. No server setting removes that; it is why non-idempotent calls between services need retries with idempotency keys regardless of how carefully you deploy.
Size the grace period for the sum.terminationGracePeriodSeconds, 30 s by default, has to cover the preStop sleep and the drain after it. A 5 s sleep plus Spring’s 30 s shutdown phase means a slow request can be SIGKILLed at the 30 s mark. Either keep sleep + timeout-per-shutdown-phase under the grace period, or raise the grace period.
Part 3 — CPU limits, JVM ergonomics, and scaling on the right signal
What the JVM decides from your pod spec
With no JVM options, the JVM reads the container’s cgroup limits and picks a processor count, a garbage collector, a heap and thread counts. java -XX:+PrintFlagsFinal -version in a pod per resource shape:
CPUs are the limit, rounded up — 500m is 1, 1500m is 2. With no limit, the JVM sees every core on the node.
Requests are ignored. Every row has a 100m request. JDK 19 stopped using CPU shares for the processor count (JDK-8281181); requests.cpu affects scheduling and nothing the JVM decides.
G1 needs two CPUs and about 1792 MB. Below either, the JVM does not consider itself a server-class machine and picks SerialGC. Two CPUs with 1700Mi: Serial. With 1800Mi: G1. A typical 512Mi–1Gi Spring Boot pod therefore runs SerialGC without anyone having chosen it.
The heap is 25 % of the memory limit. A 512Mi pod gets 128 MB of heap and leaves 384 MB for everything else. -XX:MaxRAMPercentage=75 is the usual correction.
Why CPU limits throttle your GC
A CPU limit is a CFS bandwidth quota: limits.cpu: 1 means 100 ms of CPU per 100 ms period, shared by every thread in the container. When the quota is spent, every thread stops until the next period — including the garbage collector in the middle of a stop-the-world pause. One pod, five resource shapes, the same allocation-heavy load from four clients for 60 seconds:
variant | GC thr | pauses pause sum max | throttled thr. time | requests p50 p99
A 500m CPU, defaults | Serial 0 | 2189 9256ms 188.0ms | 601/618 37.1s | 6469 15ms 200ms
B 1 CPU, defaults | Serial 0 | 4279 10695ms 82.1ms | 205/615 4.6s | 15279 12ms 79ms
C 1 CPU, G1 forced with ActiveProcessorCount=2 | G1 2 | 1783 10124ms 49.9ms | 569/619 5.3s | 10463 13ms 107ms
D 1.5 CPUs, defaults | G1 2 | 1960 8558ms 31.9ms | 223/620 1.3s | 23517 7ms 51ms
E 2 CPUs, defaults | G1 2 | 1878 8465ms 40.3ms | 0/618 0.0s | 22181 7ms 50ms
And how much CPU the collector actually received during its own pauses, summed from -Xlog:gc+cpu over every collection:
collections user+sys real cpu/real
A 2189 5.32s 10.71s 0.50
B 4279 11.12s 11.12s 1.00
C 1783 9.24s 10.11s 0.91
D 1960 12.03s 8.69s 1.38
E 1878 12.58s 8.59s 1.46
At 500m the container was throttled for 37 of 60 seconds, in 97 % of CFS periods, and the single Serial GC thread got CPU for only half the wall time of its own pauses: every pause stretched to double length by the quota, the longest to 188 ms, p99 latency 200 ms. That is the fingerprint: a cpu/real ratio well below the number of GC threads.
“Force G1 on small pods” made the 1-CPU pod worse.-XX:ActiveProcessorCount=2 -XX:+UseG1GC is common advice for getting a “proper” collector into a small container. On a 1-CPU quota it gives two parallel GC threads plus concurrent marking the same 100 ms per period: throttled in 92 % of periods against 33 % for Serial, 32 % fewer requests (10,463 against 15,279), p99 107 ms against 79 ms. The two GC threads received 0.91 CPU-seconds per second of pause — they were taking turns. Never tell the JVM it has more processors than its quota.
Row D is the subtle one. 1500m rounds up to two processors, so G1 runs two threads on one and a half CPUs of quota — throttled in 36 % of periods, and still the best throughput in the table. Throttling is not automatically a disaster; it is a cost that rises as the quota shrinks relative to the number of threads that want to run at once. D and E are within noise of each other because the node has exactly two CPUs, shared with the load generator, so a two-CPU limit meant little here.
The practical conclusions: know which collector your pod shape gives you, never raise ActiveProcessorCount above the quota, watch container_cpu_cfs_throttled_periods_total rather than average CPU, and seriously consider no CPU limit — a CPU request sized for steady state guarantees a share under contention and never throttles, while a limit throttles even on an idle node. Keep the memory limit; memory is not compressible.
Scaling on work in progress, not CPU
The previous section is also the argument against the HPA’s favourite metric. CPU is a poor scaling signal for a JVM: startup and JIT compilation burn it while serving nothing, the collector competes with requests for the same quota, and a service waiting on a database can be saturated at 20 % CPU. What you want to scale on is work in progress — here, requests in flight.
The one piece of configuration that matters is the adapter rule. resources.overrides is what maps the Prometheus pod label onto a Kubernetes Pod — without it the series is in Prometheus and never appears in the API:
Twelve seconds from load to the first scale-up: a 5 s scrape, the adapter’s query, the HPA’s 15 s sync. At t=43 the per-pod average was 10.3 on two pods against a target of 5, so the HPA asked for five and was capped at four — 30 clients on four pods settles around 7.5, still above target, which is what a ceiling looks like. Scale-down took 40–60 s after the load stopped, because the rule averages over 30 s and the scale-down stabilization window holds the highest recent recommendation. That window is shortened to 30 s for the demo; the default is 300 s, and in production you want it. The load generator saw 7,209 successful requests and no failures through two scale-downs — the preStop sleep from Part 2 doing its job.
The long tail
How the lab was built, including three things that broke on the way — a runc oom_score_adj failure, a proxy leaking into the API server, and the kubelet garbage-collecting images it was about to need: chapter 1
The cgroup v1 and v2 file names for CPU quota and throttling, and -XshowSettings:system: chapter 3
The keep-alive race that no server setting removes, and sizing terminationGracePeriodSeconds: chapter 5
Counters in the adapter rule, and when KEDA is the better choice: chapter 6
What to change on Monday. Take shared dependencies out of your probe groups. Add a startupProbe and a native preStop: sleep — and check that the hook you already have does not need a shell your image no longer has. Set -XX:MaxRAMPercentage. Then look at your CPU limits with the throttling metric in front of you, and decide per service whether you want a limit at all.
Do not copy the numbers in this article into your manifests. Copy the experiments: every script here runs against any cluster, and your node size, traffic and heap are the ones that matter.
Further reading
Companion project — the service, manifests, experiments and every transcript
No Comments yet!