asmhatreandClaude Opus 5
a065696478
Companion code for "Deploying Spring Boot 4 on Kubernetes: Probes, Graceful Shutdown, Limits and JVM Ergonomics". A dependency outage under three probe-group setups, a rolling restart under load four ways (three runs each), the JVM's ergonomic choices for nine pod shapes, one GC-heavy load under five CPU limits with throttling counters, and an HPA driven by a Micrometer gauge through prometheus-adapter. Measured on k3s v1.36.4. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01C3TETMrqVUWeFkNtz3Jbo3
74 lines
4.7 KiB
Markdown
74 lines
4.7 KiB
Markdown
# 4. CPU limits and the garbage collector
|
|
|
|
[← 3. JVM ergonomics](03-jvm-ergonomics.md) · [Index](../README.md) · Next: [5. Graceful shutdown →](05-graceful-shutdown.md)
|
|
|
|
A CPU limit in Kubernetes is a CFS bandwidth quota: `limits.cpu: 1` means 100 ms of CPU time per
|
|
100 ms period, *shared by every thread in the container*. When the container has used its quota,
|
|
all its threads stop until the next period. Not slow down - stop. That includes the garbage
|
|
collector in the middle of a stop-the-world pause.
|
|
|
|
## The experiment
|
|
|
|
[`demo-gc-throttling.sh`](../scripts/demo-gc-throttling.sh) runs one pod
|
|
([`k8s/gc-lab.yaml`](../k8s/gc-lab.yaml)) under five resource shapes and drives it for 60 s with 4
|
|
clients calling `/alloc?mb=16` - allocation-heavy work with a slowly churning old generation. It
|
|
reads GC pauses from `-Xlog:gc,gc+cpu`, CFS throttling from `cpu.stat` before and after, and
|
|
latency from the load generator ([`gc-throttling.txt`](output/gc-throttling.txt)):
|
|
|
|
```
|
|
variant | GC thr | pauses pause sum max | throttled thr. time | requests p50 p99
|
|
A 500m CPU, defaults | Serial 0 | 2189 9256ms 188.0ms | 601/618 37.1s | 6469 15ms 200ms
|
|
B 1 CPU, defaults | Serial 0 | 4279 10695ms 82.1ms | 205/615 4.6s | 15279 12ms 79ms
|
|
C 1 CPU, G1 forced with ActiveProcessorCount=2 | G1 2 | 1783 10124ms 49.9ms | 569/619 5.3s | 10463 13ms 107ms
|
|
D 1.5 CPUs, defaults | G1 2 | 1960 8558ms 31.9ms | 223/620 1.3s | 23517 7ms 51ms
|
|
E 2 CPUs, defaults | G1 2 | 1878 8465ms 40.3ms | 0/618 0.0s | 22181 7ms 50ms
|
|
```
|
|
|
|
And how much CPU the collector actually got while its pauses were running
|
|
([`gc-cpu-ratio.txt`](output/gc-cpu-ratio.txt)):
|
|
|
|
```
|
|
collections user+sys real cpu/real
|
|
A 2189 5.32s 10.71s 0.50
|
|
B 4279 11.12s 11.12s 1.00
|
|
C 1783 9.24s 10.11s 0.91
|
|
D 1960 12.03s 8.69s 1.38
|
|
E 1878 12.58s 8.59s 1.46
|
|
```
|
|
|
|
`-Xlog:gc+cpu` reports in 10 ms steps, so a single collection's ratio means little; summed over
|
|
two thousand collections, it is a fair measure of how much of each pause the collector spent
|
|
waiting for CPU.
|
|
|
|
## What it says
|
|
|
|
- **At 500m the container spent 37 of 60 seconds throttled**, in 97 % of CFS periods. The single
|
|
Serial GC thread got CPU for only half the wall time of its own pauses (`cpu/real` 0.50): every
|
|
pause was stretched to double by the quota, and the longest reached 188 ms. p99 latency: 200 ms.
|
|
- **"Fixing" a 1-CPU pod by forcing G1 made it worse.** `-XX:ActiveProcessorCount=2 -XX:+UseG1GC`
|
|
is a common recommendation for getting G1 on a small pod. On a 1-CPU quota, its two parallel GC
|
|
threads plus concurrent marking burn the quota twice as fast: throttled in **92 %** of periods
|
|
(against 33 % for Serial on the same quota), **32 % fewer requests** (10,463 vs 15,279), p99 107 ms
|
|
vs 79 ms. Two GC threads received 0.91 CPU-seconds per pause-second - they were taking turns.
|
|
- **1.5 CPUs is the interesting row.** The JVM rounds up to 2 processors, so G1 gets 2 threads on 1.5
|
|
CPUs of quota - throttled in 36 % of periods, yet the best throughput here. Throttling is not
|
|
automatically a disaster; it is a cost that grows as the quota shrinks relative to the threads
|
|
that want to run at once.
|
|
- **D and E are within noise of each other.** The node has exactly 2 CPUs, shared with k3s and
|
|
the load generator, so a 2-CPU limit mostly meant "no limit". On a bigger node, E would pull ahead.
|
|
|
|
## What to do about it
|
|
|
|
1. **Know what you are running.** A 1-CPU, 1 GiB pod runs SerialGC ([chapter 3](03-jvm-ergonomics.md)).
|
|
That can be the right collector for it - B beat C - but it should be a decision.
|
|
2. **Do not raise `ActiveProcessorCount` above the quota** to get a "better" collector. You buy
|
|
more threads competing for the same 100 ms.
|
|
3. **Consider no CPU limit at all**, with a CPU request sized for steady state. Requests guarantee a
|
|
share under contention and never throttle; limits throttle even on an idle node. Keep the memory
|
|
limit - memory is not compressible.
|
|
4. **Watch throttling, not just CPU usage.** `container_cpu_cfs_throttled_periods_total` from
|
|
cAdvisor (or `nr_throttled` in `cpu.stat`) is the signal; a pod can show 40 % average CPU while
|
|
being throttled in most periods.
|
|
|
|
These are single 60 s runs on a small node: the directions are robust, the exact numbers are not.
|