spring-boot-demo/kubernetes-deployment/docs/04-cpu-limits-and-gc.md

4. CPU limits and the garbage collector

← 3. JVM ergonomics · Index · Next: 5. Graceful shutdown →

A CPU limit in Kubernetes is a CFS bandwidth quota: limits.cpu: 1 means 100 ms of CPU time per 100 ms period, shared by every thread in the container. When the container has used its quota, all its threads stop until the next period. Not slow down - stop. That includes the garbage collector in the middle of a stop-the-world pause.

The experiment

demo-gc-throttling.sh runs one pod (k8s/gc-lab.yaml) under five resource shapes and drives it for 60 s with 4 clients calling /alloc?mb=16 - allocation-heavy work with a slowly churning old generation. It reads GC pauses from -Xlog:gc,gc+cpu, CFS throttling from cpu.stat before and after, and latency from the load generator (gc-throttling.txt):

   variant                                          | GC        thr | pauses pause sum      max |  throttled  thr. time | requests    p50    p99
A  500m CPU, defaults                               | Serial      0 |   2189     9256ms  188.0ms |  601/618       37.1s |     6469    15ms   200ms
B  1 CPU, defaults                                  | Serial      0 |   4279    10695ms   82.1ms |  205/615        4.6s |    15279    12ms    79ms
C  1 CPU, G1 forced with ActiveProcessorCount=2     | G1          2 |   1783    10124ms   49.9ms |  569/619        5.3s |    10463    13ms   107ms
D  1.5 CPUs, defaults                               | G1          2 |   1960     8558ms   31.9ms |  223/620        1.3s |    23517     7ms    51ms
E  2 CPUs, defaults                                 | G1          2 |   1878     8465ms   40.3ms |    0/618        0.0s |    22181     7ms    50ms

And how much CPU the collector actually got while its pauses were running (gc-cpu-ratio.txt):

    collections     user+sys         real   cpu/real
A          2189        5.32s       10.71s       0.50
B          4279       11.12s       11.12s       1.00
C          1783        9.24s       10.11s       0.91
D          1960       12.03s        8.69s       1.38
E          1878       12.58s        8.59s       1.46

-Xlog:gc+cpu reports in 10 ms steps, so a single collection's ratio means little; summed over two thousand collections, it is a fair measure of how much of each pause the collector spent waiting for CPU.

What it says

• At 500m the container spent 37 of 60 seconds throttled, in 97 % of CFS periods. The single Serial GC thread got CPU for only half the wall time of its own pauses (cpu/real 0.50): every pause was stretched to double by the quota, and the longest reached 188 ms. p99 latency: 200 ms.
• "Fixing" a 1-CPU pod by forcing G1 made it worse. -XX:ActiveProcessorCount=2 -XX:+UseG1GC is a common recommendation for getting G1 on a small pod. On a 1-CPU quota, its two parallel GC threads plus concurrent marking burn the quota twice as fast: throttled in 92 % of periods (against 33 % for Serial on the same quota), 32 % fewer requests (10,463 vs 15,279), p99 107 ms vs 79 ms. Two GC threads received 0.91 CPU-seconds per pause-second - they were taking turns.
• 1.5 CPUs is the interesting row. The JVM rounds up to 2 processors, so G1 gets 2 threads on 1.5 CPUs of quota - throttled in 36 % of periods, yet the best throughput here. Throttling is not automatically a disaster; it is a cost that grows as the quota shrinks relative to the threads that want to run at once.
• D and E are within noise of each other. The node has exactly 2 CPUs, shared with k3s and the load generator, so a 2-CPU limit mostly meant "no limit". On a bigger node, E would pull ahead.

What to do about it

1. Know what you are running. A 1-CPU, 1 GiB pod runs SerialGC (chapter 3). That can be the right collector for it - B beat C - but it should be a decision.
2. Do not raise ActiveProcessorCount above the quota to get a "better" collector. You buy more threads competing for the same 100 ms.
3. Consider no CPU limit at all, with a CPU request sized for steady state. Requests guarantee a share under contention and never throttle; limits throttle even on an idle node. Keep the memory limit - memory is not compressible.
4. Watch throttling, not just CPU usage. container_cpu_cfs_throttled_periods_total from cAdvisor (or nr_throttled in cpu.stat) is the signal; a pod can show 40 % average CPU while being throttled in most periods.

These are single 60 s runs on a small node: the directions are robust, the exact numbers are not.