# 4. CPU limits and the garbage collector

[← 3. JVM ergonomics](03-jvm-ergonomics.md) · [Index](../README.md) · Next: [5. Graceful shutdown →](05-graceful-shutdown.md)

A CPU limit in Kubernetes is a CFS bandwidth quota: `limits.cpu: 1` means 100 ms of CPU time per
100 ms period, *shared by every thread in the container*. When the container has used its quota,
all its threads stop until the next period. Not slow down - stop. That includes the garbage
collector in the middle of a stop-the-world pause.

## The experiment

[`demo-gc-throttling.sh`](../scripts/demo-gc-throttling.sh) runs one pod
([`k8s/gc-lab.yaml`](../k8s/gc-lab.yaml)) under five resource shapes and drives it for 60 s with 4
clients calling `/alloc?mb=16` - allocation-heavy work with a slowly churning old generation. It
reads GC pauses from `-Xlog:gc,gc+cpu`, CFS throttling from `cpu.stat` before and after, and
latency from the load generator ([`gc-throttling.txt`](output/gc-throttling.txt)):

```
   variant                                          | GC        thr | pauses pause sum      max |  throttled  thr. time | requests    p50    p99
A  500m CPU, defaults                               | Serial      0 |   2189     9256ms  188.0ms |  601/618       37.1s |     6469    15ms   200ms
B  1 CPU, defaults                                  | Serial      0 |   4279    10695ms   82.1ms |  205/615        4.6s |    15279    12ms    79ms
C  1 CPU, G1 forced with ActiveProcessorCount=2     | G1          2 |   1783    10124ms   49.9ms |  569/619        5.3s |    10463    13ms   107ms
D  1.5 CPUs, defaults                               | G1          2 |   1960     8558ms   31.9ms |  223/620        1.3s |    23517     7ms    51ms
E  2 CPUs, defaults                                 | G1          2 |   1878     8465ms   40.3ms |    0/618        0.0s |    22181     7ms    50ms
```

And how much CPU the collector actually got while its pauses were running
([`gc-cpu-ratio.txt`](output/gc-cpu-ratio.txt)):

```
    collections     user+sys         real   cpu/real
A          2189        5.32s       10.71s       0.50
B          4279       11.12s       11.12s       1.00
C          1783        9.24s       10.11s       0.91
D          1960       12.03s        8.69s       1.38
E          1878       12.58s        8.59s       1.46
```

`-Xlog:gc+cpu` reports in 10 ms steps, so a single collection's ratio means little; summed over
two thousand collections, it is a fair measure of how much of each pause the collector spent
waiting for CPU.

## What it says

- **At 500m the container spent 37 of 60 seconds throttled**, in 97 % of CFS periods. The single
  Serial GC thread got CPU for only half the wall time of its own pauses (`cpu/real` 0.50): every
  pause was stretched to double by the quota, and the longest reached 188 ms. p99 latency: 200 ms.
- **"Fixing" a 1-CPU pod by forcing G1 made it worse.** `-XX:ActiveProcessorCount=2 -XX:+UseG1GC`
  is a common recommendation for getting G1 on a small pod. On a 1-CPU quota, its two parallel GC
  threads plus concurrent marking burn the quota twice as fast: throttled in **92 %** of periods
  (against 33 % for Serial on the same quota), **32 % fewer requests** (10,463 vs 15,279), p99 107 ms
  vs 79 ms. Two GC threads received 0.91 CPU-seconds per pause-second - they were taking turns.
- **1.5 CPUs is the interesting row.** The JVM rounds up to 2 processors, so G1 gets 2 threads on 1.5
  CPUs of quota - throttled in 36 % of periods, yet the best throughput here. Throttling is not
  automatically a disaster; it is a cost that grows as the quota shrinks relative to the threads
  that want to run at once.
- **D and E are within noise of each other.** The node has exactly 2 CPUs, shared with k3s and
  the load generator, so a 2-CPU limit mostly meant "no limit". On a bigger node, E would pull ahead.

## What to do about it

1. **Know what you are running.** A 1-CPU, 1 GiB pod runs SerialGC ([chapter 3](03-jvm-ergonomics.md)).
   That can be the right collector for it - B beat C - but it should be a decision.
2. **Do not raise `ActiveProcessorCount` above the quota** to get a "better" collector. You buy
   more threads competing for the same 100 ms.
3. **Consider no CPU limit at all**, with a CPU request sized for steady state. Requests guarantee a
   share under contention and never throttle; limits throttle even on an idle node. Keep the memory
   limit - memory is not compressible.
4. **Watch throttling, not just CPU usage.** `container_cpu_cfs_throttled_periods_total` from
   cAdvisor (or `nr_throttled` in `cpu.stat`) is the signal; a pod can show 40 % average CPU while
   being throttled in most periods.

These are single 60 s runs on a small node: the directions are robust, the exact numbers are not.