# 3. Allocation stalls — the ZGC failure mode your dashboard cannot see
> Runnable companion: [`tuning/.../AllocationStallDemo.java`](../tuning/src/com/ankurm/zgc/tuning/AllocationStallDemo.java)
## 3.1 The claim, and why it is misleading
Every ZGC article ends with the same chart: pause times, flat, under a millisecond, at any heap
size. The chart is accurate. The implied conclusion — *therefore ZGC has no latency problem* — is
not, because **on ZGC, pauses are not where latency comes from.**
ZGC's stop-the-world pauses are O(number of GC roots). They do not scale with heap size or live-set
size, and they stay in the tens of microseconds. There are three of them per cycle: `Pause Mark
Start`, `Pause Mark End`, `Pause Relocate Start`.
What actually hurts is different. When the application allocates faster than the collector can
reclaim, a thread that requests memory and cannot be given a page is **stalled** until the collector
frees one. That stall:
- can last **tens or hundreds of milliseconds**,
- blocks only the allocating thread, not the whole JVM,
- and therefore **never appears in a pause-time metric.**
A ZGC service can report a p99.99 pause of 0.2 ms and a p99.99 request latency of 400 ms
simultaneously. Both numbers are correct. If your only GC SLI is pause time, you will never find the
second one.
## 3.2 Seeing it
`AllocationStallDemo` starts a JFR recording, deliberately drives a small heap into the ground, then
reads its own recording back. Measured on this repository's reference machine (Ryzen 5 5600U,
Windows 11, JDK 25.0.3), 512 MB heap, 15 seconds of allocation:
```console
$ java -XX:+UseZGC -Xms512m -Xmx512m -Dseconds=15 \
-cp out com.ankurm.zgc.tuning.AllocationStallDemo
young collections : 1,319
old collections : 164
page allocations : 30,768
ALLOCATION STALLS : 25,343
total time threads spent stalled : 100,989 ms
mean stall : 3.98 ms
longest stall : 49.01 ms
```
Read that again: **101 seconds of cumulative stalled thread time inside a 15-second run.** Twelve
allocator threads, each spending a large fraction of the run waiting for memory. The pause chart for
this run is a flat line near zero.
## 3.3 The threshold that hides them
`jdk.ZAllocationStall` has a **default JFR threshold of 10 ms**. Anything shorter is not recorded.
In the run above the *mean* stall was 3.98 ms — meaning a default JFR profile would have discarded
the majority of them and reported a much healthier picture.
If you take one operational thing from this repository, take this:
```java
recording.enable("jdk.ZAllocationStall").withoutThreshold();
```
or in a `.jfc` settings file:
```xml
true
0 ms
true
```
With stack traces on, you get the exact allocation site that stalled — which is usually far more
actionable than any GC log line.
The same events are visible in the unified log without JFR:
```console
-Xlog:gc+alloc=debug
```
produces `Allocation Stall` lines. The `GcLogParser` in this repository counts them.
## 3.4 The stalls have a *shape*
The demo groups stalls by ZGC page class, which turns "we are out of memory" into a diagnosis:
- **Small** page stalls — ordinary allocation pressure. The fix is more heap, an earlier collection
trigger, or allocating less.
- **Medium** page stalls — objects in the 256 KB–4 MB band. Often a buffer-pool or serialization
layer. Frequently fixable in application code.
- **Large** page stalls — one page per object. ZGC does not relocate large pages, so a heap with
many long-lived large objects fragments in a way ZGC cannot compact away.
A run dominated by Large-page stalls has a completely different remedy from one dominated by Small,
and no pause-time chart distinguishes them.
## 3.5 Mitigations, in the order you should try them
### `SoftMaxHeapSize` — start collecting earlier
```
-XX:SoftMaxHeapSize=400m # with -Xmx512m
```
ZGC's heuristic aims to keep the heap under `SoftMaxHeapSize` but is *allowed* to exceed it, up to
`-Xmx`, rather than stall the application. Setting it below `-Xmx` therefore creates a deliberate
reserve: normal operation targets the soft limit, and a spike can eat into the gap instead of
hitting a wall. On JDK 25 it defaults to `-Xmx`, i.e. no reserve at all.
It is a `manageable` flag, so you can change it on a running JVM — genuinely useful during an
incident:
```console
$ jcmd VM.set_flag SoftMaxHeapSize 400m
```
### `ZAllocationSpikeTolerance` — assume bigger spikes
```
-XX:ZAllocationSpikeTolerance=5 # default 2.0
```
ZGC predicts when to start a cycle from the observed allocation rate multiplied by this tolerance
factor. The default of 2.0 assumes allocation can double. A bursty service — one that goes from idle
to full throttle in a second — routinely violates that, and the collector starts a cycle too late.
Raising the tolerance makes ZGC start earlier and more often: you pay CPU to buy headroom.
### `-XX:ZCollectionIntervalMinor` / `Major` — collect on a clock
Both default to `-1.0` (disabled). Setting `ZCollectionIntervalMinor=1` forces a young collection
every second regardless of allocation rate. This is a blunt instrument, but it is the right one for
a service with a very spiky duty cycle where the rate-based heuristic keeps being surprised.
### More heap
The honest answer, when you can afford it. ZGC trades memory for latency by design; a ZGC heap sized
like a G1 heap is a ZGC heap that stalls.
## 3.6 What the same pressure looks like on G1
Run the identical demo under `-XX:+UseG1GC` and there are no allocation stalls, because G1 does not
have that mechanism. Instead you get **to-space exhaustion** (an evacuation failure) followed by a
**Full GC** — a genuine, whole-application, stop-the-world compaction that will show up in your
pause chart as a multi-hundred-millisecond spike.
So the failure modes are mirror images:
| | ZGC | G1 |
|---|---|---|
| Symptom under memory pressure | allocation stalls | evacuation failure → Full GC |
| Affects | the allocating thread(s) | every thread |
| Visible in pause metrics | **no** | yes |
| Typical magnitude here | 4 ms mean, 49 ms max, thousands of them | hundreds of ms, few of them |
G1's failure is louder and easier to find. ZGC's is quieter and easier to live with — provided you
are actually measuring it.
---
Next: [04-tuning-reference.md](04-tuning-reference.md).