zgc-jdk25-benchmarks/docs/03-allocation-stalls.md

3. Allocation stalls — the ZGC failure mode your dashboard cannot see

Runnable companion: tuning/.../AllocationStallDemo.java

3.1 The claim, and why it is misleading

Every ZGC article ends with the same chart: pause times, flat, under a millisecond, at any heap size. The chart is accurate. The implied conclusion — therefore ZGC has no latency problem — is not, because on ZGC, pauses are not where latency comes from.

ZGC's stop-the-world pauses are O(number of GC roots). They do not scale with heap size or live-set size, and they stay in the tens of microseconds. There are three of them per cycle: Pause Mark Start, Pause Mark End, Pause Relocate Start.

What actually hurts is different. When the application allocates faster than the collector can reclaim, a thread that requests memory and cannot be given a page is stalled until the collector frees one. That stall:

• can last tens or hundreds of milliseconds,
• blocks only the allocating thread, not the whole JVM,
• and therefore never appears in a pause-time metric.

A ZGC service can report a p99.99 pause of 0.2 ms and a p99.99 request latency of 400 ms simultaneously. Both numbers are correct. If your only GC SLI is pause time, you will never find the second one.

3.2 Seeing it

AllocationStallDemo starts a JFR recording, deliberately drives a small heap into the ground, then reads its own recording back. Measured on this repository's reference machine (Ryzen 5 5600U, Windows 11, JDK 25.0.3), 512 MB heap, 15 seconds of allocation:

$ java -XX:+UseZGC -Xms512m -Xmx512m -Dseconds=15 \
       -cp out com.ankurm.zgc.tuning.AllocationStallDemo

young collections : 1,319
old collections   : 164
page allocations  : 30,768
ALLOCATION STALLS : 25,343

total time threads spent stalled : 100,989 ms
mean stall                       : 3.98 ms
longest stall                    : 49.01 ms

Read that again: 101 seconds of cumulative stalled thread time inside a 15-second run. Twelve allocator threads, each spending a large fraction of the run waiting for memory. The pause chart for this run is a flat line near zero.

3.3 The threshold that hides them

jdk.ZAllocationStall has a default JFR threshold of 10 ms. Anything shorter is not recorded. In the run above the mean stall was 3.98 ms — meaning a default JFR profile would have discarded the majority of them and reported a much healthier picture.

If you take one operational thing from this repository, take this:

recording.enable("jdk.ZAllocationStall").withoutThreshold();

or in a .jfc settings file:

<event name="jdk.ZAllocationStall">
  <setting name="enabled">true</setting>
  <setting name="threshold">0 ms</setting>
  <setting name="stackTrace">true</setting>
</event>

With stack traces on, you get the exact allocation site that stalled — which is usually far more actionable than any GC log line.

The same events are visible in the unified log without JFR:

-Xlog:gc+alloc=debug

produces Allocation Stall lines. The GcLogParser in this repository counts them.

3.4 The stalls have a shape

The demo groups stalls by ZGC page class, which turns "we are out of memory" into a diagnosis:

• Small page stalls — ordinary allocation pressure. The fix is more heap, an earlier collection trigger, or allocating less.
• Medium page stalls — objects in the 256 KB–4 MB band. Often a buffer-pool or serialization layer. Frequently fixable in application code.
• Large page stalls — one page per object. ZGC does not relocate large pages, so a heap with many long-lived large objects fragments in a way ZGC cannot compact away.

A run dominated by Large-page stalls has a completely different remedy from one dominated by Small, and no pause-time chart distinguishes them.

3.5 Mitigations, in the order you should try them

SoftMaxHeapSize — start collecting earlier

-XX:SoftMaxHeapSize=400m   # with -Xmx512m

ZGC's heuristic aims to keep the heap under SoftMaxHeapSize but is allowed to exceed it, up to -Xmx, rather than stall the application. Setting it below -Xmx therefore creates a deliberate reserve: normal operation targets the soft limit, and a spike can eat into the gap instead of hitting a wall. On JDK 25 it defaults to -Xmx, i.e. no reserve at all.

It is a manageable flag, so you can change it on a running JVM — genuinely useful during an incident:

$ jcmd <pid> VM.set_flag SoftMaxHeapSize 400m

ZAllocationSpikeTolerance — assume bigger spikes

-XX:ZAllocationSpikeTolerance=5     # default 2.0

ZGC predicts when to start a cycle from the observed allocation rate multiplied by this tolerance factor. The default of 2.0 assumes allocation can double. A bursty service — one that goes from idle to full throttle in a second — routinely violates that, and the collector starts a cycle too late. Raising the tolerance makes ZGC start earlier and more often: you pay CPU to buy headroom.

-XX:ZCollectionIntervalMinor / Major — collect on a clock

Both default to -1.0 (disabled). Setting ZCollectionIntervalMinor=1 forces a young collection every second regardless of allocation rate. This is a blunt instrument, but it is the right one for a service with a very spiky duty cycle where the rate-based heuristic keeps being surprised.

More heap

The honest answer, when you can afford it. ZGC trades memory for latency by design; a ZGC heap sized like a G1 heap is a ZGC heap that stalls.

3.6 What the same pressure looks like on G1

Run the identical demo under -XX:+UseG1GC and there are no allocation stalls, because G1 does not have that mechanism. Instead you get to-space exhaustion (an evacuation failure) followed by a Full GC — a genuine, whole-application, stop-the-world compaction that will show up in your pause chart as a multi-hundred-millisecond spike.

So the failure modes are mirror images:

	ZGC	G1
Symptom under memory pressure	allocation stalls	evacuation failure → Full GC
Affects	the allocating thread(s)	every thread
Visible in pause metrics	no	yes
Typical magnitude here	4 ms mean, 49 ms max, thousands of them	hundreds of ms, few of them

G1's failure is louder and easier to find. ZGC's is quieter and easier to live with — provided you are actually measuring it.

---

Next: 04-tuning-reference.md.