zgc-jdk25-benchmarks/docs/06-methodology.md

6. Methodology — how to not publish a meaningless GC benchmark

Most GC comparisons on the internet are wrong in one of a small number of specific, fixable ways. This chapter lists them, because a benchmark's credibility is entirely a function of which of these it avoided.

6.1 Coordinated omission

The mistake. A closed-loop harness — while (true) { doRequest(); } — stops issuing load exactly when the system is least able to serve it. If a 100 ms pause freezes the application, the load generator does not send requests during those 100 ms. It records one slow request per thread and moves on. The pause was real; your users experienced it as 100 ms of queueing across ~2000 requests; the benchmark reports it as a handful of samples.

The fix. Fix the arrival schedule up front and measure from intended arrival, not from when the thread actually got around to the request.

long intended = start + i * interArrivalNanos;
parkUntil(intended);
long actualStart = System.nanoTime();
handleRequest();
long done = System.nanoTime();

responseTime.record(done - intended);      // honest
serviceTime.record(done - actualStart);    // what a closed-loop harness reports

How much it matters. This repository records both and prints the ratio. From the 60-second run on the reference machine:

Collector	p99.9 response time	p99.9 service time	ratio
ZGC	1.437 ms	0.143 ms	10x
G1	95.169 ms	0.033 ms	2910x

A closed-loop benchmark of this exact workload would have concluded that G1's p99.9 is 0.033 ms — better than ZGC's 0.143 ms — and published a chart showing G1 winning on tail latency. The honest numbers say G1's p99.9 is 66x worse.

Note that the distortion is not uniform: it is far larger for the collector with larger pauses. So coordinated omission does not merely add noise, it systematically favours the collector you are trying to evaluate against.

6.2 Benchmarking the sleep instead of the system

The first version of this harness used LockSupport.parkNanos(remaining) between requests. On Windows a park rounds up to roughly 1 ms, and the inter-arrival interval was 200 µs. Every park overshot by several intervals; the thread then blasted through the backlog and parked again. The harness reported a p50 of 0.590 ms for a request that takes 3 µs, and the shape of the distribution looked plausibly like a GC effect.

The fix is to park only when the deadline is far away (3 ms here, comfortably beyond the timer slack) and busy-spin the rest. It costs a core per worker thread, which is why the worker count defaults to a third of the logical CPUs — see §6.4.

If your latency harness's p50 is much larger than the work it does, suspect your sleep before you suspect the collector.

6.3 Comparing pause time against cycle time

Covered in detail in 05-logging-and-jfr.md. In one line: ZGC's GarbageCollectorMXBean "Cycles" beans and its Concurrent ... log lines are not freeze time, and summing them against G1's pause numbers produces a comparison that is off by two orders of magnitude in G1's favour.

6.4 Starving the collector of CPU

ZGC does concurrent work on its own threads. If the benchmark's load generator occupies every core — which a busy-spinning open-loop harness will happily do — those threads cannot run, the collector falls behind, and you measure a scheduling problem while believing you measured a GC.

This repository's harness therefore defaults to availableProcessors() / 3 worker threads and states so in its output. On the 12-thread reference machine that is 4 spinning workers, leaving 8 logical CPUs for the JVM's own threads.

6.5 Letting the heap resize during measurement

-Xms ≠ -Xmx means the heap grows and shrinks during the run, and you are partly measuring the resize policy. Every run in this repository uses -Xms2g -Xmx2g.

A side effect worth knowing: ZUncommit automatically becomes false when -Xms == -Xmx (verified in results/05-pages-zgc.txt). Fixing the heap therefore also removes uncommit behaviour from the comparison. That is the right call for a benchmark and the wrong call if uncommit is the thing you wanted to study.

6.6 Forgetting that JMH forks a fresh JVM

This one silently invalidates a large fraction of published GC microbenchmarks:

# WRONG -- the fork does not inherit the outer JVM's GC flags
java -XX:+UseZGC -jar benchmarks.jar

# RIGHT
java -jar benchmarks.jar -jvmArgs "-XX:+UseZGC -Xms2g -Xmx2g"

JMH starts a separate JVM for each trial. Flags on the outer java command configure the harness, not the forked JVM under measurement. Run it the wrong way twice and you get two runs of the default collector, a pleasingly small difference between them, and a blog post.

Verify with -XX:+PrintFlagsFinal inside the fork, or simply read JMH's own # VM options: banner — it prints the fork's arguments.

6.7 Measuring an empty heap

A collector with nothing to trace is fast. A benchmark that only allocates immediately-dead garbage measures the TLAB allocation path and nothing else — and every collector has essentially the same TLAB path.

The interesting variable is the live set: how much genuinely reachable data must be marked, and how often it is mutated. This repository's harness holds ~585 MB live in a 2 GB heap and rewrites a random slot on 20% of requests. PromotionBench parameterises the same idea directly through survivorDepth.

6.8 Dead-code elimination

If the JIT can prove your allocation is unused, it deletes it, and you benchmark an empty loop. Use JMH's Blackhole or return values, and — critically — check gc.alloc.rate.norm from -prof gc. It reports bytes allocated per operation. If that number is not what the source code implies, escape analysis removed something and the benchmark is measuring nothing.

6.9 One run is not a measurement

Everything in results/ is a single run on a single laptop, and is presented as an illustration of shape, not as a benchmark result you should plan capacity from. Ryzen 5 5600U, Windows 11, 15.3 GB RAM, JDK 25.0.3 — with browser tabs, an OS, and thermal limits. Server hardware, Linux, and a quiet machine will produce different absolute numbers.

What generalises is the relationship: G1 wins the median, ZGC wins the tail, and the gap widens as the live set grows. What does not generalise is any specific millisecond figure.

---

Next: 07-corner-cases.md.