# 6. Methodology — how to not publish a meaningless GC benchmark

Most GC comparisons on the internet are wrong in one of a small number of specific, fixable ways.
This chapter lists them, because a benchmark's credibility is entirely a function of which of these
it avoided.

## 6.1 Coordinated omission

**The mistake.** A closed-loop harness — `while (true) { doRequest(); }` — stops issuing load exactly
when the system is least able to serve it. If a 100 ms pause freezes the application, the load
generator does not send requests during those 100 ms. It records one slow request per thread and
moves on. The pause was real; your users experienced it as 100 ms of queueing across ~2000 requests;
the benchmark reports it as a handful of samples.

**The fix.** Fix the arrival schedule up front and measure from *intended* arrival, not from when
the thread actually got around to the request.

```java
long intended = start + i * interArrivalNanos;
parkUntil(intended);
long actualStart = System.nanoTime();
handleRequest();
long done = System.nanoTime();

responseTime.record(done - intended);      // honest
serviceTime.record(done - actualStart);    // what a closed-loop harness reports
```

**How much it matters.** This repository records both and prints the ratio. From the 60-second run
on the reference machine:

| Collector | p99.9 response time | p99.9 service time | ratio |
|---|---|---|---|
| ZGC | 1.437 ms | 0.143 ms | 10x |
| G1 | 95.169 ms | 0.033 ms | **2910x** |

A closed-loop benchmark of this exact workload would have concluded that G1's p99.9 is 0.033 ms —
*better* than ZGC's 0.143 ms — and published a chart showing G1 winning on tail latency. The honest
numbers say G1's p99.9 is 66x worse.

Note that the distortion is not uniform: it is far larger for the collector with larger pauses. So
coordinated omission does not merely add noise, it **systematically favours the collector you are
trying to evaluate against**.

## 6.2 Benchmarking the sleep instead of the system

The first version of this harness used `LockSupport.parkNanos(remaining)` between requests. On
Windows a park rounds up to roughly 1 ms, and the inter-arrival interval was 200 µs. Every park
overshot by several intervals; the thread then blasted through the backlog and parked again. The
harness reported a p50 of 0.590 ms for a request that takes 3 µs, and the shape of the distribution
looked plausibly like a GC effect.

The fix is to park only when the deadline is far away (3 ms here, comfortably beyond the timer
slack) and busy-spin the rest. It costs a core per worker thread, which is why the worker count
defaults to a *third* of the logical CPUs — see §6.4.

If your latency harness's p50 is much larger than the work it does, suspect your sleep before you
suspect the collector.

## 6.3 Comparing pause time against cycle time

Covered in detail in [05-logging-and-jfr.md](05-logging-and-jfr.md). In one line: ZGC's
`GarbageCollectorMXBean` "Cycles" beans and its `Concurrent ...` log lines are **not** freeze time,
and summing them against G1's pause numbers produces a comparison that is off by two orders of
magnitude in G1's favour.

## 6.4 Starving the collector of CPU

ZGC does concurrent work on its own threads. If the benchmark's load generator occupies every core —
which a busy-spinning open-loop harness will happily do — those threads cannot run, the collector
falls behind, and you measure a scheduling problem while believing you measured a GC.

This repository's harness therefore defaults to `availableProcessors() / 3` worker threads and states
so in its output. On the 12-thread reference machine that is 4 spinning workers, leaving 8 logical
CPUs for the JVM's own threads.

## 6.5 Letting the heap resize during measurement

`-Xms` ≠ `-Xmx` means the heap grows and shrinks during the run, and you are partly measuring the
resize policy. Every run in this repository uses `-Xms2g -Xmx2g`.

A side effect worth knowing: **`ZUncommit` automatically becomes `false` when `-Xms == -Xmx`**
(verified in [`results/05-pages-zgc.txt`](../results/05-pages-zgc.txt)). Fixing the heap therefore
also removes uncommit behaviour from the comparison. That is the right call for a benchmark and the
wrong call if uncommit is the thing you wanted to study.

## 6.6 Forgetting that JMH forks a fresh JVM

This one silently invalidates a large fraction of published GC microbenchmarks:

```console
# WRONG -- the fork does not inherit the outer JVM's GC flags
java -XX:+UseZGC -jar benchmarks.jar

# RIGHT
java -jar benchmarks.jar -jvmArgs "-XX:+UseZGC -Xms2g -Xmx2g"
```

JMH starts a separate JVM for each trial. Flags on the outer `java` command configure the *harness*,
not the forked JVM under measurement. Run it the wrong way twice and you get two runs of the default
collector, a pleasingly small difference between them, and a blog post.

Verify with `-XX:+PrintFlagsFinal` inside the fork, or simply read JMH's own `# VM options:` banner —
it prints the fork's arguments.

## 6.7 Measuring an empty heap

A collector with nothing to trace is fast. A benchmark that only allocates immediately-dead garbage
measures the TLAB allocation path and nothing else — and every collector has essentially the same
TLAB path.

The interesting variable is the **live set**: how much genuinely reachable data must be marked, and
how often it is mutated. This repository's harness holds ~585 MB live in a 2 GB heap and rewrites a
random slot on 20% of requests. `PromotionBench` parameterises the same idea directly through
`survivorDepth`.

## 6.8 Dead-code elimination

If the JIT can prove your allocation is unused, it deletes it, and you benchmark an empty loop. Use
JMH's `Blackhole` or return values, and — critically — **check `gc.alloc.rate.norm`** from
`-prof gc`. It reports bytes allocated per operation. If that number is not what the source code
implies, escape analysis removed something and the benchmark is measuring nothing.

## 6.9 One run is not a measurement

Everything in [`results/`](../results) is a single run on a single laptop, and is presented as an
illustration of *shape*, not as a benchmark result you should plan capacity from. Ryzen 5 5600U,
Windows 11, 15.3 GB RAM, JDK 25.0.3 — with browser tabs, an OS, and thermal limits. Server hardware,
Linux, and a quiet machine will produce different absolute numbers.

What generalises is the relationship: **G1 wins the median, ZGC wins the tail, and the gap widens as
the live set grows.** What does not generalise is any specific millisecond figure.

---

Next: [07-corner-cases.md](07-corner-cases.md).