# 2. Benchmark methodology and results

[Previous: 01-enabling-virtual-threads.md](01-enabling-virtual-threads.md) | [README](../README.md) | Next: [03-pinning-diagnosis.md](03-pinning-diagnosis.md)

Source: [`DemoController.java`](../src/main/java/com/ankurm/vthreads/DemoController.java).
Test: [`LoadBenchmarkTest.java`](../src/test/java/com/ankurm/vthreads/LoadBenchmarkTest.java).
Transcripts: [`docs/output/01-io-bound-benchmark.txt`](output/01-io-bound-benchmark.txt), [`docs/output/02-cpu-bound-benchmark.txt`](output/02-cpu-bound-benchmark.txt).

## Why this repo doesn't reuse the old post's numbers

The version of this post it replaces benchmarked Boot 3.4.0 / JDK 21.0.3 with `wrk` (1,000
connections, 30s) on a 4-core/8GB box. Neither `wrk` nor an equivalent load generator is
available in this repo's build environment, and the environment itself is a 2 vCPU sandbox,
not a 4-core server. Rather than copy the old numbers forward with a changed version number
next to them, this repo builds its own concurrent load generator (`java.net.http.HttpClient`
backed by a virtual-thread executor, itself only used as the *client* -- see
`fireConcurrent()`) and reports what actually happened on this run, on this hardware, honestly
labeled as such.

## I/O-bound: `/io`, `Thread.sleep(300)`

600 concurrent requests, each a 300ms simulated downstream call:

```
platform threads : total=600 success=600 wall=1256ms p50=762ms p99=1193ms
virtual threads  : total=600 success=600 wall=675ms p50=531ms p99=654ms
```

600 concurrent requests against Tomcat's default 200-thread platform pool queue in roughly
three sequential batches of 200; each batch pays the full 300ms, so total wall time lands
around 3 x 300ms plus scheduling overhead -- which is what the platform-thread row shows.
Virtual threads create one thread per request and unmount for the duration of the sleep, so
wall time stays close to a single 300ms round, regardless of how far past 200 the concurrency
goes. The shape matches the original 2025 benchmark; the absolute numbers are this sandbox's,
not that server's.

## CPU-bound: `/cpu`, 20,000x SHA-256 per request

This is the scenario worth reading carefully, because the first version of this benchmark
was wrong, and the fix is itself worth knowing about.

**First cut, concurrency=16, no warm-up round:** virtual threads finished 4x faster than
platform threads. That result doesn't make physical sense -- a CPU-bound virtual thread never
unmounts, so it competes for the same physical cores a platform thread would, and should show
no structural advantage. The cause: `MessageDigest.digest` and the servlet dispatch path get
JIT-compiled **per class**, not per Spring context, and this benchmark starts a fresh
`ApplicationContext` for each of the two scenarios inside the same JVM. Platform threads always
ran first; by the time the virtual-thread scenario ran, the hot loop was already JIT-warmed --
a benchmarking artifact with nothing to do with thread model, caught by the numbers being
implausible rather than by inspecting the code.

**Fix:** an untimed warm-up round (`fireConcurrent` at a smaller concurrency, discarded)
through the same code path before every timed measurement, in both scenarios.

**After the fix, concurrency=60** (chosen deliberately above this box's 2 vCPUs, so both
thread models are forced into real core-bound saturation rather than fitting comfortably
alongside each other):

```
platform threads : total=60 success=60 wall=166ms p50=146ms p99=162ms
virtual threads  : total=60 success=60 wall=191ms p50=181ms p99=189ms
```

Close, with platform threads and virtual threads trading the lead by a few percent across
repeated runs (see the test's own run-to-run notes in
[`LoadBenchmarkTest.java`](../src/test/java/com/ankurm/vthreads/LoadBenchmarkTest.java)) --
consistent with the "no meaningful difference for CPU-bound work" claim every virtual threads
article makes, but arrived at here by running it, catching a real measurement bug, fixing the
methodology, and re-running it, rather than by assuming the claim was self-evidently true.

<blockquote>If your own CPU-bound benchmark shows virtual threads winning decisively, check
your warm-up before you check your thread model. JIT compilation state leaking between two
scenarios measured in the same JVM run is an easy way to manufacture a result that isn't
real.</blockquote>

- If you're benchmarking your own service: match concurrency to something past your real
  platform-thread pool size for the I/O case, and past your core count for the CPU case --
  otherwise neither test forces the behaviour you're trying to observe.
- JEP 444 (virtual threads, JDK 21): <https://openjdk.org/jeps/444> (`rel=nofollow`)

Next: [03-pinning-diagnosis.md](03-pinning-diagnosis.md).