asmhatre
f506b01389
Companion module for the rewritten post 'Virtual Threads on Spring Boot 4.1: The Benchmarks, Re-Run, and the Pinning Advice That Expired', retitled and re-benchmarked on Boot 4.1.1 / JDK 25.0.4.1 (the original post was written against Boot 3.4 / JDK 21). Covers: I/O-bound and CPU-bound throughput (platform vs virtual threads, including a JIT-warmup benchmarking bug this build caught and fixed), JEP 491 proof that synchronized no longer pins a virtual thread's carrier across a blocking call as of JDK 24 (obsoleting the old avoid-synchronized advice), proof that -Djdk.tracePinnedThreads=full is inert on JDK 25, and JEP 506's finalized ScopedValue API (JDK 25 GA, no --enable-preview, and a different shape than the old preview API). Kept as its own module rather than a new top-level repository, alongside the existing async/ module, which already has a stronger dual-JDK JEP 491 proof that this module's docs cross-link to instead of duplicating.
82 lines
4.7 KiB
Markdown
82 lines
4.7 KiB
Markdown
# 2. Benchmark methodology and results
|
|
|
|
[Previous: 01-enabling-virtual-threads.md](01-enabling-virtual-threads.md) | [README](../README.md) | Next: [03-pinning-diagnosis.md](03-pinning-diagnosis.md)
|
|
|
|
Source: [`DemoController.java`](../src/main/java/com/ankurm/vthreads/DemoController.java).
|
|
Test: [`LoadBenchmarkTest.java`](../src/test/java/com/ankurm/vthreads/LoadBenchmarkTest.java).
|
|
Transcripts: [`docs/output/01-io-bound-benchmark.txt`](output/01-io-bound-benchmark.txt), [`docs/output/02-cpu-bound-benchmark.txt`](output/02-cpu-bound-benchmark.txt).
|
|
|
|
## Why this repo doesn't reuse the old post's numbers
|
|
|
|
The version of this post it replaces benchmarked Boot 3.4.0 / JDK 21.0.3 with `wrk` (1,000
|
|
connections, 30s) on a 4-core/8GB box. Neither `wrk` nor an equivalent load generator is
|
|
available in this repo's build environment, and the environment itself is a 2 vCPU sandbox,
|
|
not a 4-core server. Rather than copy the old numbers forward with a changed version number
|
|
next to them, this repo builds its own concurrent load generator (`java.net.http.HttpClient`
|
|
backed by a virtual-thread executor, itself only used as the *client* -- see
|
|
`fireConcurrent()`) and reports what actually happened on this run, on this hardware, honestly
|
|
labeled as such.
|
|
|
|
## I/O-bound: `/io`, `Thread.sleep(300)`
|
|
|
|
600 concurrent requests, each a 300ms simulated downstream call:
|
|
|
|
```
|
|
platform threads : total=600 success=600 wall=1256ms p50=762ms p99=1193ms
|
|
virtual threads : total=600 success=600 wall=675ms p50=531ms p99=654ms
|
|
```
|
|
|
|
600 concurrent requests against Tomcat's default 200-thread platform pool queue in roughly
|
|
three sequential batches of 200; each batch pays the full 300ms, so total wall time lands
|
|
around 3 x 300ms plus scheduling overhead -- which is what the platform-thread row shows.
|
|
Virtual threads create one thread per request and unmount for the duration of the sleep, so
|
|
wall time stays close to a single 300ms round, regardless of how far past 200 the concurrency
|
|
goes. The shape matches the original 2025 benchmark; the absolute numbers are this sandbox's,
|
|
not that server's.
|
|
|
|
## CPU-bound: `/cpu`, 20,000x SHA-256 per request
|
|
|
|
This is the scenario worth reading carefully, because the first version of this benchmark
|
|
was wrong, and the fix is itself worth knowing about.
|
|
|
|
**First cut, concurrency=16, no warm-up round:** virtual threads finished 4x faster than
|
|
platform threads. That result doesn't make physical sense -- a CPU-bound virtual thread never
|
|
unmounts, so it competes for the same physical cores a platform thread would, and should show
|
|
no structural advantage. The cause: `MessageDigest.digest` and the servlet dispatch path get
|
|
JIT-compiled **per class**, not per Spring context, and this benchmark starts a fresh
|
|
`ApplicationContext` for each of the two scenarios inside the same JVM. Platform threads always
|
|
ran first; by the time the virtual-thread scenario ran, the hot loop was already JIT-warmed --
|
|
a benchmarking artifact with nothing to do with thread model, caught by the numbers being
|
|
implausible rather than by inspecting the code.
|
|
|
|
**Fix:** an untimed warm-up round (`fireConcurrent` at a smaller concurrency, discarded)
|
|
through the same code path before every timed measurement, in both scenarios.
|
|
|
|
**After the fix, concurrency=60** (chosen deliberately above this box's 2 vCPUs, so both
|
|
thread models are forced into real core-bound saturation rather than fitting comfortably
|
|
alongside each other):
|
|
|
|
```
|
|
platform threads : total=60 success=60 wall=166ms p50=146ms p99=162ms
|
|
virtual threads : total=60 success=60 wall=191ms p50=181ms p99=189ms
|
|
```
|
|
|
|
Close, with platform threads and virtual threads trading the lead by a few percent across
|
|
repeated runs (see the test's own run-to-run notes in
|
|
[`LoadBenchmarkTest.java`](../src/test/java/com/ankurm/vthreads/LoadBenchmarkTest.java)) --
|
|
consistent with the "no meaningful difference for CPU-bound work" claim every virtual threads
|
|
article makes, but arrived at here by running it, catching a real measurement bug, fixing the
|
|
methodology, and re-running it, rather than by assuming the claim was self-evidently true.
|
|
|
|
<blockquote>If your own CPU-bound benchmark shows virtual threads winning decisively, check
|
|
your warm-up before you check your thread model. JIT compilation state leaking between two
|
|
scenarios measured in the same JVM run is an easy way to manufacture a result that isn't
|
|
real.</blockquote>
|
|
|
|
- If you're benchmarking your own service: match concurrency to something past your real
|
|
platform-thread pool size for the I/O case, and past your core count for the CPU case --
|
|
otherwise neither test forces the behaviour you're trying to observe.
|
|
- JEP 444 (virtual threads, JDK 21): <https://openjdk.org/jeps/444> (`rel=nofollow`)
|
|
|
|
Next: [03-pinning-diagnosis.md](03-pinning-diagnosis.md).
|