Diagnosing Virtual Thread Pinning in Production: JFR Events, jcmd, and Real Fixes
JEP 491 fixed synchronized-block pinning, not native-frame pinning. A real JNI reproduction, the jdk.VirtualThreadPinned JFR event’s exact fields, reading pinned vs unmounted threads from a jcmd dump, the measured throughput cost, and a bounded-executor fix.
A team upgraded to JDK 25, saw JEP 491 in the release notes, and did the obvious cleanup: ripped -Djdk.tracePinnedThreads=full out of their JVM flags, deleted the code-review rule about avoiding synchronized around blocking calls, and moved on. Weeks later, p99 latency on one endpoint started climbing under load, in a way that looked exactly like the pinning problem they thought they’d just been told was solved. It wasn’t a monitor. It was a legacy JNI-backed compression library three layers down in a dependency, and the diagnostic flag they’d trusted for two years told them, correctly as far as it was concerned, that nothing was wrong – because on JDK 24+ it tells you that no matter what’s actually happening. This article reproduces that exact gap with a real native library, not a description of one, and works through what actually still catches it.
Versions. Primary runtime JDK 25.0.4.1+1 (Temurin, LTS), compared against JDK 21.0.10 (OpenJDK) for the before/after. Native library built with the system gcc, not committed to the repo. All timing on a 2-vCPU x86-64 virtual machine – the same sandbox as the rest of this series.
What JEP 491 actually fixed
Before JDK 24, a virtual thread that entered a synchronized block and then blocked inside it – on I/O, on Thread.sleep, on anything – pinned its carrier thread for the duration, because the continuation backing the virtual thread couldn’t be frozen while a monitor was held. JEP 491, “Synchronize Virtual Threads without Pinning,” shipped GA in JDK 24 and removed that specific cause: a virtual thread can now acquire, hold, and release a monitor independently of its carrier. MonitorPinningDemo.java confirms it directly – the same class, run with -Djdk.tracePinnedThreads=full on both runtimes:
Output: 01-monitor-pinning-jdk21-vs-jdk25.txt, source: MonitorPinningDemo.java. This ground has already been covered thoroughly on this site – the synchronized vs ReentrantLock vs StampedLock post benchmarks the lock types directly, and the Spring Boot 4.1 virtual threads rewrite proves the same JEP 491 fix end to end on a real application, including the finding that the diagnostic flag goes silent for the monitor case. This article picks up exactly where that one explicitly stopped: it says native-callback pinning is “still real… out of scope for a Spring Boot demo.” Here’s the demo.
A continuation can’t be frozen across a native stack frame – full stop, on any JDK version, because there’s no mechanism to save and restore native call state the way the JVM saves and restores Java frames. So a virtual thread running JNI native code (or an FFM downcall) that itself blocks pins its carrier for as long as that blocking lasts, regardless of whether JEP 491 shipped. vtpinning.c is a five-line JNI function that calls straight back into a Java method that sleeps, so the sleep executes with native frames still on the stack – the smallest reproduction of the case JEP 491’s own text calls out:
private static native void blockingNativeCall();
// Called back FROM native code, while native frames are still on this thread's stack.
private static void sleepCallback() {
Thread.sleep(SLEEP_MILLIS);
}
The flag you’ve been told to use doesn’t work anymore – for either case
Run that exact same native-pinning class on JDK 25, with the exact same flag:
$ java -Djdk.tracePinnedThreads=full NativePinningDemo (JDK 24+ runtime - flag is inert)
done
Reported honestly, not smoothed over.-Djdk.tracePinnedThreads doesn’t just fall silent for the pinning JEP 491 fixed – it falls silent for the pinning JEP 491 left completely alone, too. The same native call that still measurably costs wall-clock time below (a fact, not a maybe) produces zero output from the flag every article, including this site’s own Spring Boot 4.1 rewrite, tells you to reach for. That post verified the flag goes quiet for the fixed case; this one verifies it’s also quiet for the case that’s still happening. A team that kept it in their JVM args through a JDK 24+ upgrade gets no signal from it going forward, for either kind of pinning – not an error, not a deprecation notice, just silence that reads as “nothing to see here.”
What replaces it: the jdk.VirtualThreadPinned event
JEP 491 kept the JFR event that reports pinning, and widened what it captures. Enabling it with a zero-millisecond threshold and a stack trace, then running the same native-pinning class:
Output: 03-jfr-virtualthreadpinned-event.txt. Everything the trace flag used to give you is here, plus two things it never had: duration as an actual measured number instead of a stack trace you had to time yourself, and carrierThread naming exactly which platform thread got stuck. pinnedReason distinguishes "Native or VM frame on stack" from a monitor cause the same way the old flag’s reason:NATIVE / reason:MONITOR did. In production this gets enabled the same way as any other JFR event – jcmd <pid> JFR.start settings=profile against a running process, or baked into -XX:StartFlightRecording at startup – and the default profile settings already include it at a sensible threshold, so most teams won’t need the zero-millisecond override used here to guarantee this specific 200ms demo shows up.
JFR needs a recording running before the pin happens. Sometimes you only have a live process and a hunch. ThreadDumpPinnedDemo starts one virtual thread pinned via native code and one merely sleeping (unpinned) at the same time, and a jcmd <pid> Thread.dump_to_file -format=json taken a second later catches both mid-flight:
Output: 04-jcmd-thread-dump-pinned-vs-unpinned.txt, full untrimmed dump entries, source: ThreadDumpPinnedDemo.java. Both virtual threads are TIMED_WAITING, both are asleep – the dump’s state field alone doesn’t distinguish them. Two things do: the pinned entry carries a "carrier": "23" field naming the exact platform thread’s tid it’s stuck to, and its stack includes VirtualThread.parkOnCarrierThread. The unpinned entry has neither – no carrier field at all, because it genuinely unmounted, and a plain parkNanos in its stack instead. Cross-reference the carrier tid against the same dump’s platform-thread entries and you get the exact worker thread a native call is holding hostage – no recording, no waiting for the next occurrence, just one dump.
Diagnostics aside, does this actually matter for throughput, or is it a curiosity? Capping the carrier pool to 2 (-Djdk.virtualThreadScheduler.parallelism=2) and running 4 virtual threads that each block ~1000ms, once through the native-pinning path and once through a plain Thread.sleep:
Output: 05-pinning-throughput-cost.txt, source: PinningThroughputDemo.java. The plain path finishes in ~1000ms because all four virtual threads unmount and share the 2 carriers freely. The native path takes almost exactly double, because each pinned carrier is occupied for the full sleep and can’t run a second virtual thread until it’s released – 4 threads over 2 carriers serializes into two sequential batches. This is the identical mechanism JEP 491 eliminated for monitors, still fully in force for native frames, and the number scales with however many concurrent native-pinning calls your process actually makes.
There’s no JVM flag and no JEP that removes native-frame pinning – it’s a property of what a continuation can and can’t freeze across, not a bug in a specific release. The fix available today is containing the blast radius: never let a pinning native call run directly on the same carrier pool the rest of the application shares. Route it through a small, explicitly-sized platform-thread executor instead, and call that executor from the virtual thread with a plain Future.get() – which parks normally, with no native frame on that thread’s own stack, so it unmounts and leaves the shared pool alone:
ExecutorService nativePool = Executors.newFixedThreadPool(2);
// direct: the virtual thread itself pins one of the shared carriers
blockingNativeCall();
// bounded: only nativePool's threads ever pin; this virtual thread just parks on the Future
Future<?> f = nativePool.submit(BoundedNativeCallDemo::blockingNativeCall);
f.get();
With the carrier pool capped to 2, 2 concurrent native-pinning calls, and 6 unrelated virtual threads doing an ordinary 100ms sleep competing for the same pool:
Output: 06-bounded-executor-fix.txt, source: BoundedNativeCallDemo.java. Calling the native code directly pins both of the 2 carriers for the full run, so the 6 unrelated virtual threads – which have nothing to do with the native call – don’t get to run at all until ~924ms. Routing the identical native calls through a 2-thread dedicated executor lets the unrelated work finish in ~126ms, close to its own 100ms sleep, while the native calls complete on their own separate threads in the background. The pinning cost is still there – it hasn’t vanished – but it no longer reaches into work that has nothing to do with it.
JDBC drivers: the corrected advice, with one caveat
Older JDBC driver warnings about virtual thread pinning were usually about synchronized blocks in the driver’s own socket I/O path – a real problem through JDK 23, and already corrected on this site: on JDK 24+, JEP 491 fixes that class of driver pinning for free, no driver upgrade required. The caveat this article adds: that correction covers synchronized-based pinning specifically. A driver that reaches native code directly – a native compression or encryption library, a thick client mode that shells out through JNI – keeps pinning on every JDK version, for the exact reason demonstrated above, and no JEP targeting monitors touches it. Verify with the jdk.VirtualThreadPinned event against your actual driver rather than assuming either the old warning or the JEP 491 correction covers your specific setup.
Diagnosing suspected pinning in production? Delete -Djdk.tracePinnedThreads from your flags on JDK 24+ – it reports nothing, for either cause, and will not tell you it’s not working. Enable jdk.VirtualThreadPinned instead, either at startup or on demand against a running process with jcmd <pid> JFR.start. If you only have a single point-in-time dump, a jcmd Thread.dump_to_file -format=json shows it too: look for a "carrier" field on a blocked virtual thread, and parkOnCarrierThread in its stack. And if the pinning call is something you can’t rewrite – a third-party native library, a legacy driver – a small dedicated executor contains the cost instead of eliminating it.
No Comments yet!