Add jfr-demo: JFR + Mission Control profiling of a live Spring Boot app

Custom OrderProcessed event, CPU/allocation/lock workloads, a diagnostic
endpoint over the JVM's live FlightRecorder state, a JUnit test using the
jdk.jfr.Recording API, and nine captured recordings (flagship, mid-flight
snapshot, isolated vs concurrent threshold comparisons, and a real
dump+stop duplication bug) with their docs/output/*.txt transcripts.
This commit is contained in:
2026-10-01 04:37:26 +00:00
commit c7eccd1db5
40 changed files with 1326 additions and 0 deletions
+16
View File
@@ -0,0 +1,16 @@
# docs/output/01-order-event-test.txt
# Generated by OrderProcessedEventTest.capturesOneEventPerOrder()
# jdk.jfr.Recording started in-process, 20 orders processed, recording stopped and dumped to a temp .jfr file,
# then read back with jdk.jfr.consumer.RecordingFile.
events recorded: 20
events with slowReason: 4
events without reason: 16
first event fields:
eventType = com.ankurm.jfr.OrderProcessed
orderId = TEST-0
warehouse = BLR-2
itemCount = 3
duration(ms) = 5
hasStackTrace = true
@@ -0,0 +1,14 @@
$ jcmd 5287 JFR.start name=demo settings=profile maxsize=256m
5287:
Started recording 15.
Use jcmd 5287 JFR.dump name=demo filename=FILEPATH to copy recording data to file.
$ curl -s localhost:8080/api/jfr/recordings
[{"id":15,"name":"demo","state":"RUNNING","startTime":"2026-10-01T04:30:41.255843977Z","maxSize":268435456,"maxAge":"null","destination":"null"}]
$ jcmd 5287 JFR.stop name=demo filename=/tmp/throwaway-demo.jfr
5287:
Stopped recording "demo", 251.6 kB written to:
/tmp/throwaway-demo.jfr
+39
View File
@@ -0,0 +1,39 @@
Top stack frames from jdk.ExecutionSample, live-demo.jfr, during six concurrent
GET /api/cpu/fibonacci?n=38 calls, aggregated by the top (leaf) frame only.
$ jfr print --events jdk.ExecutionSample --stack-depth 1 recordings/live-demo.jfr \
| grep -A1 stackTrace | grep -v 'stackTrace\|^--' | sed 's/^\s*//' | sort | uniq -c | sort -rn
62 com.ankurm.jfr.service.WorkloadService.fibonacci(int) line: 40
13 com.ankurm.jfr.service.WorkloadService.churnAllocations(int) line: 56
9 com.ankurm.jfr.service.WorkloadService.churnAllocations(int) line: 52
2 jdk.internal.util.DecimalDigits.uncheckedGetCharsLatin1(int, int, byte[]) line: 168
1 org.apache.tomcat.util.net.NioEndpoint$Poller.run() line: 941
1 org.apache.tomcat.util.http.MimeHeaders.addValue(byte[], int, int) line: 327
1 org.apache.catalina.mapper.Mapper.compareIgnoreCase(CharChunk, int, int, String) line: 1401
1 org.apache.catalina.core.ApplicationFilterFactory.createFilterChain(ServletRequest, Wrapper, Servlet) line: 75
1 jdk.internal.util.DecimalDigits.stringSize(int) line: 101
1 jdk.internal.misc.Unsafe.allocateUninitializedArray(Class, int) line: 1396
1 java.util.concurrent.locks.LockSupport.unpark(Thread) line: 181
1 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.signalAll() line: 1590
1 java.util.Arrays.copyOf(Object[], int) line: 3478
1 java.nio.Buffer.position(int) line: 325
1 java.lang.invoke.DirectMethodHandle.allocateInstance(Object) line: 500
total jdk.ExecutionSample events in recording: 98
fibonacci() share of samples: 62 / 98 = 63.3%
$ jfr print --events jdk.ExecutionSample recordings/live-demo.jfr | head -13
jdk.ExecutionSample {
startTime = 09:59:03.699 (2026-10-01)
sampledThread = "http-nio-8080-exec-1" (javaThreadId = 30)
state = "STATE_RUNNABLE"
stackTrace = [
com.ankurm.jfr.service.WorkloadService.fibonacci(int) line: 40
com.ankurm.jfr.service.WorkloadService.fibonacci(int) line: 40
com.ankurm.jfr.service.WorkloadService.fibonacci(int) line: 40
com.ankurm.jfr.service.WorkloadService.fibonacci(int) line: 40
com.ankurm.jfr.service.WorkloadService.fibonacci(int) line: 40
...
]
}
+35
View File
@@ -0,0 +1,35 @@
Allocation pressure from three concurrent GET /api/alloc/churn?iterations=400000
calls, captured by jdk.ObjectAllocationSample in recordings/live-demo.jfr.
$ jfr print --events jdk.ObjectAllocationSample recordings/live-demo.jfr | grep objectClass \
| sed 's/^\s*//' | sort | uniq -c | sort -rn
154 objectClass = java.lang.Object[] (classLoader = bootstrap)
111 objectClass = byte[] (classLoader = bootstrap)
21 objectClass = java.util.ArrayList (classLoader = bootstrap)
19 objectClass = java.lang.String (classLoader = bootstrap)
4 objectClass = java.time.Instant (classLoader = bootstrap)
2 objectClass = java.io.EOFException (classLoader = bootstrap)
1 objectClass = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode (classLoader = bootstrap)
1 objectClass = java.util.concurrent.ConcurrentHashMap$KeyIterator (classLoader = bootstrap)
1 objectClass = java.util.LinkedHashMap$Entry (classLoader = bootstrap)
1 objectClass = java.util.HashSet (classLoader = bootstrap)
Top allocating stack frames (--stack-depth 1), same recording:
153 java.util.ArrayList.<init>(int) line: 157
58 java.lang.String.encodeUTF8(byte, byte[], boolean) line: 1305
49 jdk.internal.misc.Unsafe.allocateUninitializedArray(Class, int) line: 1396
21 com.ankurm.jfr.service.WorkloadService.churnAllocations(int) line: 55
18 com.ankurm.jfr.service.WorkloadService.churnAllocations(int) line: 53
4 java.time.Instant.create(long, int) line: 416
4 java.nio.HeapByteBuffer.<init>(int, int, MemorySegment) line: 75
The two biggest allocators by class (Object[] and byte[]) both trace back to one
line in WorkloadService.churnAllocations: `new ArrayList<>(bytes.length)` followed
by boxing each byte into the list. Object[] is the list's own backing array
(resized as autoboxed Byte objects are added); byte[] is `s.getBytes()`. Total
events: 317 (jdk.ObjectAllocationSample, throttled to 300/s by the "profile"
settings used for this recording -- see docs/output/08-concurrent-recordings-
shared-threshold.txt for why throttle and threshold settings between concurrently
active recordings interact in a way that is easy to misread).
@@ -0,0 +1,45 @@
One GET /api/lock/contend?workers=16&holdMillis=20 call: 16 threads release
simultaneously (via a CountDownLatch) and fight over one synchronized block that
holds for 20 ms. One thread enters immediately; 15 queue up. jdk.JavaMonitorEnter
records each waiter's queueing time.
$ jfr print --events jdk.JavaMonitorEnter recordings/live-demo.jfr | head -15
jdk.JavaMonitorEnter {
startTime = 09:59:07.393 (2026-10-01)
duration = 20.4 ms
monitorClass = java.lang.Object (classLoader = bootstrap)
previousOwner = "pool-135-thread-1" (javaThreadId = 365)
address = 0x7FCAB40A4860
eventThread = "pool-135-thread-16" (javaThreadId = 380)
stackTrace = [
com.ankurm.jfr.service.WorkloadService.holdLedger(int) line: 100
com.ankurm.jfr.service.WorkloadService.lambda$contend$0(CountDownLatch, CountDownLatch, AtomicLong, int) line: 89
java.util.concurrent.Executors$RunnableAdapter.call() line: 545
java.util.concurrent.FutureTask.run() line: 328
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1090
]
}
$ jfr print --events jdk.JavaMonitorEnter recordings/live-demo.jfr | grep duration | sort | uniq -c
1 duration = 101 ms
1 duration = 122 ms
1 duration = 142 ms
1 duration = 162 ms
1 duration = 182 ms
1 duration = 20.4 ms
1 duration = 203 ms
1 duration = 223 ms
1 duration = 243 ms
1 duration = 263 ms
1 duration = 283 ms
1 duration = 303 ms
1 duration = 40.6 ms
1 duration = 60.9 ms
1 duration = 81.0 ms
15 events, one per waiting thread, stepping up in ~20 ms increments (20.4, 40.6,
60.9, 81.0, 101, 122, ... 303 ms) -- exactly the signature of 16 threads queueing
one at a time for a monitor held 20 ms at a time: the Nth waiter waits roughly
(N-1) x 20 ms. The total wall-clock time the HTTP endpoint reported for this call
was 326 ms, consistent with the last waiter's ~303 ms queueing time plus its own
20 ms hold.
+38
View File
@@ -0,0 +1,38 @@
60 GET /api/orders/process calls against the live app while recording "live2"
(settings=profile) was running. OrderProcessedEvent has @Threshold("20 ms"), and
only ~1/3 of simulated orders take the slow path (35 ms, "inventory-lock-wait");
the rest are fast (0-5 ms) and should be filtered out before ever being written.
$ jfr print --events com.ankurm.jfr.OrderProcessed recordings/live-demo.jfr | head -21
com.ankurm.jfr.OrderProcessed {
startTime = 09:59:08.076 (2026-10-01)
duration = 35.2 ms
orderId = "ORD-61"
warehouse = "BOM-1"
amountCents = 0
itemCount = 4
slowReason = "inventory-lock-wait"
eventThread = "http-nio-8080-exec-2" (javaThreadId = 31)
stackTrace = [
com.ankurm.jfr.service.OrderService.processOrder(String, int) line: 39
com.ankurm.jfr.web.OrderController.process(int) line: 25
jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Object, Object[]) line: 104
java.lang.reflect.Method.invoke(Object, Object[]) line: 565
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(Object[]) line: 252
]
}
com.ankurm.jfr.OrderProcessed {
startTime = 09:59:08.156 (2026-10-01)
duration = 35.1 ms
orderId = "ORD-63"
$ jfr print --events com.ankurm.jfr.OrderProcessed recordings/live-demo.jfr | grep -c '^com.ankurm'
23
23 of 60 orders were kept -- every one of them carries slowReason =
"inventory-lock-wait" and a duration at or above the 20 ms threshold. None of
the fast-path orders (duration well under 20 ms) appear at all: the threshold
discarded them before OrderProcessedEvent.commit() ever ran, which is the whole
point of checking event.shouldCommit() after event.end() instead of always
calling commit() unconditionally.
@@ -0,0 +1,43 @@
Same workload (40 rounds of GET /api/lock/contend?workers=2&holdMillis=12, each
round producing exactly one waiter blocked for ~12 ms) run against three
ISOLATED recordings, one at a time, so each sees its own settings only.
--- settings=default (jdk.JavaMonitorEnter threshold = 20 ms) ---
$ jcmd 5287 JFR.start name=soloDefault settings=default
$ for i in $(seq 1 40); do curl -s "localhost:8080/api/lock/contend?workers=2&holdMillis=12" -o /dev/null; done
$ jcmd 5287 JFR.stop name=soloDefault filename=recordings/solo-default.jfr
$ jfr print --events jdk.JavaMonitorEnter recordings/solo-default.jfr | grep -c duration
0
--- settings=profile (jdk.JavaMonitorEnter threshold = 10 ms) ---
$ jcmd 5287 JFR.start name=soloProfile settings=profile
$ for i in $(seq 1 40); do curl -s "localhost:8080/api/lock/contend?workers=2&holdMillis=12" -o /dev/null; done
$ jcmd 5287 JFR.stop name=soloProfile filename=recordings/solo-profile.jfr
$ jfr print --events jdk.JavaMonitorEnter recordings/solo-profile.jfr | grep -c duration
40
$ jfr print --events jdk.JavaMonitorEnter recordings/solo-profile.jfr | grep duration | sort | uniq -c
1 duration = 10.2 ms
13 duration = 12.1 ms
11 duration = 12.2 ms
9 duration = 12.3 ms
1 duration = 12.4 ms
1 duration = 12.6 ms
1 duration = 13.6 ms
1 duration = 14.2 ms
1 duration = 14.9 ms
1 duration = 30.3 ms
--- settings=default again, this time with a 35 ms hold (above BOTH thresholds) ---
$ jcmd 5287 JFR.start name=soloDefault2 settings=default
$ for i in $(seq 1 10); do curl -s "localhost:8080/api/lock/contend?workers=2&holdMillis=35" -o /dev/null; done
$ jcmd 5287 JFR.stop name=soloDefault2 filename=recordings/solo-default-longhold.jfr
$ jfr print --events jdk.JavaMonitorEnter recordings/solo-default-longhold.jfr | grep -c duration
10
Conclusion, with real counts to back it: a monitor hold of ~12 ms is invisible to
"settings=default" (0 of 40 waits captured) but fully visible to "settings=profile"
(40 of 40). The same default recording catches every wait once the hold is raised
to 35 ms (10 of 10). This is `jdk.JavaMonitorEnter`'s threshold doing exactly what
the shipped default.jfc / profile.jfc files say (20 ms vs 10 ms) -- see
docs/output/08-concurrent-recordings-shared-threshold.txt for what happens to this
picture the moment a SECOND recording with a lower threshold is also running.
@@ -0,0 +1,32 @@
The same 40-round, ~12 ms-wait workload as 07-threshold-isolated-comparison.txt,
but this time two recordings are started AT THE SAME TIME on the same JVM: one
with settings=default (JavaMonitorEnter threshold 20 ms) and one with
settings=profile (threshold 10 ms).
$ jcmd 5287 JFR.start name=cmpA settings=default
$ jcmd 5287 JFR.start name=cmpB settings=profile
$ for i in $(seq 1 40); do curl -s "localhost:8080/api/lock/contend?workers=2&holdMillis=12" -o /dev/null; done
$ jcmd 5287 JFR.stop name=cmpA filename=recordings/compare-default.jfr
$ jcmd 5287 JFR.stop name=cmpB filename=recordings/compare-profile.jfr
$ jfr summary recordings/compare-default.jfr | grep JavaMonitorEnter
jdk.JavaMonitorEnter 40 1080
$ jfr summary recordings/compare-profile.jfr | grep JavaMonitorEnter
jdk.JavaMonitorEnter 40 1080
Both recordings show all 40 events -- including in "compare-default.jfr", the one
configured with a 20 ms threshold, for waits that are only ~12 ms long. Compare
this to 07-threshold-isolated-comparison.txt, where the exact same default
settings, run ALONE with the exact same workload, captured zero of these waits.
The difference is that the two recordings were active at the same time. Duration
thresholds for events like jdk.JavaMonitorEnter are implemented as a single
instrumentation check shared by the JVM across every currently-active recording,
not a private filter per recording -- the effective threshold the running code
uses is the MINIMUM threshold requested by any active recording. A second,
unrelated recording (a colleague's ad hoc `jcmd JFR.start`, an APM agent's own
continuous recording) with a lower threshold for an event type silently widens
what every other concurrently running recording captures for that same event
type. "My recording uses the default profile" is not, by itself, enough to know
what your recording actually contains on a box where something else might also be
recording.
+49
View File
@@ -0,0 +1,49 @@
While building the main walkthrough, this exact sequence was run against recording
"live" (settings=profile, filename=recordings/profile-run.jfr set at JFR.start):
$ jcmd 5287 JFR.start name=live settings=profile filename=recordings/profile-run.jfr maxsize=256m
$ # ... cpu, allocation, lock-contention and 60 order calls ...
$ jcmd 5287 JFR.dump name=live filename=recordings/profile-run.jfr # same path as JFR.start's filename=
$ jcmd 5287 JFR.stop name=live # no filename -> falls back to the
# recording's own configured destination,
# which is that SAME path
$ jfr summary recordings/profile-run.jfr | grep -i 'Chunks\|JavaMonitorEnter\|OrderProcessed'
Chunks: 3
jdk.JavaMonitorEnter 30 768
com.ankurm.jfr.OrderProcessed 56 1988
The live app had exactly one lock/contend call with 16 workers (expected 15
JavaMonitorEnter events, one per waiting thread -- confirmed independently in
docs/output/05-lock-contention-live-demo.txt) and 60 order calls with a 20 ms
threshold (expected roughly a third to clear the threshold). 30 and 56 are both
suspiciously close to exactly double a plausible real count.
$ jfr print --events com.ankurm.jfr.OrderProcessed --json recordings/profile-run.jfr \
| python3 -c "import json,sys; d=json.load(sys.stdin); e=d['recording']['events']; \
print('printed:', len(e)); \
print('unique (orderId,startTime):', len({(x['values']['orderId'], x['values']['startTime']) for x in e}))"
printed: 56
unique (orderId,startTime): 28
Every one of the 56 printed events is an exact duplicate of one of 28 real
events -- same orderId, same startTime, same duration, same stack trace, each
appearing exactly twice. The same 2x duplication shows up in every other event
type in this file, including jdk.JavaMonitorEnter (30 printed, 15 real -- matching
the independently-confirmed count of 15 waiters for a 16-worker contend call).
The cause: `JFR.dump` was pointed at the SAME file path the recording was already
configured to write to via `filename=` on `JFR.start`. The subsequent `JFR.stop`,
given no filename of its own, fell back to that same configured destination and
wrote the complete recording there again, on top of the file the mid-flight dump
had already written -- doubling every event from before the dump. The fix used
for the rest of this companion project's recordings: either never set `filename=`
on `JFR.start` at all (only ever name a file explicitly on `JFR.dump` / `JFR.stop`,
each one a distinct path -- see recordings/live-demo.jfr and
recordings/mid-flight-snapshot.jfr, produced this way with zero duplication), or
if a recording does have a configured destination, never `JFR.dump` to that exact
same path mid-flight.
recordings/dump-duplication-bug.jfr is kept in this repository, unedited, as a
real example a reader can open in JDK Mission Control and see the doubled events
for themselves.
+42
View File
@@ -0,0 +1,42 @@
The complete, correctly-produced flagship recording for this post:
recordings/live-demo.jfr. Produced by one JFR.start, one mid-flight JFR.dump to a
DIFFERENT file (recordings/mid-flight-snapshot.jfr, to demonstrate inspecting a
running recording without stopping it), and one final JFR.stop with its own
filename= -- the pattern that avoids the duplication bug documented in
docs/output/09-dump-duplication-bug.txt.
$ jfr summary recordings/live-demo.jfr | head -19
Version: 2.1
Chunks: 2
Start: 2026-10-01 04:29:03 (UTC)
Duration: 6 s
Event Type Count Size (bytes)
=============================================================
jdk.ModuleExport 1756 20764
jdk.ActiveSetting 1137 30480
jdk.BooleanFlag 992 31418
jdk.GCPhaseParallel 570 15793
jdk.InitialEnvironmentVariable 360 46224
jdk.ModuleRequire 318 3498
jdk.ObjectAllocationSample 317 5081
jdk.LongFlag 274 9074
jdk.NativeMethodSample 211 2532
jdk.UnsignedLongFlag 186 6468
jdk.ThreadPark 130 5283
jdk.ThreadAllocationStatistics 116 1337
The four event types this post is actually about, pulled out of the same file:
$ jfr summary recordings/live-demo.jfr | grep -E 'ExecutionSample|ObjectAllocationSample|JavaMonitorEnter|OrderProcessed'
jdk.ObjectAllocationSample 317 5081
jdk.ExecutionSample 98 1176
com.ankurm.jfr.OrderProcessed 23 809
jdk.JavaMonitorEnter 15 414
317 allocation samples, 98 CPU samples (63% of them in the one hot method --
docs/output/03-hot-method-samples.txt), 23 kept custom business events out of 60
calls (docs/output/06-custom-event-live-demo.txt), and 15 monitor-enter events
from one 16-worker contention burst (docs/output/05-lock-contention-live-demo.txt)
-- all from a recording six seconds long, on an application that otherwise just
sits idle.
+15
View File
@@ -0,0 +1,15 @@
$ java -cp target/classes com.ankurm.jfr.events.BrokenEventPatternDemo
calls: 200
exceptions thrown: 41
events committed: 159
expected if lossless: 200
events missing: 41 (== exceptions thrown)
BrokenEventPattern.processWithoutTryFinally() has no try/finally around
event.commit(); when riskyStep() throws (about one call in five, by design),
execution leaves the method from the catch in the demo driver, not from the line
that would have called commit(). The event threshold was forced to 0 ms for this
run specifically to rule out "the event was just too fast to pass the default
threshold" as an alternative explanation -- every one of the 41 missing events is
attributable to the missing try/finally, not to filtering. No exception, log line,
or JFR.check output calls this out: the only way to notice is to count.