JMH Tutorial: Writing Java Microbenchmarks That Don’t Lie
A beginner-first JMH tutorial, verified on JDK 25 and JMH 1.37: dead-code elimination, Blackhole, constant folding, warmup, forks, state scopes, -prof gc and percentiles, each shown with real captured output.
You write a loop, wrap it in System.nanoTime(), divide by the iteration count, and the answer is 0.000 nanoseconds per operation. Either you have written the fastest method in the history of the JVM, or the measurement is lying. It is the second one, and it is lying in a specific, explainable way.
This post teaches JMH, the Java Microbenchmark Harness, from the first wrong number to the last trustworthy one. It starts from what a hand-rolled timing loop gets wrong, then adds one JMH feature per section — Blackhole, constant folding, warmup, forks, state scopes, profilers, percentiles — each one introduced by the specific lie it prevents. Every number below came out of a real run on JDK 25 and is stored in the companion repository; each code block and output block links to the file it came from.
You do not need to have used JMH before. The sections are in difficulty order, and the collapsible “going deeper” panels hold the intermediate and reference-level material, so you can read straight through on a first pass and open the panels later.
Versions this was built and run against. JMH 1.37 — the latest GA release on Maven Central (maven-metadata.xml lists no later non-milestone version), announced on 3 August 2023. JDK 25.0.5+8 (Temurin, LTS), Maven 3.9.11, JUnit Jupiter 5.11.0. Host: a shared 2-vCPU sandbox, so treat the numbers as shape (which is bigger, by roughly how much) rather than as figures to three significant digits. Where an error bar is wider than a difference, the post says so.
A timing loop around nanoTime() measures the wrong thing
Here is the benchmark almost everyone writes first. It takes the cost of Math.log over 20 million iterations, twice: once throwing the result away, once adding it into a running total that is printed at the end. The code is in NaiveTimingDemo.java.
/** Result thrown away - nothing observable depends on the loop. */
static void discard() {
for (int i = 1; i <= N; i++) {
Math.log(i);
}
}
/** Result kept and printed - the loop must really run. */
static double keep() {
double sum = 0;
for (int i = 1; i <= N; i++) {
sum += Math.log(i);
}
return sum;
}
Two different things are wrong here, and they are worth separating because JMH has a separate answer for each.
The discarded loop reports 0.000 ns/op from round 3. Nothing observable depends on what Math.log returns, so once the JIT compiler has compiled the loop it is allowed to delete the work. That is called dead-code elimination. The figure is not the cost of Math.log; it is the cost of a loop that no longer exists.
The kept loop never gives the same answer twice. It starts at 9.170 ns/op and ends at 7.419 — about 19% faster over six rounds, with no change to the code. The JVM begins by interpreting your bytecode and compiles the hot parts as it learns which ones are hot, so early rounds include the cost of the JVM figuring out your program. That phase is called warmup. A timing loop that runs once reports a number from somewhere in the middle of that curve, and you cannot tell where.
Neither column is a benchmark result. A trustworthy microbenchmark needs a harness that keeps results alive so they cannot be deleted, runs the code long enough to get past warmup, and repeats the whole measurement in fresh JVMs so one unlucky start does not decide the answer. That is the job description of JMH, and the rest of this post is that job description, one duty at a time.
Going deeper: what else a hand-rolled loop cannot do
A nanoTime() loop has no answer for four more problems, each covered by its own section below:
Constant folding — the JIT computes the answer at compile time because the input is a constant (section 4).
Cross-contamination — benchmarks run one after another in the same JVM change each other’s results (section 6).
Shared state — where the data lives decides whether you are measuring the operation or the contention around it (section 7).
Tail latency — an average over millions of calls hides the calls that hurt (section 9).
JMH is not a library you call from main. You annotate ordinary methods with @Benchmark, and an annotation processor runs at compile time and generates the code that does the measuring: the timing loop, the warmup, the forking of fresh JVMs, and the plumbing that stops results being optimised away. Your method is the thing being measured; everything around it is generated.
One run has a fixed shape. JMH starts a fresh JVM (a fork), runs several warmup iterations whose results it throws away, then several measurement iterations whose results it keeps, and repeats that for each fork. Inside every iteration it calls your method in a loop for a fixed amount of time and consumes the return value.
The diagram is the whole execution model. Everything JMH reports is a statistic over the green boxes; everything amber exists to get the JVM into a steady state first. The sections below each add a rule about one part of this picture.
To use JMH you need two dependencies, the annotation processor wired into the compiler, and the shade plugin so the result is one runnable jar. This is the essential part of pom.xml:
The first lines JMH prints say what it is about to do. These are the first nine lines of output/02-deadcode-jmh.txt:
# JMH version: 1.37
# VM version: JDK 25.0.5, OpenJDK 64-Bit Server VM, 25.0.5+8-LTS
# VM invoker: /tmp/tools/jdk-25.0.5+8/bin/java
# Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 3 iterations, 1 s each
# Measurement: 5 iterations, 1 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
Read a few of those lines now and the rest of the post will make more sense. Warmup: 3 iterations, 1 s each and Measurement: 5 iterations, 1 s each are the amber and green boxes from the diagram, set by annotations on the class. Benchmark mode: Average time, time/op says the score is nanoseconds per call, lower is better. And Blackhole mode: compiler is a detail that matters in the next section.
Going deeper: defaults, and where the annotations come from
If you set nothing, JMH uses its own defaults. Reading the constants in org.openjdk.jmh.runner.Defaults from the 1.37 jar with javap -constants gives 5 warmup iterations, 5 measurement iterations and 5 forks (MEASUREMENT_FORKS = 5, WARMUP_FORKS = 0). The benchmark classes in this module override those with @Warmup, @Measurement and @Fork to keep the full capture short; any of them can also be overridden per run on the command line (-wi, -i, -f), which is how output/09 was produced.
The annotation processor writes its generated classes into the jmh_generated package and a META-INF/BenchmarkList file that the runner reads. The module’s test class reads that file and pins how many entries it holds — 21 for 19 methods, because each @BenchmarkMode value on a method becomes a separate entry (SortBenchmark.sortCopy carries three).
Going deeper
README.md — versions, quickstart and a file-by-file table for the module.
JmhBasicsTest.java — the deterministic claims of this post as assertions.
If nothing uses the result, the compiler deletes the work
Section 1 showed dead-code elimination at work in a hand-written loop. JMH does not stop the JIT from doing it — the JIT is behaving correctly — so the benchmark has to give the result somewhere to go. There are two ways: return it from the method, or pass it to a Blackhole, a JMH object whose only job is to receive values so the compiler cannot prove they are unused.
DeadCodeBenchmark.java measures the same Math.log call four ways, next to an empty method that acts as the baseline:
@Benchmark
public void baseline() {
// intentionally empty
}
@Benchmark
public void deadCode() {
Math.log(field); // result discarded
}
@Benchmark
public double returned() {
return Math.log(field); // JMH feeds the return value to a Blackhole
}
@Benchmark
public void viaBlackhole(Blackhole bh) {
bh.consume(Math.log(field));
}
The picture is the argument. deadCode scores 0.364 ± 0.045 ns, indistinguishable from the empty method’s 0.345 ± 0.045. It is about 21 times cheaper than returned, which no implementation of Math.log could be; the benchmark simply did no logarithm. returned and viaBlackhole (7.659 and 8.088 ns, error bars overlapping) are the honest figures: roughly 7–8 ns for the call on this host.
The rule that falls out: every benchmark method should return something that depends on its work, or feed it to a Blackhole. A void benchmark whose body computes a value and discards it is the benchmark equivalent of a test with no assertions.
Check the “Blackhole mode” line before comparing runs. On this JVM JMH printed Blackhole mode: compiler (auto-detected…) in the header, and at the end of the run it printed a notice that the JVM experimentally supports compiler blackholes, that they are in use, and that results from different blackhole modes can differ significantly. The notice, from the end of output/02-deadcode-jmh.txt:
NOTE: Current JVM experimentally supports Compiler Blackholes, and they are in use. Please exercise
extra caution when trusting the results, look into the generated code to check the benchmark still
works, and factor in a small probability of new VM bugs. Additionally, while comparisons between
different JVMs are already problematic, the performance difference caused by different Blackhole
modes can be very significant. Please make sure you use the consistent Blackhole mode for comparisons.
If you compare a number from your laptop with a number from a CI machine, make sure both runs printed the same mode.
Going deeper: the Blackhole API and compiler blackholesBlackhole has a consume overload for every primitive type (byte, boolean, char, short, int, long, float, double) and for Object, plus a static consumeCPU(long) — this is what javap prints for the class in the 1.37 jar. Returning a value from the benchmark method is equivalent to consuming it, so a Blackhole parameter earns its place when one invocation produces several values you need to keep alive.
The header line says the blackhole mode was auto-detected, and tells you the switch that turns auto-detection off: -Djmh.blackhole.autoDetect=false. I did not run with detection off, so this post makes no claim about how large the difference between the two modes is on this host; the JMH notice quoted above is the authority for “can be very significant”.
Going deeper
DeadCodeBenchmark.java — all eight benchmarks, including the baseline, with a comment on what each is expected to show.
If the input is a constant, the compiler does the work before the benchmark starts
Returning the result fixes dead code but leaves the opposite problem. If the JIT can prove the input never changes, it can compute the answer once, at compile time, and the benchmark then measures reading a precomputed number. This is constant folding.
The remedy is to read inputs from fields the JIT cannot treat as constants. Four more benchmarks in DeadCodeBenchmark.java, this time around Math.pow:
static final double CONSTANT = Math.PI;
double field = Math.PI; // plain mutable field: the JIT cannot assume its value
final double finalField = Math.PI; // final: measured below to be folded exactly like a literal constant
@Benchmark
public double powOfConstant() {
return Math.pow(CONSTANT, 10);
}
@Benchmark
public double powOfFinalField() {
return Math.pow(finalField, 10);
}
@Benchmark
public double powOfField() {
return Math.pow(field, 10);
}
@Benchmark
public double logOfConstant() {
return Math.log(CONSTANT);
}
Two inputs the JIT could see through (a static final constant and a final instance field) came out at 0.441 and 0.367 ns — the same as an empty method. The mutable field, which the JIT must re-read on every call, shows the real cost: 14.512 ns, about 33 times more. If you had benchmarked pow with either of the first two you would have reported 30–40 times too fast.
I got this wrong in the first draft of the code. My comment on the final instance field claimed the JIT would not trust it. The measurement says otherwise (0.367 ns, the same as a literal constant), and the JMH sample on constant folding lists a final field as a mistake, so I corrected the Javadoc and the README before publishing. The lesson is the one the sample teaches: read benchmark inputs from non-final instance fields of a @State class.
And folding is not universal.logOfConstant ran at 7.036 ns — real work, no folding — while powOfConstant folded completely. Whether a constant gets folded depends on the operation. I did not investigate why Math.log behaves differently here, so I will not guess; the practical point is that you cannot tell by looking at the source which of your benchmarks the JIT has quietly turned into a constant.
Going deeper: how to catch folding in your own benchmarks
The empty-method baseline from section 3 is also your folding detector. Any benchmark that scores within error of the baseline is measuring nothing. This is how both the dead deadCode result and the folded powOfConstant and powOfFinalField results are recognisable in the table: they sit on top of the baseline’s 0.345 ± 0.045 ns.
A second check costs one line: change the input and see whether the score moves. A benchmark whose result does not depend on its input is not measuring the operation.
For cases where you need to prevent the JIT inlining or compiling a particular method, JMH has @CompilerControl with modes including DONT_INLINE, EXCLUDE, INLINE and COMPILE_ONLY (the enum members in the 1.37 jar). Those are powerful and easy to misuse; reach for them only after the simpler checks above.
Going deeper
JMHSample_10_ConstantFold — the JMH sample on constant folding, with the same four inputs in miniature.
Warmup: the JVM is still learning your code while JMH measures it
The first benchmark in section 1 drifted by about 19% over six rounds. JMH handles this with the amber boxes from the diagram: warmup iterations that run the benchmark, print their time, and are then thrown away. The measurement iterations start only after that.
SortBenchmark.java sorts a copy of a 1,000-element array. It clones first, because sorting the original in place would leave every later call sorting an already-sorted array. Ten warmup iterations of 200 ms, then five measured ones (command in scripts/run-all.sh):
@Benchmark
public int[] sortCopy() {
int[] copy = original.clone(); // sorting in place would make every call after the first trivial
Arrays.sort(copy);
return copy;
}
Two things to read off the chart. The first iteration costs 9.502 µs and by the end of warmup the same code costs about 4.359 µs — roughly twice as fast — so a benchmark without warmup would have reported a number that is not the program’s steady-state behaviour. And the descent is not smooth: iterations 6 and 7 are 6.260 and 6.648 µs, higher than iteration 5’s 4.362. On a shared 2-vCPU host I cannot tell you whether that is the JIT recompiling, the host being busy, or both, and this data does not separate them. What it does show is why “warm up for N iterations” cannot be derived from theory: you read the trace for your benchmark on your machine.
The five measured iterations still vary by about 29% (from 4.753 to 6.121 µs). That variation is what the ± column in the summary reports: 5.414 ± 2.138 means the 99.9% confidence interval on the mean is wide enough that a difference of 10% between two such results would not be meaningful.
Going deeper: choosing warmup and measurement settings
The settings live in two annotations, @Warmup(iterations = 3, time = 1) and @Measurement(iterations = 5, time = 1), or on the command line as -wi/-w and -i/-r (that is how this run used 10 warmup iterations of 200 ms). JMH’s defaults are 5 and 5 iterations (Defaults in the 1.37 jar).
A practical procedure: run once with a generous warmup and read the trace as above. When the curve has been flat for a few iterations, that is your warmup count; add a margin. If the measured iterations vary a lot, more iterations or more forks reduce the error bar; a longer warmup does not.
One caution that comes straight from this data: the whole warmup here is two seconds on a noisy host. Do not generalise the “about twice as fast” ratio to other code; it is a property of this sort on this JVM.
Going deeper
SortBenchmark.java — this benchmark, which also drives the modes demo in section 9.
Forks: one JVM per benchmark, or the benchmarks contaminate each other
Warmup raises a harder problem. If two benchmarks run one after the other in the same JVM, the second one starts with whatever the JVM learned during the first. HotSpot records which concrete types it has seen at each call site and optimises for them. A call site that has only ever seen one type can be compiled very aggressively; the moment a second type shows up, that optimisation is thrown away.
ForkingBenchmark.java sets this up on purpose. Three benchmarks all call through the same method, measure. Two pass an AddOne; the middle one passes an AddTwo:
public interface Step { int apply(int x); }
/** The single shared call site whose type profile gets polluted. */
public int measure(Step step) {
int acc = 0;
for (int i = 0; i < 10; i++) {
acc = step.apply(acc);
}
return acc;
}
@Benchmark
public int measure_1_c1() { return measure(one); }
@Benchmark
public int measure_2_c2() { return measure(two); }
@Benchmark
public int measure_3_c1() { return measure(one); }
Run it twice, once with every benchmark sharing one JVM (-f 0) and once with a fresh JVM per benchmark (-f 1). With -f 0 JMH prints a warning, which is in output/04-forking-f0.txt:
# *** WARNING: Non-forked runs may silently omit JVM options, mess up profilers, disable compiler hints, etc. ***
# *** WARNING: Use non-forked runs only for debugging purposes, not for actual performance runs. ***
The shared-JVM run gives 0.912 ns for measure_1_c1, then 2.455 and 2.347 for the other two. The third benchmark is identical to the first — same method, same AddOne — and is about 2.6 times slower, because by then the call site has seen two types. With a fresh JVM per benchmark the three results are 0.728, 0.792 and 0.733 ns, all within each other’s error bars, which is what three benchmarks that all do the same amount of work should look like.
The mechanism — type profiles at a shared call site — is the one the JMH sample on forking describes; my numbers show the symptom, and I did not inspect the JIT’s inlining decisions to confirm the cause. The symptom is the part that matters for you: the order your benchmarks happen to run in changed their results by about 3.2x.
Non-forked runs are for debugging only. The warning JMH prints says so itself, and it adds that non-forked runs can silently omit JVM options and disturb profilers. This is also why LifecycleDemo (section 7) uses forks(0): it needs to read counters in-process, and it makes no timing claims.
Going deeper: forks and the Cnt column
Forking is the default. Defaults.MEASUREMENT_FORKS in the 1.37 jar is 5 and WARMUP_FORKS is 0. @Fork(2) on DeadCodeBenchmark is why the Cnt column in section 3 reads 10: two forks times five measurement iterations. With -f 1 -i 8, as in output/05, Cnt is 8.
More forks tell you something iterations cannot: whether the result depends on how a particular JVM happened to compile things. Two forks that disagree with each other by more than their error bars are reporting JIT variance, and that is information about your benchmark, not noise to average away.
Going deeper
State: where the data lives decides what you are measuring
Benchmarks need data — an array to sum, a counter to increment. In JMH that data lives in a @State class, and its scope says who shares it. Scope.Thread gives every benchmark thread its own instance; Scope.Benchmark gives all threads the same one (a third scope, Scope.Group, exists for thread groups).
StateScopeBenchmark.java increments an AtomicLong from two threads. The only difference between the two benchmarks is the scope of the state object:
@State(Scope.Benchmark)
public static class Shared {
final AtomicLong counter = new AtomicLong();
}
@State(Scope.Thread)
public static class PerThread {
final AtomicLong counter = new AtomicLong();
}
@Benchmark
public long sharedCounter(Shared state) {
return state.counter.incrementAndGet();
}
@Benchmark
public long perThreadCounter(PerThread state) {
return state.counter.incrementAndGet();
}
The same incrementAndGet() costs 7.950 ns when each thread has its own counter and 31.598 ns when two threads share one — about 4.0 times more. The shared version has a wide error bar (± 9.365), so read it as “several times slower”, not as a precise multiple. Neither number is wrong; they answer different questions. Scope.Thread measures the operation. Scope.Benchmark measures the operation while another core is fighting you for the same memory. Choosing the wrong scope would give you the right answer to the wrong question.
State also has a lifecycle. @Setup and @TearDown methods run at one of three levels: once per fork (Trial), once per iteration (Iteration), or around every single call (Invocation). LifecycleDemo.java counts them for a run of 1 warmup and 2 measurement iterations:
The counts are exact, and the test suite pins them: trial 1, iteration 3, and invocation setup equal to the number of benchmark calls (1,039,152 in this run — the count varies by run, but equality is asserted). That last line is the important one. A setup at Level.Invocation fires on every call, so it is only the right choice when each call genuinely needs fresh state, such as sorting an array in place. For everything else, set up at Trial or Iteration and keep the measured path as thin as possible.
Going deeper: @Param, and building data outside the measured region@Param runs one benchmark at several input sizes, and a @Setup method builds the data outside the measured region. SumBenchmark.java sums an int[] at three sizes, with a fixed random seed so every run sees the same data:
@Param({"10", "1000", "100000"})
public int size;
int[] data;
@Setup
public void fill() {
Random random = new Random(42); // fixed seed: the same data on every run
data = new int[size];
for (int i = 0; i < size; i++) {
data[i] = random.nextInt(100);
}
}
@Benchmark
public long sum() {
long total = 0;
for (int v : data) {
total += v;
}
return total;
}
Going from 10 to 1,000 elements (100 times the data) cost 5.1 times the time; going from 1,000 to 100,000 cost 136 times. Per element that is 1.07 ns, 0.054 ns and 0.074 ns. The per-element cost is not constant, so a benchmark at one size predicts nothing reliable about another, which is the reason @Param exists. I have not investigated what causes the shape of this curve, so I will not offer an explanation; the point is the dependence.
Going deeper
JmhBasicsTest.java — lifecycleLevelsFireExactlyAsDocumented asserts the counts above.
Allocation: ask a profiler, do not guess
“This method allocates an object, so it must be slow” is a guess, and the JVM can make it false. If an object never leaves the method that creates it, HotSpot’s escape analysis can avoid allocating it at all. You cannot see that from the source. JMH ships profilers for exactly this kind of question, and the GC profiler is the one to learn first: -prof gc reports how many bytes each benchmark call allocated.
AllocationBenchmark.java has three methods that all contain new:
record Point(int x, int y) { }
@Benchmark
public int doesNotEscape() {
Point p = new Point(a, b);
return p.x() + p.y();
}
@Benchmark
public Point escapes() {
return new Point(a, b);
}
@Benchmark
public int[] arrayEscapes() {
return new int[4];
}
Run with java -jar target/benchmarks.jar AllocationBenchmark -prof gc. The column that matters is gc.alloc.rate.norm, bytes per benchmark call, from output/07-allocation-gc.txt:
All three methods contain a new, and two of them allocate. doesNotEscape creates a Point, reads two fields, and returns an int; the profiler reports essentially no allocation (≈ 10⁻⁵ B/op), which is what you would expect if escape analysis removed the object. escapes returns the Point, so it must exist on the heap: 24 bytes. arrayEscapes returns a four-element array: 32 bytes.
The practical consequence: a benchmark that appears to measure “the cost of creating a Point” may be measuring nothing of the kind, and the profiler is how you find out which you have. That is the same lesson as sections 3 and 4 from a different angle: the number is only as good as your evidence that the thing you meant to measure actually ran.
Going deeper: which profilers exist, and which ran here
JMH lists the profilers available on a given machine with -lprof. On this sandbox, from output/11-list-profilers.txt:
Supported profilers:
cl: Classloader profiling via standard MBeans
comp: JIT compiler profiling via standard MBeans
gc: GC profiling via standard MBeans
jfr: Java Flight Recorder profiler
mempool: Memory pool/footprint profiling via standard MBeans
pauses: Pauses profiler
safepoints: Safepoints profiler
stack: Simple and naive Java stack profiler
The rest of that file is the other half of the answer: the profilers that are not available here, each with the reason. perf-based ones (perf, perfasm, perfnorm, perfc2c) fail with Cannot run program "perf", and async fails because the async-profiler library is not on the library path. That is worth knowing in advance: the assembly-level profilers need OS-level tooling installed, and a missing tool shows up as an entry in the unsupported list rather than as an error in your benchmark.
This post uses only gc. I did not run the others, so I make no claims about their output.
Going deeper
output/07-allocation-gc.txt — the full profiler output, including the other secondary columns (allocation rate, GC count, GC time).
An average hides the slow calls
Everything so far reported an average. That is the right question for “which of these two implementations is faster” and the wrong one for “how bad does it get”. JMH has four benchmark modes, set with @BenchmarkMode: Throughput (calls per unit time), AverageTime (time per call), SampleTime (a sampled distribution of individual call times), and SingleShotTime (one call, for cold-start measurements). There is also All.
SortBenchmark.java from section 5 carries three modes at once, so one run reports throughput, average time, and sampled latency with percentiles. The percentile block for one iteration of sampling mode, from output/10-sort-modes.txt:
The median call took 4.280 µs. One call in a hundred took 20.843 µs or more, one in a thousand 62.631, and one in ten thousand 953.847 µs — about 223 times the median. An average over this distribution would have reported roughly 5–7 µs and hidden all of it. If what you care about is a latency target rather than total throughput, the percentile block is the result and the mean is a distraction.
One caution that belongs with every tail number: on a shared host, a large share of the extreme tail is the host, not your code. The p0.9999 value was 4063.431 µs in iteration 1 and 953.847 µs in iteration 3 of the same run, so use the tail to see that it exists and to compare two implementations on the same machine, not to quote an absolute figure.
Going deeper: reading the three modes in one run
The same method reports in three units in output/10-sort-modes.txt: throughput in ops/us, average time in us/op, and the sampled distribution above. They describe the same program, so they should agree: throughput of about 0.16–0.20 ops/µs is the reciprocal of an average of roughly 5–6 µs per call, and they do.
The @OutputTimeUnit annotation sets the unit for all of them. Because each mode is a separate entry in the generated benchmark list, a method with three modes takes three times as long to run, which is why section 2’s test counts 21 entries for 19 methods.
Going deeper
Each row is a symptom from this post, the cause, and where it was demonstrated. Run down it before you believe a benchmark result, and again before you quote one.
What you see
Likely cause
What to do
Section
Score within error of the empty baseline
Dead-code elimination, or the input was folded
Return the result or use a Blackhole; read inputs from non-final fields
3, 4
Result does not change when you change the input
Constant folding
Move the input into a @State field; vary it and re-run
4
Early iterations much slower than later ones
Warmup
Read the warmup trace; set warmup until it flattens
5
A benchmark’s result depends on which others ran
Shared JIT profile in a non-forked run
Fork at least once; never trust -f 0 for numbers
6
Multi-threaded result far worse than single-threaded
Shared state, or the wrong @State scope
Decide deliberately between Thread and Benchmark
7
Setup cost visible in the score
Level.Invocation setup on a fast benchmark
Set up at Trial or Iteration level
7
You expect allocation but see none, or the reverse
Escape analysis
Run with -prof gc
8
Average looks fine, users complain
Tail latency
Use SampleTime and read the percentiles
9
Error bar larger than the difference you are claiming
Not enough forks or iterations, or a noisy host
More forks; run on a quiet machine; report only large differences
5, 6
Should you microbenchmark this at all? JMH answers one narrow question well: which of these two small, self-contained pieces of code is faster, on this JVM and this hardware? It does not tell you whether the change will make your service faster, because real services spend their time in I/O, locks, caches and garbage collection that a microbenchmark deliberately removes. Use JMH to choose between two implementations of the same small thing; use a profiler and a load test on the real application to find out where the time actually goes. And because every number in this post carries the caveat of a shared 2-vCPU host, treat an unexpected result as a prompt to re-run on a quiet machine before you act on it.
No Comments yet!