JUnit 6 is excellent for verifying correctness, but measuring how fast your code runs requires a different approach. A naive long start = System.currentTimeMillis() before and after a method call is not a reliable benchmark — JVM warm-up, JIT compilation, and garbage collection all introduce noise that makes single-run timing meaningless. This guide covers professional performance testing techniques for Java: from JUnit 6’s @Timeout for basic bounds, to full JMH micro-benchmarks integrated into a Maven build.
Three Levels of Performance Testing
| Level | Tool | Purpose | Accuracy |
|---|---|---|---|
| Basic bounds | JUnit 6 @Timeout | Fail if a test exceeds a time limit | Low — single cold run |
| Regression detection | JUnit 6 + assertTimeout | Catch performance regressions in CI | Medium |
| Micro-benchmarking | JMH (Java Microbenchmark Harness) | Precise throughput and latency measurement | High — warmed JVM, statistics |
Level 1: @Timeout — Simple Time Bounds
Use @Timeout to fail a test if it takes too long. This catches catastrophic performance regressions (infinite loops, accidental N+1 queries, missing index):
import org.junit.jupiter.api.Timeout;
import java.util.concurrent.TimeUnit;
class OrderSearchPerformanceTest {
// @Timeout: test fails if it takes longer than the specified duration
// Applied at method level
@Test
@Timeout(value = 2, unit = TimeUnit.SECONDS)
@Tag("performance")
@DisplayName("Searching orders by customer email returns within 2 seconds")
void searchingOrdersByEmailReturnsWithinTwoSeconds() {
List<Order> results = orderRepository.findByEmail("[email protected]");
assertNotNull(results);
// If this takes >2s, test fails with:
// org.opentest4j.AssertionFailedError: method timed out after 2 seconds
}
// Apply @Timeout at class level to set a default for ALL tests in the class
@Timeout(5) // 5 seconds default for every test
class AllSearchesMustBeFast {
@Test
void searchByEmailIsWithin5Seconds() { /* ... */ }
@Test
@Timeout(1) // override: this specific test must complete in 1 second
void searchByIdIsWithin1Second() { /* ... */ }
}
}
// Global timeout: set in junit-platform.properties
// junit.jupiter.execution.timeout.default=30s (safety net for all tests)
Level 2: assertTimeout — Inline Performance Bounds
import java.time.Duration;
import static org.junit.jupiter.api.Assertions.assertTimeout;
class DataProcessorPerformanceTest {
@Test
@Tag("performance")
@DisplayName("Processing 10,000 records completes within 500ms")
void processingTenThousandRecordsCompletesWithinFiveHundredMs() {
List<Record> tenThousandRecords = generateTestRecords(10_000);
// assertTimeout: runs the executable and fails if it takes longer than the duration
// The executable always runs to completion (unlike assertTimeoutPreemptively)
List<ProcessedRecord> results = assertTimeout(
Duration.ofMillis(500),
() -> dataProcessor.processAll(tenThousandRecords),
"Processing 10,000 records should complete within 500ms on any hardware"
);
// Can also assert on the returned result
assertEquals(10_000, results.size(), "All records must be processed");
}
@Test
@DisplayName("Sorting 100,000 integers is faster than Comparable baseline")
void customSorterFasterThanJavaBaseline() {
List<Integer> data = IntStream.range(0, 100_000)
.boxed().collect(Collectors.toList());
Collections.shuffle(data, new Random(42)); // reproducible input
// Measure Java built-in sort as baseline
long javaStart = System.nanoTime();
Collections.sort(new ArrayList<>(data));
long javaMs = (System.nanoTime() - javaStart) / 1_000_000;
// Measure custom sorter
long customStart = System.nanoTime();
customSorter.sort(new ArrayList<>(data));
long customMs = (System.nanoTime() - customStart) / 1_000_000;
assertTrue(customMs <= javaMs * 2,
"Custom sorter should not be more than 2x slower than Java's built-in sort. "
+ "Java: " + javaMs + "ms, Custom: " + customMs + "ms");
}
}
Level 3: JMH — Professional Micro-Benchmarking
For accurate, statistically sound performance measurements, use JMH (Java Microbenchmark Harness) — the tool used by the OpenJDK team itself.
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-core</artifactId>
<version>1.37</version>
</dependency>
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-generator-annprocess</artifactId>
<version>1.37</version>
</dependency>
import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.infra.Blackhole;
import java.util.concurrent.TimeUnit;
// JMH benchmark for OrderService.findByEmail() performance
@BenchmarkMode(Mode.AverageTime) // measure average time per operation
@OutputTimeUnit(TimeUnit.MICROSECONDS) // report in microseconds
@State(Scope.Thread) // each benchmark thread has its own state
@Warmup(iterations = 5, time = 1) // 5 warm-up iterations x 1 second each
@Measurement(iterations = 10, time = 1) // 10 measurement iterations
@Fork(2) // run in 2 separate JVM forks for independence
public class OrderSearchBenchmark {
private OrderRepository orderRepository;
private List<String> testEmails;
// @Setup: runs once before the benchmark to prepare state
@Setup
public void setUp() {
orderRepository = new InMemoryOrderRepository();
testEmails = generateTestEmails(1000);
seedTestOrders(orderRepository, testEmails);
}
// @Benchmark: the method to measure
@Benchmark
public void findOrderByEmail(Blackhole blackhole) {
// Blackhole: consumes the result to prevent JIT dead-code elimination
String email = testEmails.get(0);
List<Order> orders = orderRepository.findByEmail(email);
blackhole.consume(orders);
}
@Benchmark
public void findOrdersByStatusAndPage(Blackhole blackhole) {
List<Order> orders = orderRepository.findByStatus(OrderStatus.PENDING, 0, 20);
blackhole.consume(orders);
}
}
// Run from main or JUnit 6 test:
public class RunBenchmarks {
public static void main(String[] args) throws RunnerException {
Options options = new OptionsBuilder()
.include(OrderSearchBenchmark.class.getSimpleName())
.build();
new Runner(options).run();
}
}
JMH Output
Benchmark Mode Cnt Score Error Units
OrderSearchBenchmark.findOrderByEmail avgt 20 12.847 ± 0.234 us/op
OrderSearchBenchmark.findOrdersByStatusAndPage avgt 20 18.423 ± 0.891 us/op
Interpretation:
findByEmail: 12.847 microseconds average ± 0.234 us margin of error
Across 20 measurement iterations (2 forks × 10 iterations)
JVM fully warmed up, GC pressure accounted for
Integrating JMH Results as JUnit 6 Tests
import org.openjdk.jmh.results.RunResult;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;
import java.util.Collection;
// Run JMH benchmark FROM a JUnit 6 test and assert on results
@Tag("performance")
class OrderSearchPerformanceBenchmarkTest {
@Test
@Timeout(value = 5, unit = TimeUnit.MINUTES) // JMH needs time to warm up
@DisplayName("findByEmail benchmark: average time must be under 50 microseconds")
void findByEmailBenchmarkMeetsPerformanceTarget() throws Exception {
Options benchmarkOptions = new OptionsBuilder()
.include(OrderSearchBenchmark.class.getSimpleName() + ".findOrderByEmail")
.warmupIterations(3) // fewer iterations for CI speed
.measurementIterations(5)
.forks(1)
.build();
Collection<RunResult> results = new Runner(benchmarkOptions).run();
for (RunResult result : results) {
double averageTimeUs = result.getPrimaryResult().getScore();
// Assert: must be under 50 microseconds per call
assertTrue(averageTimeUs < 50.0,
"findByEmail average time " + averageTimeUs
+ " us exceeds 50 us threshold — performance regression detected!");
}
}
}
Frequently Asked Questions (FAQs)
Q1: Why is System.currentTimeMillis() not suitable for benchmarking?
Single-run timing with System.currentTimeMillis() produces wildly inaccurate results because: (1) the JVM starts with interpreted execution and JIT-compiles hot methods after a warm-up period — the first few runs are orders of magnitude slower than steady-state, (2) garbage collection pauses can add hundreds of milliseconds randomly, (3) OS scheduling can pause your thread unexpectedly. JMH handles all of these with warm-up iterations, multiple forks, and statistical analysis.
Q2: What is the Blackhole in JMH and why is it necessary?
The JIT compiler eliminates dead code — if you compute a result and never use it, the JIT may remove the computation entirely, making your benchmark measure near-zero time. Blackhole.consume(result) prevents this by telling JMH "this result is used" without actually doing anything with it. Always pass benchmark results to a Blackhole in JMH to ensure the code you intend to measure is actually executed.
Q3: Should I run JMH benchmarks in CI on every commit?
No — JMH benchmarks take minutes (warm-up + measurement iterations) and their results are sensitive to CI infrastructure variability. Run them in a dedicated scheduled nightly job on a consistent, lightly loaded machine. Store results in a database or spreadsheet to track trends over time. For CI, use @Timeout and assertTimeout for quick regression detection — they are fast enough for every commit pipeline.
Q4: How do I benchmark Spring Boot service methods?
The cleanest approach is to test the pure Java method in isolation (no Spring context) using JMH’s @Setup to create the service with mocked or in-memory dependencies. For database-involved benchmarks, use a real database (Testcontainers) and annotate with @State(Scope.Benchmark) so the DB connection is shared across iterations. Avoid loading the full Spring context in JMH benchmarks — context startup adds seconds of irrelevant overhead to every measurement.
Q5: What is the difference between throughput and average time in JMH?
Mode.Throughput measures operations per second — how many times your method can be called per second. Higher is better. Mode.AverageTime measures the average time per single operation — how many microseconds each call takes. Lower is better. Use AverageTime for latency-sensitive code (each call must be fast). Use Throughput for batch processing (overall capacity matters more than individual call latency).
See Also
- Parallel Test Execution in JUnit 6: Configuration and Pitfalls
- Why Your JUnit Tests Are Slow (Performance Optimization Guide)
- Running JUnit 6 Tests in CI/CD Pipelines
- Mutation Testing with PIT and JUnit 6
- JUnit 6 Tutorial: Complete Series Index
Conclusion
Performance testing in Java has three levels of precision. Use @Timeout for quick safety nets that catch catastrophic regressions in CI. Use assertTimeout for inline performance assertions that fail the build on obvious slowdowns. Use JMH for statistically rigorous micro-benchmarks of critical hot paths. Combine all three for a complete performance testing strategy that catches both sudden regressions and gradual degradation over time.
Next: AI-Powered Test Generation with JUnit 6 — explore how modern AI tools accelerate test writing while keeping you in control of quality.