Performance Testing Using JUnit 6 (Benchmarks & Techniques)

JUnit 6 is excellent for verifying correctness, but measuring how fast your code runs requires a different approach. A naive long start = System.currentTimeMillis() before and after a method call is not a reliable benchmark — JVM warm-up, JIT compilation, and garbage collection all introduce noise that makes single-run timing meaningless. This guide covers professional performance testing techniques for Java: from JUnit 6’s @Timeout for basic bounds, to full JMH micro-benchmarks integrated into a Maven build.

Three Levels of Performance Testing

LevelToolPurposeAccuracy
Basic boundsJUnit 6 @TimeoutFail if a test exceeds a time limitLow — single cold run
Regression detectionJUnit 6 + assertTimeoutCatch performance regressions in CIMedium
Micro-benchmarkingJMH (Java Microbenchmark Harness)Precise throughput and latency measurementHigh — warmed JVM, statistics

Level 1: @Timeout — Simple Time Bounds

Use @Timeout to fail a test if it takes too long. This catches catastrophic performance regressions (infinite loops, accidental N+1 queries, missing index):

import org.junit.jupiter.api.Timeout;
import java.util.concurrent.TimeUnit;

class OrderSearchPerformanceTest {

    // @Timeout: test fails if it takes longer than the specified duration
    // Applied at method level
    @Test
    @Timeout(value = 2, unit = TimeUnit.SECONDS)
    @Tag("performance")
    @DisplayName("Searching orders by customer email returns within 2 seconds")
    void searchingOrdersByEmailReturnsWithinTwoSeconds() {
        List<Order> results = orderRepository.findByEmail("[email protected]");
        assertNotNull(results);
        // If this takes >2s, test fails with:
        // org.opentest4j.AssertionFailedError: method timed out after 2 seconds
    }

    // Apply @Timeout at class level to set a default for ALL tests in the class
    @Timeout(5) // 5 seconds default for every test
    class AllSearchesMustBeFast {

        @Test
        void searchByEmailIsWithin5Seconds() { /* ... */ }

        @Test
        @Timeout(1) // override: this specific test must complete in 1 second
        void searchByIdIsWithin1Second() { /* ... */ }
    }
}

// Global timeout: set in junit-platform.properties
// junit.jupiter.execution.timeout.default=30s  (safety net for all tests)

Level 2: assertTimeout — Inline Performance Bounds

import java.time.Duration;
import static org.junit.jupiter.api.Assertions.assertTimeout;

class DataProcessorPerformanceTest {

    @Test
    @Tag("performance")
    @DisplayName("Processing 10,000 records completes within 500ms")
    void processingTenThousandRecordsCompletesWithinFiveHundredMs() {
        List<Record> tenThousandRecords = generateTestRecords(10_000);

        // assertTimeout: runs the executable and fails if it takes longer than the duration
        // The executable always runs to completion (unlike assertTimeoutPreemptively)
        List<ProcessedRecord> results = assertTimeout(
            Duration.ofMillis(500),
            () -> dataProcessor.processAll(tenThousandRecords),
            "Processing 10,000 records should complete within 500ms on any hardware"
        );

        // Can also assert on the returned result
        assertEquals(10_000, results.size(), "All records must be processed");
    }

    @Test
    @DisplayName("Sorting 100,000 integers is faster than Comparable baseline")
    void customSorterFasterThanJavaBaseline() {
        List<Integer> data = IntStream.range(0, 100_000)
            .boxed().collect(Collectors.toList());
        Collections.shuffle(data, new Random(42)); // reproducible input

        // Measure Java built-in sort as baseline
        long javaStart = System.nanoTime();
        Collections.sort(new ArrayList<>(data));
        long javaMs = (System.nanoTime() - javaStart) / 1_000_000;

        // Measure custom sorter
        long customStart = System.nanoTime();
        customSorter.sort(new ArrayList<>(data));
        long customMs = (System.nanoTime() - customStart) / 1_000_000;

        assertTrue(customMs <= javaMs * 2,
            "Custom sorter should not be more than 2x slower than Java's built-in sort. "
            + "Java: " + javaMs + "ms, Custom: " + customMs + "ms");
    }
}

Level 3: JMH — Professional Micro-Benchmarking

For accurate, statistically sound performance measurements, use JMH (Java Microbenchmark Harness) — the tool used by the OpenJDK team itself.

<dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-core</artifactId>
    <version>1.37</version>
</dependency>
<dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-generator-annprocess</artifactId>
    <version>1.37</version>
</dependency>
import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.infra.Blackhole;
import java.util.concurrent.TimeUnit;

// JMH benchmark for OrderService.findByEmail() performance
@BenchmarkMode(Mode.AverageTime)   // measure average time per operation
@OutputTimeUnit(TimeUnit.MICROSECONDS)  // report in microseconds
@State(Scope.Thread)               // each benchmark thread has its own state
@Warmup(iterations = 5, time = 1)  // 5 warm-up iterations x 1 second each
@Measurement(iterations = 10, time = 1) // 10 measurement iterations
@Fork(2)                           // run in 2 separate JVM forks for independence
public class OrderSearchBenchmark {

    private OrderRepository orderRepository;
    private List<String> testEmails;

    // @Setup: runs once before the benchmark to prepare state
    @Setup
    public void setUp() {
        orderRepository = new InMemoryOrderRepository();
        testEmails      = generateTestEmails(1000);
        seedTestOrders(orderRepository, testEmails);
    }

    // @Benchmark: the method to measure
    @Benchmark
    public void findOrderByEmail(Blackhole blackhole) {
        // Blackhole: consumes the result to prevent JIT dead-code elimination
        String email  = testEmails.get(0);
        List<Order> orders = orderRepository.findByEmail(email);
        blackhole.consume(orders);
    }

    @Benchmark
    public void findOrdersByStatusAndPage(Blackhole blackhole) {
        List<Order> orders = orderRepository.findByStatus(OrderStatus.PENDING, 0, 20);
        blackhole.consume(orders);
    }
}

// Run from main or JUnit 6 test:
public class RunBenchmarks {
    public static void main(String[] args) throws RunnerException {
        Options options = new OptionsBuilder()
            .include(OrderSearchBenchmark.class.getSimpleName())
            .build();
        new Runner(options).run();
    }
}

JMH Output

Benchmark                                  Mode   Cnt    Score   Error  Units
OrderSearchBenchmark.findOrderByEmail      avgt    20   12.847 ± 0.234  us/op
OrderSearchBenchmark.findOrdersByStatusAndPage avgt  20   18.423 ± 0.891  us/op

Interpretation:
  findByEmail: 12.847 microseconds average ± 0.234 us margin of error
  Across 20 measurement iterations (2 forks × 10 iterations)
  JVM fully warmed up, GC pressure accounted for

Integrating JMH Results as JUnit 6 Tests

import org.openjdk.jmh.results.RunResult;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;
import java.util.Collection;

// Run JMH benchmark FROM a JUnit 6 test and assert on results
@Tag("performance")
class OrderSearchPerformanceBenchmarkTest {

    @Test
    @Timeout(value = 5, unit = TimeUnit.MINUTES) // JMH needs time to warm up
    @DisplayName("findByEmail benchmark: average time must be under 50 microseconds")
    void findByEmailBenchmarkMeetsPerformanceTarget() throws Exception {
        Options benchmarkOptions = new OptionsBuilder()
            .include(OrderSearchBenchmark.class.getSimpleName() + ".findOrderByEmail")
            .warmupIterations(3)   // fewer iterations for CI speed
            .measurementIterations(5)
            .forks(1)
            .build();

        Collection<RunResult> results = new Runner(benchmarkOptions).run();

        for (RunResult result : results) {
            double averageTimeUs = result.getPrimaryResult().getScore();
            // Assert: must be under 50 microseconds per call
            assertTrue(averageTimeUs < 50.0,
                "findByEmail average time " + averageTimeUs
                + " us exceeds 50 us threshold — performance regression detected!");
        }
    }
}

Frequently Asked Questions (FAQs)

Q1: Why is System.currentTimeMillis() not suitable for benchmarking?

Single-run timing with System.currentTimeMillis() produces wildly inaccurate results because: (1) the JVM starts with interpreted execution and JIT-compiles hot methods after a warm-up period — the first few runs are orders of magnitude slower than steady-state, (2) garbage collection pauses can add hundreds of milliseconds randomly, (3) OS scheduling can pause your thread unexpectedly. JMH handles all of these with warm-up iterations, multiple forks, and statistical analysis.

Q2: What is the Blackhole in JMH and why is it necessary?

The JIT compiler eliminates dead code — if you compute a result and never use it, the JIT may remove the computation entirely, making your benchmark measure near-zero time. Blackhole.consume(result) prevents this by telling JMH "this result is used" without actually doing anything with it. Always pass benchmark results to a Blackhole in JMH to ensure the code you intend to measure is actually executed.

Q3: Should I run JMH benchmarks in CI on every commit?

No — JMH benchmarks take minutes (warm-up + measurement iterations) and their results are sensitive to CI infrastructure variability. Run them in a dedicated scheduled nightly job on a consistent, lightly loaded machine. Store results in a database or spreadsheet to track trends over time. For CI, use @Timeout and assertTimeout for quick regression detection — they are fast enough for every commit pipeline.

Q4: How do I benchmark Spring Boot service methods?

The cleanest approach is to test the pure Java method in isolation (no Spring context) using JMH’s @Setup to create the service with mocked or in-memory dependencies. For database-involved benchmarks, use a real database (Testcontainers) and annotate with @State(Scope.Benchmark) so the DB connection is shared across iterations. Avoid loading the full Spring context in JMH benchmarks — context startup adds seconds of irrelevant overhead to every measurement.

Q5: What is the difference between throughput and average time in JMH?

Mode.Throughput measures operations per second — how many times your method can be called per second. Higher is better. Mode.AverageTime measures the average time per single operation — how many microseconds each call takes. Lower is better. Use AverageTime for latency-sensitive code (each call must be fast). Use Throughput for batch processing (overall capacity matters more than individual call latency).

See Also

Conclusion

Performance testing in Java has three levels of precision. Use @Timeout for quick safety nets that catch catastrophic regressions in CI. Use assertTimeout for inline performance assertions that fail the build on obvious slowdowns. Use JMH for statistically rigorous micro-benchmarks of critical hot paths. Combine all three for a complete performance testing strategy that catches both sudden regressions and gradual degradation over time.

Next: AI-Powered Test Generation with JUnit 6 — explore how modern AI tools accelerate test writing while keeping you in control of quality.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.