Test Reporting in JUnit 6: Allure vs Surefire vs Custom Reports

Raw JUnit XML output from Maven Surefire is functional but not human-friendly. Test reporting transforms those results into meaningful dashboards that developers, QA engineers, and product owners can navigate, filter, and act on. This guide compares the three most popular reporting options for JUnit 6 projects — Maven Surefire Reports, Allure Framework, and custom HTML reports — with complete setup, screenshots of what each produces, and guidance on which to choose.

Comparison at a Glance

	Maven Surefire Report	Allure Framework	Custom HTML
Setup effort	Minimal (plugin only)	Medium (annotation + CLI)	High (build it yourself)
Report quality	Basic	Rich, interactive	Fully custom
History/trends	No	Yes (with Allure server)	Optional (manual build)
Annotations in tests	None needed	`@Step`, `@Description`	Custom
CI integration	Direct (XML output)	Allure GitHub Action	Upload artifact
Best for	Quick feedback, simple builds	QA-facing dashboards, large suites	Branded internal portals

Option 1: Maven Surefire HTML Report

The simplest option. Maven Surefire generates XML reports automatically. Add the surefire-report plugin to generate an HTML summary:

<!-- Add to your reporting section in pom.xml -->
<reporting>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-report-plugin</artifactId>
            <version>3.2.5</version>
        </plugin>
    </plugins>
</reporting>

# Generate surefire HTML report after running tests
mvn surefire-report:report

# Or generate as part of the site lifecycle
mvn test site

# Output: target/site/surefire-report.html

Surefire Report Summary

Test Suite: OrderServiceTest
  Tests: 12  Errors: 0  Failures: 0  Skipped: 0  Success Rate: 100%  Time: 0.41s

Test Suite: OrderRepositoryIT
  Tests: 6   Errors: 0  Failures: 1  Skipped: 0  Success Rate: 83%   Time: 3.21s
  FAILED: findByEmail_withNullEmail_throwsException (0.04s)
    java.lang.AssertionError: Expected IllegalArgumentException but no exception was thrown

Option 2: Allure Framework (Recommended for Large Projects)

Allure produces beautiful, interactive reports with test history, trend graphs, and categorised failures. It is the industry standard for QA-facing test reporting.

<!-- Allure JUnit 5/6 integration -->
<dependency>
    <groupId>io.qameta.allure</groupId>
    <artifactId>allure-junit5</artifactId>
    <version>2.27.0</version>
    <scope>test</scope>
</dependency>

<!-- Allure Maven plugin for report generation -->
<plugin>
    <groupId>io.qameta.allure</groupId>
    <artifactId>allure-maven</artifactId>
    <version>2.13.0</version>
</plugin>

Add Allure annotations to your tests for richer reports:

import io.qameta.allure.*;
import org.junit.jupiter.api.*;
import static org.junit.jupiter.api.Assertions.*;

// Allure annotations enrich the test report with metadata and steps
@Feature("Order Management")    // groups tests by feature in the report
@Story("Order Creation")        // groups tests by user story
@DisplayName("OrderService — creation tests")
class OrderServiceAllureTest {

    @Test
    @Description("Verifies that a new order is saved with PENDING status and triggers a confirmation email")
    @Severity(SeverityLevel.CRITICAL)  // shown in report: BLOCKER/CRITICAL/NORMAL/MINOR
    @Tag("smoke")                      // also works as a JUnit 6 tag
    @DisplayName("Creating a valid order sets status to PENDING")
    void creatingValidOrderSetsPendingStatus() {

        // @Step: each step appears as a collapsible node in the Allure report
        Order order = createTestOrder("[email protected]", 99.99);
        OrderResult result = placeOrder(order);
        verifyOrderStatus(result, OrderStatus.PENDING);
    }

    @Step("Create test order for {email} with total {total}")
    private Order createTestOrder(String email, double total) {
        return new Order(null, email, total, null);
    }

    @Step("Place order through OrderService")
    private OrderResult placeOrder(Order order) {
        return orderService.place(order);
    }

    @Step("Verify order status is {expectedStatus}")
    private void verifyOrderStatus(OrderResult result, OrderStatus expectedStatus) {
        assertEquals(expectedStatus, result.getStatus(),
            "Order status should be " + expectedStatus);
    }

    @Test
    @Description("Verifies that orders with negative total are rejected immediately")
    @Severity(SeverityLevel.BLOCKER)
    @Issue("JIRA-1234")  // links directly to your issue tracker in the report
    @TmsLink("TC-567")   // links to test management system (e.g. TestRail)
    @DisplayName("Negative order total throws IllegalArgumentException")
    void negativeOrderTotalThrowsException() {
        IllegalArgumentException ex = assertThrows(
            IllegalArgumentException.class,
            () -> orderService.place(new Order(null, "[email protected]", -1.0, null))
        );
        assertTrue(ex.getMessage().contains("positive"));
    }
}

# Run tests (Allure listener collects data in target/allure-results/)
mvn test

# Generate and serve the Allure report (opens in browser)
mvn allure:serve

# Generate static HTML report
mvn allure:report
# Output: target/site/allure-maven-plugin/index.html

Allure in GitHub Actions

# .github/workflows/ci.yml (Allure report publishing)
      - name: Run tests
        run: mvn test --batch-mode

      # Upload raw Allure results as artifact
      - name: Upload Allure results
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: allure-results
          path: target/allure-results/

      # Generate and publish Allure report to GitHub Pages
      - name: Generate Allure report
        uses: simple-elf/allure-report-action@v1
        if: always()
        with:
          allure_results: target/allure-results
          allure_history: allure-history

      - name: Publish Allure report to GitHub Pages
        uses: peaceiris/actions-gh-pages@v3
        if: always()
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          publish_branch: gh-pages
          publish_dir: allure-history

Option 3: Custom HTML Report with JUnit Platform Listener

import org.junit.platform.launcher.TestExecutionListener;
import org.junit.platform.launcher.TestIdentifier;
import org.junit.platform.engine.TestExecutionResult;
import java.io.*;
import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;

/**
 * SimpleHtmlReportListener: generates a minimal but branded HTML report.
 * Register via META-INF/services/org.junit.platform.launcher.TestExecutionListener
 */
public class SimpleHtmlReportListener implements TestExecutionListener {

    private final StringBuilder htmlBuilder = new StringBuilder();
    private int passCount = 0, failCount = 0, skipCount = 0;

    @Override
    public void testPlanExecutionStarted(TestPlan testPlan) {
        htmlBuilder.append("<!DOCTYPE html><html><head>");
        htmlBuilder.append("<title>Test Report</title>");
        htmlBuilder.append("<style>body{font-family:Arial;margin:20px}");
        htmlBuilder.append(".pass{color:green}.fail{color:red}.skip{color:orange}</style>");
        htmlBuilder.append("</head><body>");
        htmlBuilder.append("<h1>JUnit 6 Test Report</h1>");
        htmlBuilder.append("<p>Generated: ").append(
            LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss"))
        ).append("</p><table border='1' cellpadding='5'>");
        htmlBuilder.append("<tr><th>Test</th><th>Status</th><th>Message</th></tr>");
    }

    @Override
    public void executionFinished(TestIdentifier id, TestExecutionResult result) {
        if (!id.isTest()) return;
        String status, cssClass, message = "";
        switch (result.getStatus()) {
            case SUCCESSFUL -> { status = "PASS"; cssClass = "pass"; passCount++; }
            case FAILED     -> {
                status = "FAIL"; cssClass = "fail"; failCount++;
                message = result.getThrowable().map(Throwable::getMessage).orElse("");
            }
            default         -> { status = "SKIP"; cssClass = "skip"; skipCount++; }
        }
        htmlBuilder.append("<tr><td>").append(id.getDisplayName())
            .append("</td><td class='").append(cssClass).append("'>").append(status)
            .append("</td><td>").append(message).append("</td></tr>");
    }

    @Override
    public void testPlanExecutionFinished(TestPlan testPlan) {
        htmlBuilder.append("</table>");
        htmlBuilder.append("<h2>Summary: <span class='pass'>").append(passCount).append(" passed</span>, ")
            .append("<span class='fail'>").append(failCount).append(" failed</span>, ")
            .append("<span class='skip'>").append(skipCount).append(" skipped</span></h2>");
        htmlBuilder.append("</body></html>");
        try (PrintWriter writer = new PrintWriter("target/custom-test-report.html")) {
            writer.write(htmlBuilder.toString());
        } catch (IOException e) {
            System.err.println("Failed to write custom test report: " + e.getMessage());
        }
    }
}

Frequently Asked Questions (FAQs)

Q1: Which report format should I use for CI/CD integration?

For GitHub Actions, use the JUnit XML format (produced by Surefire automatically) with the publish-unit-test-result-action to show test results as PR checks. For Jenkins, use the built-in junit step with the XML files. For stakeholder-facing dashboards with history and trends, add Allure on top. The two are complementary — XML for CI gates, Allure for human dashboards.

Q2: Do Allure annotations affect test performance?

Negligibly. Allure’s listener writes small JSON files for each test result to target/allure-results/. The overhead is microseconds per test. The report generation step (allure:report) happens after tests complete and does not affect test execution time. The only real cost is the additional disk I/O for writing result files, which is imperceptible.

Q3: Can I filter the Allure report by feature or story?

Yes. Allure’s interactive report has a left-panel navigation with Features, Stories, Suites, Packages, and Tags views. Tests annotated with @Feature and @Story are grouped and filterable. This is especially valuable when a large suite has hundreds of tests — stakeholders can navigate to the features they own and see their test status without understanding the entire suite structure.

Q4: How do I attach screenshots or logs to Allure reports?

Use Allure.addAttachment("name", content) or the @Attachment annotation on a method that returns a byte[] or String. Common uses: attach a screenshot on UI test failure, attach a JSON request/response on API test failure, or attach a log excerpt when a test fails. These attachments appear inline in the Allure report for the specific failing test.

Q5: Can I use both Allure and Surefire XML reports in the same project?

Yes — they are completely independent and do not conflict. Surefire generates XML in target/surefire-reports/ automatically. Allure’s JUnit 5/6 listener generates JSON in target/allure-results/ separately. Both can be consumed by their respective CI tools simultaneously. This is the recommended setup: Surefire XML for CI gates (GitHub Actions checks, Jenkins build status), Allure HTML for human-readable trend reporting.

Conclusion

The right reporting tool depends on your audience. Maven Surefire reports are instant and zero-effort — perfect for developer feedback during development. Allure transforms your test results into a stakeholder-ready dashboard with trends, feature grouping, step-level detail, and issue tracker integration. Custom HTML listeners give you full control when neither standard option fits your requirements. Start with Surefire, add Allure when you need richer reporting.

Next: Unit vs Integration vs E2E Testing in JUnit 6 — understand the precise definition, purpose, and tradeoffs of each test type, and when to write each kind.

@ankurm