How to test an LLM application in Java with Spring AI's RelevancyEvaluator and FactCheckingEvaluator: what they send to the judge, why "Yes." fails, golden datasets, pass-rate gates and a free deterministic judge for CI.
The same 30,000 documents through Spring AI 2.0's VectorStore API into pgvector, Redis, Qdrant and Elasticsearch, all on library defaults: ingest time, recall@10 against exact search, latency, metadata filtering and running cost, and the one setting that fixes each surprise.
Thread.sleep in a test is a guess about timing, not a wait. This post replaces it with Awaitility's await() across a void @Async method, a real in-process @EmbeddedKafka listener, and a @Scheduled job -- plus the pom.xml trap where Spring Boot 4.1.1 split Kafka's autoconfiguration into its own module, leaving @KafkaListener silently unwired.
Testcontainers 2.0 bundles three independent changes under one version number: 61 artifacts renamed with a testcontainers- prefix, JUnit 4 support physically removed from the GenericContainer type hierarchy (not deprecated — verified by jar inspection, zero classes implement TestRule), and the default Docker API version bumped from 1.32 to 1.44, which is what actually fixes the real Docker Engine 29 compatibility break reported in testcontainers-java#11235. Includes a self-caught mistake: @EnabledIfDockerAvailable only works on the class, not the method, because TestcontainersExtension starts @Container fields in beforeAll before any method-level condition runs.
@DataJpaTest substitutes an embedded database by default, and Replace.NONE alone doesn't guarantee a real connection behind it — but the opposite trap is the real story here: @DataJpaTest DOES run your Flyway/Liquibase migrations by default, through an undocumented fifth meta-annotation that merges autoconfiguration imports across three separate jars. Built with no Docker daemon available, with an honest accounting of what ran for real versus what's verified from source.
RestTestClient is Spring Framework 7's one fluent API across four binding modes: bare controller, MockMvc, full context, and a real server. What moved into two new Spring Boot 4 test modules, the TestRestTemplate migration path, and a real, verified surprise about what "real HTTP" hides from you.
@MockBean is gone in Spring Boot 4, not deprecated. Here's what replaced it (@MockitoBean/@MockitoSpyBean, with real ApplicationContext cache hit/miss proof), plus how to fake an HTTP collaborator with WireMock or MockWebServer without Docker — including two real, captured dependency failures and the fix.
A real Kotlin + Spring Boot 4.1 module, built and run: the exact two BeanDefinitionParsingException failures a missing allopen/noarg compiler plugin produces, a JSpecify @NullMarked Java method that still NPEs because the contract isn't runtime-checked, and a suspend fun controller method proven to resume on kotlinx.coroutines.DefaultExecutor rather than a Tomcat thread.
One Widget REST API, built three times — Spring Boot 4.1.1, Quarkus 3.40.1, and Micronaut 5.2.2 — run in both JVM and GraalVM native mode, with real measured startup time, memory, throughput, artifact size and native build cost. Includes the four real native-image build failures it took to get Micronaut's build working, and the config file that silently got ignored.
A layered-architecture rule, a no-field-injection rule, and a no-cycles rule, all checked against real compiled bytecode in under two seconds with no Spring context anywhere -- plus a real captured violation, and a FreezingArchRule demonstration showing how to turn a rule on against a codebase that already fails it.