Versions tested. LangChain4j1.23.0(uploaded to Maven Central on 28 October 2026),langchain4j-easy-ragand the bundled quantized bge-small embeddings at1.23.0-beta33(these two have no GA release — only beta builds exist), JDK 25.0.5 (Temurin), JUnit 6.1.3. The companion module isadvanced-ragin the langchain4j-demo repository.
RAG is six stages, and Easy RAG quietly fills in only two
RAG — retrieval-augmented generation — means: before the LLM answers, search your own documents and paste the best matches into the prompt. That is the whole idea. LangChain4j splits the search-and-paste step into stages so you can swap each one: transform the query, route it to a source, retrieve, aggregate the results, inject them into the prompt.EmbeddingStoreIngestor.ingest(documents, store) takes no splitter and no embedding model. It finds a splitter through a service-provider file inside the langchain4j-easy-rag jar, and an embedding model through one inside the bundled bge-small jar — I confirmed both META-INF/services entries by listing the jars.
private KnowledgeBase(String classpathDir) {
List<Document> documents = ClassPathDocumentLoader.loadDocumentsRecursively(classpathDir);
// The static ingest(documents, store) is the "Easy RAG" call: no splitter, no embedding model
// configured here. It finds a splitter through a service-provider file in the easy-rag jar and an
// embedding model through one in the bge-small jar.
this.ingestion = EmbeddingStoreIngestor.ingest(documents, store);
}
The corpus is six short files for a fictional product, “Acme Cloud”: three customer-support pages (refunds, hardware returns, account security) and three developer pages (rate limits, deployments, webhooks). Here is what a first question retrieves, and what the LLM actually receives, from the transcript 01-easy-rag.txt:
Easy RAG ingestion of kb/ (6 files)
segments stored: 6
Retrieved for "How long are webhooks retried?":
0.873 webhooks.txt
0.820 refund-policy.txt
0.811 account-security.txt
User message the LLM actually received:
How long are webhooks retried?
Answer using the following information:
Two things to notice. The retrieved text is simply appended to the user’s message after a fixed sentence — there is no magic channel. And the second and third results (a refund page and an account-security page, for a question about webhooks) are filler: the retriever returns its top three no matter how poor the matches are. Most of the rest of the post is about that.
The loader trap.ClassPathDocumentLoader.loadDocuments("kb")does not descend into sub-folders. With files inkb/supportandkb/engineeringit returns nothing, and the failure shows up one step later as an error that never mentions documents. The captured transcript 09-loader-trap.txt shows it:UseloadDocuments("kb") -> 0 documents loadDocumentsRecursively("kb") -> 6 documents ingest(flat, store) -> java.lang.IllegalArgumentException: segments cannot be null or emptyloadDocumentsRecursively.
Going deeper: what Easy RAG decides for you
The splitter factory that langchain4j-easy-rag registers calls DocumentSplitters.recursive(300, 30, ...) — I confirmed the two constants by disassembling the jar, and the RAG tutorial states the same: segments of at most 300 tokens with a 30-token overlap. All six demo files are shorter than that, so each becomes exactly one segment (the test asserts 6 segments for 6 files). Real documents will split, and a split answer can end up half in one segment and half in the next — the overlap is there to soften that.
The default embedding model is a quantized bge-small-en-v1.5 that runs in-process through ONNX Runtime, so ingestion works offline. The tutorial describes it as roughly 24 MB; I did not measure that. It is a fine learning default and a reasonable first production default; the cost is that its similarity scores sit in a narrow band (everything in this post falls between about 0.67 and 0.92), which matters in the minScore section below.
The store is InMemoryEmbeddingStore, which the tutorial itself marks as a proof-of-concept choice. Swap it for pgvector, Qdrant, Chroma or similar before you ingest more than a toy corpus; nothing else in this post changes when you do.
Naive RAG fails the moment the user talks like a human
Naive RAG uses the user’s latest message, verbatim, as the search query. That works for a standalone question and falls apart for anything that leans on the conversation: “it”, “that plan”, “how long does it take”. An embedding model has no conversation to lean on. It embeds those few words and finds whatever is vaguely about durations. The test AdvancedRagTest.t02_naiveFollowUp runs both queries against the same store. The raw follow-up, then the same question written as a self-contained sentence (transcript: 02-naive-follow-up.txt):Naive RAG searches with the follow-up exactly as typed:
0.821 hardware-returns.txt
0.817 deployments.txt
0.816 refund-policy.txt
Searching with a self-contained rewrite instead ("How long does a refund for an annual plan take to show up on my card?"):
0.918 refund-policy.txt
0.857 hardware-returns.txt
0.795 deployments.txt
Going deeper: why a bigger embedding model does not fix this
It is tempting to blame the embedding model and swap it. The failure is in the query, not the model: “How long does it take to show up?” contains no information about refunds, so no model, however good, can retrieve refund pages from it with confidence. The fix has to put the missing context back into the query — which is the next section.
- Test: AdvancedRagTest.java
- Transcript: 02-naive-follow-up.txt
Query compression: let an LLM rewrite the follow-up before searching
CompressingQueryTransformer takes the user’s new message plus the chat history, asks an LLM to turn them into one standalone query, and searches with that. The user’s own words still go to the answering LLM; only the search query changes. Wiring it up is one line on the retrieval augmentor, from Pipelines.java:
public static RetrievalAugmentor compressing(ChatModel rewriter, ContentRetriever retriever) {
return DefaultRetrievalAugmentor.builder()
.queryTransformer(new CompressingQueryTransformer(rewriter))
.contentRetriever(retriever)
.build();
}
The rewriter is any ChatModel; in the tests it is a scripted stand-in that returns the rewritten sentence from the previous section, so the demo is deterministic and the only real component is retrieval. Here is what happened across two turns (transcript 03-compression.txt):
Rewriter LLM calls after turn 1: 0
Rewriter LLM calls after turn 2: 1
Turn 1 made zero rewriter calls — there was no history to compress, and the transformer did not call the LLM. Turn 2 made one. So compression costs an extra LLM call only on follow-ups, which is where you need it.
The cost nobody mentions: the rewriter prompt grows every turn
Here is the prompt the rewriter received on turn 2. Look at the “Conversation” section:Read and understand the conversation between the User and the AI. Then, analyze the new query from the User. Identify all relevant details, terms, and context from both the conversation and the new query. Reformulate this query into a clear, concise, and self-contained format suitable for information retrieval.
Conversation:
User: What is the refund window for annual plans?
Answer using the following information:
The transcript continues from there with about 1.5 KB of documentation: the whole retrieved text from turn 1 sits inside that “User:” line. That is because AI Services stores the augmented user message — question plus injected documents — in chat memory by default, and the compressing transformer reads that memory. Each turn therefore adds roughly 1.7 KB of documentation to every future rewriter prompt.
storeRetrievedContentInChatMemory(false) on the AI Service builder (in Pipelines.advancedLeanMemory). Memory then keeps only what the user typed; the answering LLM still receives the augmented message for the current turn, which the test asserts.
public static SupportAssistant advancedLeanMemory(ChatModel answerModel, RetrievalAugmentor augmentor) {
return AiServices.builder(SupportAssistant.class)
.chatModel(answerModel)
.chatMemory(MessageWindowChatMemory.withMaxMessages(10))
.retrievalAugmentor(augmentor)
.storeRetrievedContentInChatMemory(false)
.build();
}
Use lean memory with compression. The flag changes what the LLM sees on later turns, too: the model will not re-read old retrieved passages from history, so a follow-up that depends on a document quoted two turns ago has to get it back through a fresh retrieval. That is the trade the tutorial describes as keeping the history short, and it is why compression and lean memory belong together. I did not find the interaction spelled out in the tutorial; I noticed it by reading the rewriter prompt, not from a failing test.
Going deeper: the rewriter prompt and customising it
The default prompt asks for “only reformulated query and nothing else”, and the transformer’s own prompt warns not to prepend anything to the query — I did not test a chatty model, but a reply like “Sure! Here is the query: …” would be a poor search query, which is why the instruction is there. Use a small, instruction-following model for this step and consider a stricter PromptTemplate (the constructor and builder both accept one; it must use the same {{chatMemory}} and {{query}} variables as the default, which I printed from DEFAULT_PROMPT_TEMPLATE).
Rewriting is also a latency tax: a full LLM round trip sits in front of every follow-up search. The last section of this post puts a number on the calls per turn.
Query expansion: ask the question several ways
Compression fixes a missing subject. A different failure is wrong vocabulary: the user says “undo a bad release”, your docs say “rollback” and “deployment”.ExpandingQueryTransformer asks an LLM for N rewordings, searches with each, and merges the results with reciprocal rank fusion in the default content aggregator.
The test builds two cases (transcript 04-expansion.txt). In the first, the correct document, deployments.txt, is not in the single-query top three at all:
Original query: I shipped a bad release, can I undo it?
Single-query top 3:
0.807 hardware-returns.txt
0.790 webhooks.txt
0.780 refund-policy.txt
Each reworded query finds it. After fusion:
After DefaultContentAggregator (reciprocal rank fusion): 4 unique segments, in this order:
deployments.txt
refund-policy.txt
hardware-returns.txt
webhooks.txt
The right page moves from “invisible” to first. Two details are easy to miss. The result has four segments although maxResults was 3 — that limit applies to each search, not to the merged list, so expansion can hand the LLM more text than you configured. And in the second case, a question that was already fine, expansion changed nothing (same transcript, 04-expansion.txt):
Original query: How do I get my money back?
Single-query top 3:
0.852 refund-policy.txt
0.804 hardware-returns.txt
0.763 webhooks.txt
Same three documents, in the same order, after paying for one extra LLM call and three embedding lookups. Expansion helps when wording mismatches; it is a tax when it does not.
An honest limit of this demo. The rewordings are scripted — I wrote them — so they show the mechanism, not what your LLM will produce. A real model might produce variants that miss as well. The embedding search, the scores and the fusion are real; the variants are the one invented part.
Going deeper: how the fusion works and what else can merge results
The tutorial describes DefaultContentAggregator as two-stage reciprocal rank fusion: results for each query are fused across retrievers, then across queries. Reciprocal rank fusion scores a document by how high it ranked in each list, not by its raw similarity score — which is why scores from different retrievers can be combined at all.
The alternative is ReRankingContentAggregator, which sends candidates to a scoring (re-ranking) model and keeps the best. It needs a scoring model such as a hosted re-ranker, so I did not run it here; the constructor shape (a ScoringModel, a query selector, a minimum score, a max-results cap) is from javap, not from a run.
Query routing: do not search the billing docs for a webhook question
Once you have more than one source — a support knowledge base and engineering docs, say — searching one merged store lets the wrong domain leak in as filler. Take “How long are webhooks retried?”. Searching the merged store gives the webhooks page and then two support pages. ALanguageModelQueryRouter asks an LLM which source fits and searches only that one. The wiring is in Pipelines.router:
public static QueryRouter router(ChatModel routerModel, ContentRetriever support, ContentRetriever engineering) {
Map<ContentRetriever, String> options = new LinkedHashMap<>();
options.put(support, "Billing, refunds, hardware returns and account security for customers");
options.put(engineering, "API rate limits, deployments and webhooks for developers");
return LanguageModelQueryRouter.builder()
.chatModel(routerModel)
.retrieverToDescription(options)
.build();
}
Note the LinkedHashMap. The numbers the router LLM sees are assigned by iteration order, so a map without a guaranteed order (Map.of, for example) can renumber the options from one JVM run to the next. The transcript 05-routing.txt shows the prompt and the effect:
Query: How long are webhooks retried?
Searching the single merged store:
0.873 webhooks.txt
0.820 refund-policy.txt
0.811 account-security.txt
Prompt LanguageModelQueryRouter sent to the router LLM:
Based on the user query, determine the most suitable data source(s) to retrieve relevant information from the following options:
1: Billing, refunds, hardware returns and account security for customers
2: API rate limits, deployments and webhooks for developers
It is very important that your answer consists of either a single number or multiple numbers separated by commas and nothing else!
User query: How long are webhooks retried?
Router LLM answered "2" -> retrievers chosen: 1
0.873 webhooks.txt
0.802 deployments.txt
0.786 rate-limits.txt
Routing removed the two support pages from the results. It did not remove all weak matches — the engineering store still returns its top three. Routing narrows where you search; it does not decide whether anything matched. That is the next section’s job.
When the router LLM misbehaves
The router expects “a single number or multiple numbers separated by commas and nothing else”. A model that replies in prose breaks that contract, and what happens next is a configuration choice (from 05-routing.txt):Router LLM answered "The engineering one, I think.":
default fallback strategy -> retrievers chosen: 0
ROUTE_TO_ALL -> retrievers chosen: 2
FAIL -> java.lang.RuntimeException: java.lang.NumberFormatException: For input string: "The engineering one"
The default silently routes to no retriever, so the question goes to the LLM with no documents and no error — the worst failure mode for a support bot, because the answer still reads fluently. ROUTE_TO_ALL degrades to a merged search; FAIL throws a NumberFormatException wrapped in a RuntimeException. I would choose ROUTE_TO_ALL in production and log when it triggers.
Going deeper: routers you can write yourself
QueryRouter is a one-method interface: given a Query, return the retrievers to use. DefaultQueryRouter returns all of them, which is what you get when you never configure routing. A keyword or metadata-based router (route by tenant, by product, by language) is often more reliable than an LLM and costs no extra call; reach for LanguageModelQueryRouter when the decision is genuinely about meaning.
Routing and the transformer compose: with several queries and several retrievers, the augmentor fans out and runs the work in parallel on an executor (a cached thread pool with a one-second keep-alive, per the tutorial); with exactly one query and one retriever it stays on the calling thread.
What the defaults do not do: say “I have nothing for that”
The retriever’s defaults aremaxResults = 3 and minScore = 0.0 (a test prints both constants into 11-defaults.txt). A minimum score of zero means every query gets three documents, however irrelevant. Ask something off-topic and watch (transcript 07-min-score.txt):
Off-topic: What is the capital of France?
minScore 0.0 (default):
0.696 deployments.txt
0.670 rate-limits.txt
0.669 account-security.txt
minScore 0.75:
(nothing)
A question about France’s capital pulls in deployments, rate limits and account security, and the LLM is told to “answer using the following information”. A minScore of 0.75 returns nothing, and then the user’s message goes to the LLM unchanged — the test asserts that the message equals the question exactly. Now the on-topic question:
On-topic: How long does a refund take to show up on my card?
minScore 0.0:
0.906 refund-policy.txt
0.868 hardware-returns.txt
0.782 webhooks.txt
minScore 0.75:
0.906 refund-policy.txt
0.868 hardware-returns.txt
0.782 webhooks.txt
minScore 0.85:
0.906 refund-policy.txt
0.868 hardware-returns.txt
In these runs the threshold separates the off-topic query (0.67–0.70) from the on-topic one cleanly, but at 0.75 the irrelevant webhooks page at 0.782 still gets through. Raising it to 0.85 removes it — and would have thrown away the correct deployments.txt in the expansion example earlier, where it scored 0.798, 0.858 and 0.788 across the three reworded searches. The scores are specific to this embedding model and this corpus. Tune the threshold on your own questions; do not copy mine.
Shaping what the LLM is told
The default injector appends your documents after “Answer using the following information:”. When you need the model to refuse instead of improvise, and to know where each passage came from, build your own injector. The test t06_injection does both. The result (transcript 06-injection.txt):Custom template + metadataKeysToInclude(["file_name"]):
How long are webhooks retried?
Answer ONLY from the sources below. If they do not contain the answer, say you do not know.
content: Acme Cloud Webhooks
And the two metadata lines that follow the passages in that message:
file_name: webhooks.txt
file_name: refund-policy.txt
Each passage now arrives with file_name: webhooks.txt underneath it, courtesy of metadataKeysToInclude(["file_name"]), and the instruction is “answer only from the sources”. Easy RAG’s loader attached file_name for free. The template must keep {{userMessage}} and {{contents}}.
Showing your sources to the user
If the return type of the AI Service method isResult<String> instead of String, you get the retrieved content back with the answer. The interface is SourcedAssistant.java and the output (transcript 10-sources.txt):
answer: (scripted answer)
sources:
0.873 webhooks.txt
0.820 refund-policy.txt
Link those file names in your UI and users can check the answer themselves.
Going deeper: score thresholds, metadata filters and limits
A single global threshold is a blunt instrument. EmbeddingStoreContentRetriever accepts functions for maxResults, minScore and a metadata Filter that take the Query, so you can vary them per query (a stricter threshold for a high-stakes tenant, a filter by file_name or product). I confirmed the three DEFAULT_* function constants on the class; I did not exercise the filter here.
Streaming APIs have an equivalent to Result.sources(): the tutorial documents an onRetrieved callback on TokenStream. I did not run it against a streaming model.
- Test: t07_minScore
- Transcripts: 06-injection.txt, 07-min-score.txt
All of it together, and what it costs
Compression, routing, a score floor, source-labelled injection and lean memory combine on one augmentor; the test t08_fullPipelineCost wires them and counts LLM calls per turn (transcript 08-full-pipeline.txt):LLM calls per turn (compress + route + answer):
turn 1 rewriter=0 router=1 answer=1
turn 2 rewriter=1 router=1 answer=1
Two turns of conversation cost five LLM calls instead of the two a naive pipeline makes: one router call on every turn, plus a rewriter call from turn 2 onward. Each is a network round trip in front of the answer, so advanced RAG trades latency and money for accuracy. Spend them where the failures are: compression if users ask follow-ups, routing if you really have separate sources, expansion only if you measure vocabulary mismatches.
Should you build all this? Not at first. Start with Easy RAG plus a minScore you tuned on twenty real questions — that alone removes the most embarrassing answers. Add compression when you see follow-up failures in your logs, and routing when you have sources that genuinely do not belong together. Expansion is the one I would add last, and only after a measurement shows the right document sitting just outside your top results. Every stage you add is another LLM call that can itself be wrong.
Going deeper: running it yourself, and swapping in a real model
Clone the repository and run mvn test -pl advanced-rag -am with JDK 25; the tests regenerate every file in advanced-rag/output/, and they assert the same numbers quoted here, so a figure that stops being true fails the build. The first run downloads the bge-small model through Maven (it ships inside the jar).
To use a real LLM, replace each ScriptedChatModel with any ChatModel implementation — OpenAiChatModel, an Ollama model, and so on. None of the pipeline code changes. Expect the rewriter and router outputs, and therefore the exact scores, to differ from the transcripts.
- Repository: advanced-rag module
- Earlier in this series: AI Services with Spring Boot 4 and agentic workflows
Further reading
- Companion module: advanced-rag — tests, transcripts and the six-file knowledge base
- LangChain4j RAG tutorial — Easy RAG, naive and advanced RAG, chat-memory control, sources
- LangChain4j AI Services with Spring Boot 4
- Vector embeddings and semantic search in pure Java
- Spring AI vs LangChain4j in 2026
No Comments yet!