Skip to main content

Advanced RAG with LangChain4j: Query Transformation, Routing and Easy RAG

Why RAG answers fail on follow-up questions and how LangChain4j fixes it: Easy RAG, query compression, expansion, routing, score thresholds, source-labelled injection and Result.sources(). Every number comes from a test against a real embedding model, including a chat-memory cost the docs don’t warn about.

You build a support bot over your docs. A customer asks, “What is the refund window for annual plans?” and the bot nails it. Then they type the obvious follow-up — “How long does it take to show up?” — and the bot answers with something about hardware returns, or deployments. The model is fine. The retrieval was handed a sentence that contains no nouns. That one failure is the reason “advanced RAG” exists, and LangChain4j ships a small, honest toolkit for it: query compression, query expansion, query routing, score thresholds, and a content injector you can reshape. This post walks them in the order you will meet the problems, starting from Easy RAG, and every number below comes from a test that ran against a real embedding model. Nothing here needs an API key: retrieval is real, and the LLM steps (the answerer, the query rewriter, the router) are scripted stand-ins so the runs are deterministic.
Versions tested. LangChain4j 1.23.0 (uploaded to Maven Central on 28 October 2026), langchain4j-easy-rag and the bundled quantized bge-small embeddings at 1.23.0-beta33 (these two have no GA release — only beta builds exist), JDK 25.0.5 (Temurin), JUnit 6.1.3. The companion module is advanced-rag in the langchain4j-demo repository.

RAG is six stages, and Easy RAG quietly fills in only two

RAG — retrieval-augmented generation — means: before the LLM answers, search your own documents and paste the best matches into the prompt. That is the whole idea. LangChain4j splits the search-and-paste step into stages so you can swap each one: transform the query, route it to a source, retrieve, aggregate the results, inject them into the prompt.
Six stages between the user and the LLM Usermessage 1 Transformrewrite query 2 Routepick sources 3 Retrievevector search 4 Aggregatemerge, rank 5 Injectadd to prompt 6 LLM added by youEasy RAG covers Blue stages are what Easy RAG gives you. Amber stages are identity no-ops until you plug something in. Every advanced technique in this post is a replacement for one amber box.
Read the diagram left to right. Easy RAG is LangChain4j’s name for the shortest path through it: load files, let the library chunk and embed them, and search with a vector similarity lookup. The amber stages do nothing special by default — the query goes through untouched. Everything in the rest of this post replaces one amber box. Here is Easy RAG’s ingestion, in full. It lives in KnowledgeBase.java. The interesting line is the last one: EmbeddingStoreIngestor.ingest(documents, store) takes no splitter and no embedding model. It finds a splitter through a service-provider file inside the langchain4j-easy-rag jar, and an embedding model through one inside the bundled bge-small jar — I confirmed both META-INF/services entries by listing the jars.
    private KnowledgeBase(String classpathDir) {
        List<Document> documents = ClassPathDocumentLoader.loadDocumentsRecursively(classpathDir);
        // The static ingest(documents, store) is the "Easy RAG" call: no splitter, no embedding model
        // configured here. It finds a splitter through a service-provider file in the easy-rag jar and an
        // embedding model through one in the bge-small jar.
        this.ingestion = EmbeddingStoreIngestor.ingest(documents, store);
    }
The corpus is six short files for a fictional product, “Acme Cloud”: three customer-support pages (refunds, hardware returns, account security) and three developer pages (rate limits, deployments, webhooks). Here is what a first question retrieves, and what the LLM actually receives, from the transcript 01-easy-rag.txt:
Easy RAG ingestion of kb/ (6 files)
  segments stored: 6

Retrieved for "How long are webhooks retried?":
  0.873  webhooks.txt
  0.820  refund-policy.txt
  0.811  account-security.txt

User message the LLM actually received:
How long are webhooks retried?

Answer using the following information:
Two things to notice. The retrieved text is simply appended to the user’s message after a fixed sentence — there is no magic channel. And the second and third results (a refund page and an account-security page, for a question about webhooks) are filler: the retriever returns its top three no matter how poor the matches are. Most of the rest of the post is about that.
The loader trap. ClassPathDocumentLoader.loadDocuments("kb") does not descend into sub-folders. With files in kb/support and kb/engineering it returns nothing, and the failure shows up one step later as an error that never mentions documents. The captured transcript 09-loader-trap.txt shows it:

loadDocuments("kb")            -> 0 documents
loadDocumentsRecursively("kb") -> 6 documents
ingest(flat, store)             -> java.lang.IllegalArgumentException: segments cannot be null or empty
Use loadDocumentsRecursively.
Going deeper: what Easy RAG decides for you

The splitter factory that langchain4j-easy-rag registers calls DocumentSplitters.recursive(300, 30, ...) — I confirmed the two constants by disassembling the jar, and the RAG tutorial states the same: segments of at most 300 tokens with a 30-token overlap. All six demo files are shorter than that, so each becomes exactly one segment (the test asserts 6 segments for 6 files). Real documents will split, and a split answer can end up half in one segment and half in the next — the overlap is there to soften that.

The default embedding model is a quantized bge-small-en-v1.5 that runs in-process through ONNX Runtime, so ingestion works offline. The tutorial describes it as roughly 24 MB; I did not measure that. It is a fine learning default and a reasonable first production default; the cost is that its similarity scores sit in a narrow band (everything in this post falls between about 0.67 and 0.92), which matters in the minScore section below.

The store is InMemoryEmbeddingStore, which the tutorial itself marks as a proof-of-concept choice. Swap it for pgvector, Qdrant, Chroma or similar before you ingest more than a toy corpus; nothing else in this post changes when you do.

Naive RAG fails the moment the user talks like a human

Naive RAG uses the user’s latest message, verbatim, as the search query. That works for a standalone question and falls apart for anything that leans on the conversation: “it”, “that plan”, “how long does it take”. An embedding model has no conversation to lean on. It embeds those few words and finds whatever is vaguely about durations. The test AdvancedRagTest.t02_naiveFollowUp runs both queries against the same store. The raw follow-up, then the same question written as a self-contained sentence (transcript: 02-naive-follow-up.txt):
Naive RAG searches with the follow-up exactly as typed:
  0.821  hardware-returns.txt
  0.817  deployments.txt
  0.816  refund-policy.txt

Searching with a self-contained rewrite instead ("How long does a refund for an annual plan take to show up on my card?"):
  0.918  refund-policy.txt
  0.857  hardware-returns.txt
  0.795  deployments.txt
Same corpus, same embedding model, two different queries Raw follow-up: “How long does it take to show up?” hardware-returns0.821 deployments0.817 refund-policy0.816 Rewritten: “How long does a refund for an annual plan take to show up on my card?” refund-policy0.918 hardware-returns0.857 deployments0.795 Green is the page that actually answers the question. The raw follow-up is a three-way tie it loses.
The diagram is the same data as the transcript, drawn so the argument is visible: the raw follow-up scores 0.821, 0.817 and 0.816 — a three-way tie between a hardware page, a deployments page and the correct refund page, with the correct one last. The self-contained rewrite scores the right page at 0.918 with a clear gap. The embedding model is not broken; it was asked a question with the subject missing.
Going deeper: why a bigger embedding model does not fix this

It is tempting to blame the embedding model and swap it. The failure is in the query, not the model: “How long does it take to show up?” contains no information about refunds, so no model, however good, can retrieve refund pages from it with confidence. The fix has to put the missing context back into the query — which is the next section.

Query compression: let an LLM rewrite the follow-up before searching

CompressingQueryTransformer takes the user’s new message plus the chat history, asks an LLM to turn them into one standalone query, and searches with that. The user’s own words still go to the answering LLM; only the search query changes. Wiring it up is one line on the retrieval augmentor, from Pipelines.java:
    public static RetrievalAugmentor compressing(ChatModel rewriter, ContentRetriever retriever) {
        return DefaultRetrievalAugmentor.builder()
                .queryTransformer(new CompressingQueryTransformer(rewriter))
                .contentRetriever(retriever)
                .build();
    }
The rewriter is any ChatModel; in the tests it is a scripted stand-in that returns the rewritten sentence from the previous section, so the demo is deterministic and the only real component is retrieval. Here is what happened across two turns (transcript 03-compression.txt):
Rewriter LLM calls after turn 1: 0
Rewriter LLM calls after turn 2: 1
Turn 1 made zero rewriter calls — there was no history to compress, and the transformer did not call the LLM. Turn 2 made one. So compression costs an extra LLM call only on follow-ups, which is where you need it.

The cost nobody mentions: the rewriter prompt grows every turn

Here is the prompt the rewriter received on turn 2. Look at the “Conversation” section:
Read and understand the conversation between the User and the AI. Then, analyze the new query from the User. Identify all relevant details, terms, and context from both the conversation and the new query. Reformulate this query into a clear, concise, and self-contained format suitable for information retrieval.

Conversation:
User: What is the refund window for annual plans?

Answer using the following information:
The transcript continues from there with about 1.5 KB of documentation: the whole retrieved text from turn 1 sits inside that “User:” line. That is because AI Services stores the augmented user message — question plus injected documents — in chat memory by default, and the compressing transformer reads that memory. Each turn therefore adds roughly 1.7 KB of documentation to every future rewriter prompt.
Size of the prompt sent to the query rewriter, in characters Default chat memoryturn 22256turn 33998turn 45748storeRetrievedContentInChatMemory(false)turn 2565turn 3616turn 4675
The diagram plots the rewriter-prompt size on turns 2, 3 and 4 from the same test: 2,256, 3,998 and 5,748 characters with the default, versus 565, 616 and 675 after one builder call. The figures are indicative — the scripted answerer replies with a few words where a real LLM would write paragraphs — but the direction is the point. The call is storeRetrievedContentInChatMemory(false) on the AI Service builder (in Pipelines.advancedLeanMemory). Memory then keeps only what the user typed; the answering LLM still receives the augmented message for the current turn, which the test asserts.
    public static SupportAssistant advancedLeanMemory(ChatModel answerModel, RetrievalAugmentor augmentor) {
        return AiServices.builder(SupportAssistant.class)
                .chatModel(answerModel)
                .chatMemory(MessageWindowChatMemory.withMaxMessages(10))
                .retrievalAugmentor(augmentor)
                .storeRetrievedContentInChatMemory(false)
                .build();
    }
Use lean memory with compression. The flag changes what the LLM sees on later turns, too: the model will not re-read old retrieved passages from history, so a follow-up that depends on a document quoted two turns ago has to get it back through a fresh retrieval. That is the trade the tutorial describes as keeping the history short, and it is why compression and lean memory belong together. I did not find the interaction spelled out in the tutorial; I noticed it by reading the rewriter prompt, not from a failing test.
Going deeper: the rewriter prompt and customising it

The default prompt asks for “only reformulated query and nothing else”, and the transformer’s own prompt warns not to prepend anything to the query — I did not test a chatty model, but a reply like “Sure! Here is the query: …” would be a poor search query, which is why the instruction is there. Use a small, instruction-following model for this step and consider a stricter PromptTemplate (the constructor and builder both accept one; it must use the same {{chatMemory}} and {{query}} variables as the default, which I printed from DEFAULT_PROMPT_TEMPLATE).

Rewriting is also a latency tax: a full LLM round trip sits in front of every follow-up search. The last section of this post puts a number on the calls per turn.

Query expansion: ask the question several ways

Compression fixes a missing subject. A different failure is wrong vocabulary: the user says “undo a bad release”, your docs say “rollback” and “deployment”. ExpandingQueryTransformer asks an LLM for N rewordings, searches with each, and merges the results with reciprocal rank fusion in the default content aggregator. The test builds two cases (transcript 04-expansion.txt). In the first, the correct document, deployments.txt, is not in the single-query top three at all:
Original query: I shipped a bad release, can I undo it?
Single-query top 3:
  0.807  hardware-returns.txt
  0.790  webhooks.txt
  0.780  refund-policy.txt
Each reworded query finds it. After fusion:
After DefaultContentAggregator (reciprocal rank fusion): 4 unique segments, in this order:
  deployments.txt
  refund-policy.txt
  hardware-returns.txt
  webhooks.txt
The right page moves from “invisible” to first. Two details are easy to miss. The result has four segments although maxResults was 3 — that limit applies to each search, not to the merged list, so expansion can hand the LLM more text than you configured. And in the second case, a question that was already fine, expansion changed nothing (same transcript, 04-expansion.txt):
Original query: How do I get my money back?
Single-query top 3:
  0.852  refund-policy.txt
  0.804  hardware-returns.txt
  0.763  webhooks.txt
Same three documents, in the same order, after paying for one extra LLM call and three embedding lookups. Expansion helps when wording mismatches; it is a tax when it does not.
An honest limit of this demo. The rewordings are scripted — I wrote them — so they show the mechanism, not what your LLM will produce. A real model might produce variants that miss as well. The embedding search, the scores and the fusion are real; the variants are the one invented part.
Going deeper: how the fusion works and what else can merge results

The tutorial describes DefaultContentAggregator as two-stage reciprocal rank fusion: results for each query are fused across retrievers, then across queries. Reciprocal rank fusion scores a document by how high it ranked in each list, not by its raw similarity score — which is why scores from different retrievers can be combined at all.

The alternative is ReRankingContentAggregator, which sends candidates to a scoring (re-ranking) model and keeps the best. It needs a scoring model such as a hosted re-ranker, so I did not run it here; the constructor shape (a ScoringModel, a query selector, a minimum score, a max-results cap) is from javap, not from a run.

Query routing: do not search the billing docs for a webhook question

Once you have more than one source — a support knowledge base and engineering docs, say — searching one merged store lets the wrong domain leak in as filler. Take “How long are webhooks retried?”. Searching the merged store gives the webhooks page and then two support pages. A LanguageModelQueryRouter asks an LLM which source fits and searches only that one. The wiring is in Pipelines.router:
    public static QueryRouter router(ChatModel routerModel, ContentRetriever support, ContentRetriever engineering) {
        Map<ContentRetriever, String> options = new LinkedHashMap<>();
        options.put(support, "Billing, refunds, hardware returns and account security for customers");
        options.put(engineering, "API rate limits, deployments and webhooks for developers");
        return LanguageModelQueryRouter.builder()
                .chatModel(routerModel)
                .retrieverToDescription(options)
                .build();
    }
Note the LinkedHashMap. The numbers the router LLM sees are assigned by iteration order, so a map without a guaranteed order (Map.of, for example) can renumber the options from one JVM run to the next. The transcript 05-routing.txt shows the prompt and the effect:
Query: How long are webhooks retried?

Searching the single merged store:
  0.873  webhooks.txt
  0.820  refund-policy.txt
  0.811  account-security.txt

Prompt LanguageModelQueryRouter sent to the router LLM:
Based on the user query, determine the most suitable data source(s) to retrieve relevant information from the following options:
1: Billing, refunds, hardware returns and account security for customers
2: API rate limits, deployments and webhooks for developers
It is very important that your answer consists of either a single number or multiple numbers separated by commas and nothing else!
User query: How long are webhooks retried?

Router LLM answered "2" -> retrievers chosen: 1
  0.873  webhooks.txt
  0.802  deployments.txt
  0.786  rate-limits.txt
Routing removed the two support pages from the results. It did not remove all weak matches — the engineering store still returns its top three. Routing narrows where you search; it does not decide whether anything matched. That is the next section’s job.

When the router LLM misbehaves

The router expects “a single number or multiple numbers separated by commas and nothing else”. A model that replies in prose breaks that contract, and what happens next is a configuration choice (from 05-routing.txt):
Router LLM answered "The engineering one, I think.":
  default fallback strategy -> retrievers chosen: 0
  ROUTE_TO_ALL -> retrievers chosen: 2
  FAIL -> java.lang.RuntimeException: java.lang.NumberFormatException: For input string: "The engineering one"
The default silently routes to no retriever, so the question goes to the LLM with no documents and no error — the worst failure mode for a support bot, because the answer still reads fluently. ROUTE_TO_ALL degrades to a merged search; FAIL throws a NumberFormatException wrapped in a RuntimeException. I would choose ROUTE_TO_ALL in production and log when it triggers.
Going deeper: routers you can write yourself

QueryRouter is a one-method interface: given a Query, return the retrievers to use. DefaultQueryRouter returns all of them, which is what you get when you never configure routing. A keyword or metadata-based router (route by tenant, by product, by language) is often more reliable than an LLM and costs no extra call; reach for LanguageModelQueryRouter when the decision is genuinely about meaning.

Routing and the transformer compose: with several queries and several retrievers, the augmentor fans out and runs the work in parallel on an executor (a cached thread pool with a one-second keep-alive, per the tutorial); with exactly one query and one retriever it stays on the calling thread.

What the defaults do not do: say “I have nothing for that”

The retriever’s defaults are maxResults = 3 and minScore = 0.0 (a test prints both constants into 11-defaults.txt). A minimum score of zero means every query gets three documents, however irrelevant. Ask something off-topic and watch (transcript 07-min-score.txt):
Off-topic: What is the capital of France?
minScore 0.0 (default):
  0.696  deployments.txt
  0.670  rate-limits.txt
  0.669  account-security.txt
minScore 0.75:
  (nothing)
A question about France’s capital pulls in deployments, rate limits and account security, and the LLM is told to “answer using the following information”. A minScore of 0.75 returns nothing, and then the user’s message goes to the LLM unchanged — the test asserts that the message equals the question exactly. Now the on-topic question:
On-topic: How long does a refund take to show up on my card?
minScore 0.0:
  0.906  refund-policy.txt
  0.868  hardware-returns.txt
  0.782  webhooks.txt
minScore 0.75:
  0.906  refund-policy.txt
  0.868  hardware-returns.txt
  0.782  webhooks.txt
minScore 0.85:
  0.906  refund-policy.txt
  0.868  hardware-returns.txt
In these runs the threshold separates the off-topic query (0.67–0.70) from the on-topic one cleanly, but at 0.75 the irrelevant webhooks page at 0.782 still gets through. Raising it to 0.85 removes it — and would have thrown away the correct deployments.txt in the expansion example earlier, where it scored 0.798, 0.858 and 0.788 across the three reworded searches. The scores are specific to this embedding model and this corpus. Tune the threshold on your own questions; do not copy mine.

Shaping what the LLM is told

The default injector appends your documents after “Answer using the following information:”. When you need the model to refuse instead of improvise, and to know where each passage came from, build your own injector. The test t06_injection does both. The result (transcript 06-injection.txt):
Custom template + metadataKeysToInclude(["file_name"]):
How long are webhooks retried?

Answer ONLY from the sources below. If they do not contain the answer, say you do not know.

content: Acme Cloud Webhooks

And the two metadata lines that follow the passages in that message:

file_name: webhooks.txt
file_name: refund-policy.txt
Each passage now arrives with file_name: webhooks.txt underneath it, courtesy of metadataKeysToInclude(["file_name"]), and the instruction is “answer only from the sources”. Easy RAG’s loader attached file_name for free. The template must keep {{userMessage}} and {{contents}}.

Showing your sources to the user

If the return type of the AI Service method is Result<String> instead of String, you get the retrieved content back with the answer. The interface is SourcedAssistant.java and the output (transcript 10-sources.txt):
answer: (scripted answer)
sources:
  0.873  webhooks.txt
  0.820  refund-policy.txt
Link those file names in your UI and users can check the answer themselves.
Going deeper: score thresholds, metadata filters and limits

A single global threshold is a blunt instrument. EmbeddingStoreContentRetriever accepts functions for maxResults, minScore and a metadata Filter that take the Query, so you can vary them per query (a stricter threshold for a high-stakes tenant, a filter by file_name or product). I confirmed the three DEFAULT_* function constants on the class; I did not exercise the filter here.

Streaming APIs have an equivalent to Result.sources(): the tutorial documents an onRetrieved callback on TokenStream. I did not run it against a streaming model.

All of it together, and what it costs

Compression, routing, a score floor, source-labelled injection and lean memory combine on one augmentor; the test t08_fullPipelineCost wires them and counts LLM calls per turn (transcript 08-full-pipeline.txt):
LLM calls per turn (compress + route + answer):
  turn 1  rewriter=0  router=1  answer=1
  turn 2  rewriter=1  router=1  answer=1
Two turns of conversation cost five LLM calls instead of the two a naive pipeline makes: one router call on every turn, plus a rewriter call from turn 2 onward. Each is a network round trip in front of the answer, so advanced RAG trades latency and money for accuracy. Spend them where the failures are: compression if users ask follow-ups, routing if you really have separate sources, expansion only if you measure vocabulary mismatches.
Should you build all this? Not at first. Start with Easy RAG plus a minScore you tuned on twenty real questions — that alone removes the most embarrassing answers. Add compression when you see follow-up failures in your logs, and routing when you have sources that genuinely do not belong together. Expansion is the one I would add last, and only after a measurement shows the right document sitting just outside your top results. Every stage you add is another LLM call that can itself be wrong.
Going deeper: running it yourself, and swapping in a real model

Clone the repository and run mvn test -pl advanced-rag -am with JDK 25; the tests regenerate every file in advanced-rag/output/, and they assert the same numbers quoted here, so a figure that stops being true fails the build. The first run downloads the bge-small model through Maven (it ships inside the jar).

To use a real LLM, replace each ScriptedChatModel with any ChatModel implementation — OpenAiChatModel, an Ollama model, and so on. None of the pipeline code changes. Expect the rewriter and router outputs, and therefore the exact scores, to differ from the transcripts.

Further reading

No Comments yet!

This site uses Akismet to reduce spam. Learn how your comment data is processed.