Reranking RAG Results in Java: Cross-Encoder Reranker with ONNX Runtime Before the LLM
Add a cross-encoder reranker to a Spring AI RAG pipeline: ONNX Runtime and DJL tokenizers in Java, a DocumentPostProcessor, and measured quality (hit@1 41 to 49 of 60) against latency (about 10 ms per candidate).
Your RAG application retrieved the right passage and then never showed it to the model. It was ranked twelfth, the prompt only has room for five, and nothing in the logs says anything went wrong. This is the most common quiet failure in retrieval-augmented generation, and the usual fix is a reranker: a second, slower and more careful pass over a shortlist of candidates.
This page builds one in Java. A small cross-encoder model runs inside the JVM through ONNX Runtime, sits behind Spring AI’s DocumentPostProcessor, and is measured on 60 labelled questions: what it gains, what it costs in milliseconds, and where it makes answers worse. The short version of the results is that it moved the right passage to first place for 49 questions instead of 41 and cost about ten milliseconds per candidate on a two-core machine — and that almost all of the gain came from one kind of question.
Tested with: Spring Boot 4.1.2 (the current GA; 4.2 is still at release-candidate stage), Spring AI 2.0.2, ONNX Runtime 1.31.0, DJL Hugging Face tokenizers 0.39.0, JDK 25.0.5 (Temurin), Maven 3.9.11. Models: sentence-transformers/all-MiniLM-L6-v2 for retrieval and cross-encoder/ms-marco-MiniLM-L-6-v2 for reranking. Versions from the Maven Central directory index on 2026-10-24. All code is in spring-ai/rerank; scripts/run-all.sh regenerates every output/NN-*.txt file quoted below. No database, no API key, no GPU.
If you are new to the idea, read in order: the first sections explain what a reranker is and why it needs a different kind of model, the middle sections build and wire it, and the last sections measure it and list the ways it goes wrong. Deeper material sits in collapsible blocks you can open when you want it.
A passage you retrieved is not a passage the model sees
Retrieval in a RAG pipeline works in two steps. Every document is turned into a list of numbers (an embedding) ahead of time. When a question arrives it is turned into numbers the same way, and the documents whose numbers are closest are returned, best first. You then paste the first few into the prompt. The word that matters is few: a prompt has limited room and every extra passage costs money and can distract the model, so five is a common choice.
Embedding similarity is a good way to find roughly related text quickly, but it is a blunt way to decide which of those is the answer. A reranker adds a second step. Ask the retriever for a generous shortlist, 50 passages say, then let a slower model that reads the question and each passage together put the shortlist in a better order, and keep only the first five.
The diagram shows where the new step sits and why it only looks at a shortlist: its cost grows with the number of candidates, while the retriever’s does not. Here is the effect on a real question from the test set, “How do I fix E-4129?”. The passage that answers it came out of the retriever in twelfth place. After reranking it is first. The full table is in output/08-post-processor-in-the-advisor.txt:
kept id retrieval rank rerank score text
1 E-4129 12 3.52 Error E-4129 - API key lacks permission. The key
2 E-4128 2 2.21 Error E-4128 - Tax identifier invalid. The tax r
3 E-4122 1 1.48 Error E-4122 - Payment method not attached. The
4 E-4123 14 1.09 Error E-4123 - Duplicate customer email. A custo
5 E-4126 5 0.94 Error E-4126 - Coupon expired. The coupon passed
The middle column is where each kept passage stood before the reranker. Without the reranker, the answer (at rank 12) would have been cut off, and the model would have been handed five error codes that are not the one the user asked about.
A cross-encoder reads the question and the passage together
The model that does the first step is called a bi-encoder: it turns the question into one vector and each passage into another, separately, and the two are compared afterwards. That separation is what makes it fast, because the passage vectors can be computed once and stored. It is also its weakness: the model never sees the question and the passage side by side.
A cross-encoder feeds both texts into the model as a single input and reads a relevance score off the end. It can notice that the same error code appears in both. The price is that nothing can be computed ahead of time: every (question, passage) pair is a fresh run of the model.
The left panel is why retrieval is cheap and the right panel is why reranking is not. In code, the pair is one token sequence. The model expects the question first, then a separator, then the passage; the token_type_ids row tells it which tokens belong to which (0 for the question, 1 for the passage). This is the real encoding produced by the code in CrossEncoder.java for an example pair, written by PairEncodingTest.java to output/01-pair-encoding.txt:
Encoding e = ce.encode("What is E-4126?", "Error E-4126 - Refund window closed. Refunds can be issued for 120 days.");
t.line("tokens : %s", Arrays.toString(e.getTokens()));
t.line("token_type_ids: %s", Arrays.toString(e.getTypeIds()));
t.line("attention_mask: %s", Arrays.toString(e.getAttentionMask()));
Note how the error code E-4126 became the pieces e, -, 412 and ##6. The tokenizer splits unfamiliar strings into word pieces, and that is what lets a cross-encoder line up the same code in the question and the passage.
How the first-stage vectors are made in this module
The retriever uses all-MiniLM-L6-v2, which returns one vector per token. The sentence-transformers recipe is to average those token vectors, skipping padding, and scale the result to length 1; then cosine similarity is a dot product. BiEncoder.java does exactly that and BiEncoderRetriever.java ranks 94 stored vectors against the question, in memory, because a vector store would only add a database to a module about something else.
Averaging the token vectors:
for (int t = 0; t < hidden[i].length; t++) {
if (mask[i][t] == 1) {
count++;
for (int d = 0; d < dim; d++) {
v[d] += hidden[i][t][d];
}
}
}
double norm = 0;
for (int d = 0; d < dim; d++) {
v[d] /= count;
norm += v[d] * v[d];
}
norm = Math.sqrt(norm);
for (int d = 0; d < dim; d++) {
v[d] /= (float) norm;
If you already use a vector store through Spring AI, nothing here changes: any DocumentRetriever can feed the reranker. The chunking choices that decide what a “passage” is are covered in Spring AI RAG ingestion.
Score eight candidates and watch the order change
This is the smallest end-to-end example. The bi-encoder returns its top eight for “Meaning of E-4133”; the cross-encoder scores each of the eight against the question. The cosine column is the retriever’s score and the last column is the cross-encoder’s raw output, from output/02-one-question-by-hand.txt:
The answer was fifth by cosine, between four near-identical error codes whose texts differ in a few characters. The cross-encoder gives it the highest score. Two things to notice. First, the cross-encoder’s numbers are logits: any real number, higher means more relevant. They are not probabilities and they are only meaningful for ordering passages for the same question; the traps section shows what goes wrong if you treat them otherwise. Second, the retriever’s scores are all within 0.08 of each other, which is how an answer in fifth place ends up looking almost the same as one in first.
Running the model from Java
A cross-encoder is a file of model weights in the ONNX format and a tokenizer description. ONNX Runtime executes the file; the Hugging Face tokenizers library, wrapped by DJL, turns text into the integer ids the model expects. Together they are about 120 lines of Java in CrossEncoder.java. The scoring method splits the candidates into batches, tokenises each batch as (question, passage) pairs, and returns one number per passage:
public float[] score(String query, List<String> passages) {
float[] out = new float[passages.size()];
for (int from = 0; from < passages.size(); from += batchSize) {
int to = Math.min(from + batchSize, passages.size());
PairList<String, String> pairs = new PairList<>();
for (String p : passages.subList(from, to)) {
pairs.add(query, p);
}
Encoding[] enc = tokenizer.batchEncode(pairs);
float[] part = run(enc);
System.arraycopy(part, 0, out, from, part.length);
}
return out;
The batch is padded to its longest member so that it forms a rectangle, then handed to the model as three arrays: the token ids, a mask saying which positions are real tokens and not padding, and the pair marker from the previous section. The single output per row is the logit:
Map<String, OnnxTensor> inputs = new HashMap<>();
try {
inputs.put("input_ids", OnnxTensor.createTensor(env, ids));
inputs.put("attention_mask", OnnxTensor.createTensor(env, mask));
if (sendTypeIds) {
inputs.put("token_type_ids", OnnxTensor.createTensor(env, types));
}
try (OrtSession.Result r = session.run(inputs)) {
float[][] logits = (float[][]) r.get(0).getValue();
float[] out = new float[n];
for (int i = 0; i < n; i++) {
out[i] = logits[i][0];
}
return out;
}
} catch (OrtException e) {
throw new IllegalStateException(e.getMessage(), e);
Both models are public repositories on Hugging Face; Models.java downloads them on the first run (about 200 MB in total) and caches them under ~/.cache/rerank.
Why this module does not use spring-ai-transformers
Spring AI has a spring-ai-transformers module with a TransformersEmbeddingModel, and the hybrid-search module in the same repository uses it. For this module I tried it first and dropped it. In 2.0.2 it brings ONNX Runtime 1.21.1 and DJL 0.36.0, both older than the current releases (1.31.0 and 0.39.0), and when I ran its embedding model it tried to download a PyTorch native library from the network (the failure came from DJL’s PyTorch library loader), which is more than a retrieval stage that only needs ONNX Runtime should require. A reranker needs neither: ONNX Runtime and the tokenizers are enough for both stages, and the module pins the newer versions directly in its pom.xml. The cost is that I wrote the 100-line BiEncoder myself. If you already depend on Spring AI’s embedding models for retrieval, keep them; the reranker does not care where the candidates came from.
The same running-ONNX-from-Java idea, applied to a sentiment classifier, is in Running ONNX Models in Java with ONNX Runtime and DJL.
Padding, truncation and what the tokenizer is told
The tokenizer is built with truncation on, padding on and a maximum length of 256 tokens. Padding to the longest member of the batch, not to 256, matters for speed: short passages do not pay for long ones. Truncation matters for correctness: a pair longer than the maximum is cut, and the cut can remove the very sentence that answers the question. The corpus in this module has no passage longer than 256 tokens (the traps section checks), but real documents often do, which is a chunking problem to solve before the reranker. TrapsTest.java reproduces the damage.
Plugging it into Spring AI
Spring AI’s RetrievalAugmentationAdvisor already has a slot for this. After the retriever returns documents and before they are placed in the prompt, it passes them through any DocumentPostProcessor you register. The interface has one method, from a question and a list of documents to a list of documents. The reranker implements it: score every document, sort by score, keep the top N, and record both the old and new ranks in each document’s metadata so the effect can be inspected later. From RerankingPostProcessor.java:
public List<Document> process(Query query, List<Document> documents) {
if (documents.isEmpty()) {
return documents;
}
float[] scores = crossEncoder.score(query.text(), documents.stream().map(Document::getText).toList());
List<Integer> order = new ArrayList<>();
for (int i = 0; i < scores.length; i++) {
order.add(i);
}
order.sort(Comparator.comparingDouble((Integer i) -> -scores[i]));
List<Document> out = new ArrayList<>();
for (int i : order.subList(0, Math.min(topN, order.size()))) {
Document d = documents.get(i);
Map<String, Object> meta = new HashMap<>(d.getMetadata());
meta.put("retrieval_score", d.getScore());
meta.put("rerank_score", scores[i]);
out.add(d.mutate().metadata(meta).score((double) scores[i]).build());
}
return out;
Wiring it is one extra builder call. The retriever is asked for 50; the post-processor cuts that to 5. From PostProcessorTest.java:
The test uses a stub chat model that records the prompt it receives. The numbers and the prompt are in output/08-post-processor-in-the-advisor.txt: the retriever returned 50 documents and 5 reached the model.
question: How do I fix E-4129?
retriever returned 50 documents; the post-processor kept 5
And the context section of the prompt that the stub model received, with the five passages in reranked order:
Context information is below.
---------------------
Error E-4129 - API key lacks permission. The key in use does not carry the scope needed for this endpoint. Create a restricted key that includes it.
Error E-4128 - Tax identifier invalid. The tax registration number does not match the format for its country. Check the prefix and the number of digits.
Error E-4122 - Payment method not attached. The payment method is not attached to this customer. Attach it first, or pass it directly on a one-off charge.
Error E-4123 - Duplicate customer email. A customer with this email address already exists and the account is configured to enforce unique emails. Reuse the existing customer.
Error E-4126 - Coupon expired. The coupon passed its redemption end date. Create a new coupon or extend the validity in the dashboard.
Edge cases: no documents, fewer than N
A post-processor sits on the request path of every question, so the boring cases matter. An empty list comes back empty without calling the model, and a list shorter than N comes back whole, in reranked order, with nothing invented or dropped. Both are asserted in PostProcessorTest.java and written to output/09-post-processor-edge-cases.txt:
no documents in -> 0 out
3 documents in, topN = 5 -> 3 out (nothing invented, nothing dropped)
The test set is the same one used in the hybrid search post: a made-up support knowledge base for a payments API with 94 passages and 60 questions, each with exactly one passage that answers it. Four numbers are reported. hit@1 is how many questions have the right passage first, hit@3 and hit@5 are the same for the top three and top five, and MRR is the average of one over the rank of the right passage, so it rewards moving the answer from fifth to first but also from second to first. The output files call the right passage the gold passage. The first block of output/03-quality-by-pool-size.txt shows the bi-encoder on its own, then reranking the top 5, 10, 20, 50 and 94 of its results:
Two things stand out. Reranking helps substantially: the right passage is first for 49 questions instead of 41. And the shortlist does not need to be long. Reranking only the retriever’s top 5 already gets 48 of the 49, because the right passage was in the top 5 for 55 of the 60 questions. The recall block of the same file says why the larger pools still matter a little:
recall of the first stage (is the gold passage inside the pool at all?):
top 5 : 55/60
top 10 : 58/60
top 20 : 59/60
top 50 : 60/60
top 94 : 60/60
A reranker can only reorder what the retriever gives it. If the right passage is not in the shortlist, nothing the reranker does can bring it back, so the shortlist size sets a ceiling. Here the ceiling is 55/60 at five, 58 at ten, 59 at twenty and 60 at fifty.
The more interesting question is which questions improve. The corpus has four kinds of question, and splitting by kind from output/04-quality-by-question-kind.txt gives a very different picture from the average:
The exact-code questions are where the whole gain is, which makes sense: they are questions like “Explain E-4120” against 40 passages that all look alike and differ in one identifier, and an embedding smears that identifier away while a cross-encoder can match it token by token. Questions that were already answered perfectly (SKUs, and codes with surrounding words) cannot improve. The exact numbers, from output/04-quality-by-question-kind.txt:
Now the less comfortable column. For paraphrases, hit@1 went up from 15 to 17 but hit@5 went down from 23 to 20: the reranker pushed the right passage out of the top five for three questions that the retriever had got right. A cross-encoder trained on web search queries is not automatically better at every kind of question, and a reranker that is better on average can still be worse on a category you care about.
Every question whose rank changed
Of the 60 questions, 15 got a better rank, 6 got a worse one and 39 did not change. This is the list from output/05-helped-and-hurt.txt, with the right passage’s rank before and after (reranking the top 50):
gold question bi-encoder reranked
E-4106 What does error E-4106 mean? 28 3 better
E-4116 Why am I getting E-4116? 1 2 worse
E-4118 We see E-4118 in the logs, what~ 8 3 better
E-4120 Explain E-4120 2 1 better
E-4122 Support told me to look up E-41~ 4 3 better
E-4126 What is E-4126? 2 1 better
E-4129 How do I fix E-4129? 12 1 better
E-4131 Our service started returning E~ 4 1 better
E-4133 Meaning of E-4133 5 1 better
E-4135 Is E-4135 our fault? 2 1 better
E-4138 What causes E-4138 on the shop ~ 6 1 better
G-01 What happens when a customer's ~ 6 11 worse
G-05 How do I experiment safely befo~ 1 14 worse
G-07 Accept money from people who do~ 4 1 better
G-10 Stop dodgy purchases automatica~ 3 6 worse
G-12 What if our website is down whe~ 2 1 better
G-16 A client wants to stop being bi~ 3 1 better
G-17 Give our bookkeeper everything ~ 1 25 worse
G-20 How do we avoid ever handling r~ 2 1 better
G-24 One item in the basket is sold ~ 3 1 better
G-28 Which colleague altered the set~ 1 2 worse
better: 15, worse: 6, unchanged: 39
The worst case is the question about bookkeepers, where the right passage went from first to twenty-fifth. Reading the passages shows why: the question says “bookkeeper” and “everything that came in last quarter”, while the passage is titled “Handing transactions to the accountants” and talks about exporting transactions, fees and payouts for a date range. They share no content words, which is where a model that rewards word-level agreement is at a disadvantage and an embedding, which captures meaning, is not.
How far to trust these numbers. The knowledge base is synthetic and small, and the questions were written by the same person who wrote the passages. Differences of two or three questions are noise. The gain on exact codes is large enough (2 to 8 of 12) to be real here, but it is a measurement of this corpus, not a promise about yours. The combination of reranking with hybrid retrieval, which also fixes exact codes, was not measured; if you already use keyword search, measure the reranker on top of it before assuming the two gains add up. Measuring on your own data with a gold set is the only way to know, and testing LLM apps with evaluators in JUnit covers building one.
What it costs: about ten milliseconds per candidate
Quality is half the decision. The retriever took about 4 ms per question here; the reranker has to run the model once for every candidate. This is the time to rerank N candidates for one question, on a 2-vCPU cloud container with no GPU. The first column of each cell is the median of 12 calls in milliseconds, the second is the slowest, from output/06-latency-by-candidates-and-batch.txt:
machine: 2 vCPU, JDK 25.0.5
fp32 cross-encoder, max length 256 tokens; the first column of each cell is the median in ms, the second the slowest
The chart is the batch-32 column. The relationship is nearly linear, about 10 ms for each extra candidate, so the cost of a shortlist is easy to predict: a 50-candidate rerank is about half a second on this machine, a 20-candidate rerank is about a fifth of a second. Your hardware will differ, and a GPU changes the picture completely, but the shape (cost proportional to candidates) will not.
Putting quality and cost side by side is what lets you choose a pool size. output/07-latency-against-quality.txt measures each setting for both the full-precision model and an 8-bit quantised copy:
first stage (embed the question, compare with 94 vectors): median 3.8 ms
setting hit@1 hit@3 hit@5 MRR@10 rerank ms (median)
no reranker 41/60 51/60 55/60 0.782 0
fp32, top 5 48/60 55/60 55/60 0.863 70
int8, top 5 48/60 55/60 55/60 0.860 68
fp32, top 10 49/60 54/60 56/60 0.868 116
int8, top 10 49/60 54/60 56/60 0.868 106
fp32, top 20 49/60 55/60 56/60 0.870 199
int8, top 20 49/60 55/60 56/60 0.870 197
fp32, top 50 49/60 56/60 56/60 0.869 461
int8, top 50 49/60 56/60 56/60 0.869 503
Read down the fp32 rows. Reranking the top 5 costs about 70 ms and delivers 48 of the 49 hit@1 that the top 50 reaches, at about a seventh of the time. For this corpus, going past 10 candidates added nothing to hit@1, and going from 20 candidates to 50 added about 260 ms to every question. That is the real trade-off: the shortlist needs to be long enough that the right passage is usually in it, and no longer.
int8, batch size and threads
Three knobs, all measured. Quantisation: the int8 model file is 23 MB against 91 MB, and it picked the same first passage as fp32 for 58 of 60 questions (the last line of output/07-latency-against-quality.txt), with identical hit@1 at every pool size. On this CPU it was not meaningfully faster (at top 50 the two were within noise of each other), so the benefit here is size, not speed; on CPUs with fast 8-bit instructions the result can differ, so measure on the hardware you deploy to.
Batch size: in the first file the batch 1, 8 and 32 columns are within noise of each other on two cores. Batching gives a GPU its efficiency; on a small CPU the work per pair dominates.
Threads: restricting ONNX Runtime to one thread made the 50-candidate rerank take about 800 ms against about 500 ms with its default (both cores). If your service handles many requests at once, those cores are shared, so measure under load, not with one caller.
Five ways a reranker goes quietly wrong
None of the following throws an exception that would tell you something is off, except the first. Each is reproduced by a test in TrapsTest.java and written to a file named in the block.
1. The model needs an input you did not send
The model has three inputs, and BERT-style models fail loudly if token_type_ids is missing. The error is long and technical but ends with the cause; this is the one trap that announces itself. Output from output/10-trap-missing-input.txt:
model inputs : [input_ids, attention_mask, token_type_ids]
we sent : [input_ids, attention_mask]
result : Error code - ORT_FAIL - message: Non-zero status code returned while running Gather node. Name:'/bert/embeddings/token_type_embeddings/Gather' Status Message: /onnxruntime_src/include/onnxruntime/core/framework/op_kernel_context.h:45 const T* onnxruntime::OpKernelContext::Input(int) const [with T = onnxruntime::Tensor] Missing Input: token_type_ids
2. The pair is in the wrong order
A cross-encoder is trained on (query, passage) pairs. If you pass (passage, query) there is no error and every pair still gets a number; the numbers are just worse. Reranking the same top 20 both ways, from output/11-trap-pair-order.txt:
(query, passage) hit@1 49/60 MRR 0.870
(passage, query) hit@1 37/60 MRR 0.708
no error either way: both orders return a number for every pair
3. The score is not a probability
It is tempting to add a cut-off, “drop anything scoring under 2”, to avoid sending weak passages. The scores are raw logits that depend on the question, so no single threshold works. For 26 of the 60 questions even the best passage scored below zero, and the right passage itself scored anywhere from about −11 to 9. From output/12-trap-score-scale.txt:
gold passage score : min -11.5, median 2.1, max 9.0
best wrong passage score : min -11.4, median -1.4, max 7.8
questions whose best-scoring passage is below 0: 26 of 60
a fixed cut-off of 0 would drop the gold passage for 26 of 60 questions
a fixed cut-off of 2 would drop the gold passage for 30 of 60 questions
a fixed cut-off of 4 would drop the gold passage for 38 of 60 questions
Use the scores to order passages within one question. If you want a “nothing relevant found” rule, calibrate it on your own labelled data.
4. The answer is cut off by truncation
A pair that is longer than the maximum length is truncated, and nothing says so. The test puts the answer sentence at the end of a long passage and scores the same pair with two maximum lengths. From output/13-trap-truncation.txt:
question + passage = 224 tokens
score with max length 512 : -1.50
score with max length 64 : -11.03 (the answer was cut off before the model saw it)
score of the answer alone : 2.46
passages in this corpus longer than 256 tokens: 0 of 94
Two lessons: the same pair scored −1.50 with room for it and −11.03 without, and even with room the long passage scored below the answer sentence alone (2.46), because the surrounding text dilutes it. Chunk documents so that each passage is one idea.
5. The shortlist is too short
A reranker can reorder, not discover. If the right passage is not in the shortlist it never gets a chance. Five of the 60 questions had no right passage in the retriever’s top 5; reranking the top 50 brought four of them into the top 5. From output/14-trap-small-pool.txt:
gold question bi-encoder reranked top 50
E-4106 What does error E-4106 mean? 28 3
E-4118 We see E-4118 in the logs, what should we do? 8 3
E-4129 How do I fix E-4129? 12 1
E-4138 What causes E-4138 on the shop floor? 6 1
G-01 What happens when a customer's monthly payment does not go ~ 6 11
5 questions have no answer passage in the top 5; a reranker over the top 5 cannot help them,
a reranker over the top 50 brings 4 of them into the top 5
Should you add a reranker?
Maybe, but measure first. A reranker is worth its cost when the right passage is usually retrieved but often not first, and the 50-versus-5 gap in the recall numbers above is exactly that signal. If the right passage is often missing from your top 50, fix retrieval first (chunking, hybrid search, a better embedding model); a reranker cannot help there. If your questions are mostly paraphrases of the passage text, the evidence here is that a general-purpose cross-encoder can make some answers worse, so test on a held-out set of your own questions before shipping. Budget the latency honestly: about 10 ms per candidate on two CPU cores here, so a shortlist of 20 adds a fifth of a second to every question, and a GPU or a smaller shortlist is how you buy that back. The model in this post was trained on English web-search data; for other languages or very different text, look for a model that was, and check its model card for the licence before using it commercially. Finally, nothing here was tested with a live LLM answering the questions, only with whether the right passage reaches the prompt.
No Comments yet!