Add rerank module: bi-encoder retrieval plus an ONNX Runtime cross-encoder reranker behind a Spring AI DocumentPostProcessor; hit@k and MRR by pool size and question kind, latency by candidates, batch size, threads and int8, and five reproduced traps. Local MiniLM models, no database or API key.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Cfo2K4u4rF18QsPXPFnytf
This commit is contained in:
Claude
@@ -0,0 +1,6 @@
|
||||
# A (query, passage) pair as the cross-encoder receives it
|
||||
|
||||
model inputs: [input_ids, attention_mask, token_type_ids]
|
||||
tokens : [[CLS], what, is, e, -, 412, ##6, ?, [SEP], error, e, -, 412, ##6, -, ref, ##und, window, closed, ., ref, ##unds, can, be, issued, for, 120, days, ., [SEP]]
|
||||
token_type_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
|
||||
attention_mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
|
||||
@@ -0,0 +1,16 @@
|
||||
# One question, the bi-encoder's top 8, rescored
|
||||
|
||||
question: A buyer told their bank they never got the item. What now?
|
||||
gold passage: G-04
|
||||
|
||||
retrieved id cosine cross-encoder logit
|
||||
1 G-06 0.446 -7.40
|
||||
2 G-04 0.440 -8.06
|
||||
3 E-4117 0.418 -11.08
|
||||
4 E-4101 0.418 -11.30
|
||||
5 G-03 0.405 -10.33
|
||||
6 E-4140 0.403 -9.91
|
||||
7 G-08 0.399 -9.62
|
||||
8 G-24 0.391 -9.61
|
||||
|
||||
bi-encoder rank of G-04: 2; cross-encoder winner: G-06
|
||||
@@ -0,0 +1,16 @@
|
||||
# Rerank the bi-encoder's top N (60 questions, 94 passages, fp32 cross-encoder)
|
||||
|
||||
ranking hit@1 hit@3 hit@5 MRR@10
|
||||
bi-encoder alone 41/60 51/60 55/60 0.782
|
||||
bi-encoder top 5 + cross-encoder 48/60 55/60 55/60 0.863
|
||||
bi-encoder top 10 + cross-encoder 49/60 54/60 56/60 0.868
|
||||
bi-encoder top 20 + cross-encoder 49/60 55/60 56/60 0.870
|
||||
bi-encoder top 50 + cross-encoder 49/60 56/60 56/60 0.869
|
||||
bi-encoder top 94 + cross-encoder 49/60 56/60 56/60 0.869
|
||||
|
||||
recall of the first stage (is the gold passage inside the pool at all?):
|
||||
top 5 : 55/60
|
||||
top 10 : 58/60
|
||||
top 20 : 59/60
|
||||
top 50 : 60/60
|
||||
top 94 : 60/60
|
||||
@@ -0,0 +1,11 @@
|
||||
# Bi-encoder alone against bi-encoder top 50 + cross-encoder, by kind of question
|
||||
|
||||
kind ranking hit@1 hit@3 hit@5 MRR@10
|
||||
exact error code bi-encoder 2/12 5/12 8/12 0.374
|
||||
reranked 8/12 12/12 12/12 0.792
|
||||
exact SKU or setting bi-encoder 12/12 12/12 12/12 1.000
|
||||
reranked 12/12 12/12 12/12 1.000
|
||||
paraphrase bi-encoder 15/24 22/24 23/24 0.767
|
||||
reranked 17/24 20/24 20/24 0.778
|
||||
code plus words bi-encoder 12/12 12/12 12/12 1.000
|
||||
reranked 12/12 12/12 12/12 1.000
|
||||
@@ -0,0 +1,26 @@
|
||||
# Gold passage rank before and after reranking the top 50 (only questions whose rank changed)
|
||||
|
||||
gold question bi-encoder reranked
|
||||
E-4106 What does error E-4106 mean? 28 3 better
|
||||
E-4116 Why am I getting E-4116? 1 2 worse
|
||||
E-4118 We see E-4118 in the logs, what~ 8 3 better
|
||||
E-4120 Explain E-4120 2 1 better
|
||||
E-4122 Support told me to look up E-41~ 4 3 better
|
||||
E-4126 What is E-4126? 2 1 better
|
||||
E-4129 How do I fix E-4129? 12 1 better
|
||||
E-4131 Our service started returning E~ 4 1 better
|
||||
E-4133 Meaning of E-4133 5 1 better
|
||||
E-4135 Is E-4135 our fault? 2 1 better
|
||||
E-4138 What causes E-4138 on the shop ~ 6 1 better
|
||||
G-01 What happens when a customer's ~ 6 11 worse
|
||||
G-05 How do I experiment safely befo~ 1 14 worse
|
||||
G-07 Accept money from people who do~ 4 1 better
|
||||
G-10 Stop dodgy purchases automatica~ 3 6 worse
|
||||
G-12 What if our website is down whe~ 2 1 better
|
||||
G-16 A client wants to stop being bi~ 3 1 better
|
||||
G-17 Give our bookkeeper everything ~ 1 25 worse
|
||||
G-20 How do we avoid ever handling r~ 2 1 better
|
||||
G-24 One item in the basket is sold ~ 3 1 better
|
||||
G-28 Which colleague altered the set~ 1 2 worse
|
||||
|
||||
better: 15, worse: 6, unchanged: 39
|
||||
@@ -0,0 +1,10 @@
|
||||
# Time to rerank N candidates (median and slowest of 12 calls, after 3 warm-up calls)
|
||||
|
||||
machine: 2 vCPU, JDK 25.0.5
|
||||
fp32 cross-encoder, max length 256 tokens; the first column of each cell is the median in ms, the second the slowest
|
||||
|
||||
candidates batch 1 batch 8 batch 32 batch 32, 1 thread
|
||||
10 125 / 188 121 / 142 122 / 138 148 / 208
|
||||
20 205 / 226 195 / 220 220 / 257 322 / 411
|
||||
50 546 / 661 512 / 578 489 / 549 800 / 913
|
||||
94 897 / 1062 838 / 966 962 / 1075 1635 / 1928
|
||||
@@ -0,0 +1,17 @@
|
||||
# Quality and added latency for each setting (60 questions; latency is the median of 12 calls)
|
||||
|
||||
machine: 2 vCPU, JDK 25.0.5
|
||||
first stage (embed the question, compare with 94 vectors): median 3.8 ms
|
||||
|
||||
setting hit@1 hit@3 hit@5 MRR@10 rerank ms (median)
|
||||
no reranker 41/60 51/60 55/60 0.782 0
|
||||
fp32, top 5 48/60 55/60 55/60 0.863 70
|
||||
int8, top 5 48/60 55/60 55/60 0.860 68
|
||||
fp32, top 10 49/60 54/60 56/60 0.868 116
|
||||
int8, top 10 49/60 54/60 56/60 0.868 106
|
||||
fp32, top 20 49/60 55/60 56/60 0.870 199
|
||||
int8, top 20 49/60 55/60 56/60 0.870 197
|
||||
fp32, top 50 49/60 56/60 56/60 0.869 461
|
||||
int8, top 50 49/60 56/60 56/60 0.869 503
|
||||
|
||||
int8 and fp32 put the same passage first for 58 of 60 questions (top 50)
|
||||
@@ -0,0 +1,34 @@
|
||||
# RerankingPostProcessor inside RetrievalAugmentationAdvisor (retrieve 50, keep 5)
|
||||
|
||||
question: How do I fix E-4129?
|
||||
retriever returned 50 documents; the post-processor kept 5
|
||||
|
||||
kept id retrieval rank rerank score text
|
||||
1 E-4129 12 3.52 Error E-4129 - API key lacks permission. The key
|
||||
2 E-4128 2 2.21 Error E-4128 - Tax identifier invalid. The tax r
|
||||
3 E-4122 1 1.48 Error E-4122 - Payment method not attached. The
|
||||
4 E-4123 14 1.09 Error E-4123 - Duplicate customer email. A custo
|
||||
5 E-4126 5 0.94 Error E-4126 - Coupon expired. The coupon passed
|
||||
|
||||
--- the prompt the model received ---
|
||||
Context information is below.
|
||||
|
||||
---------------------
|
||||
Error E-4129 - API key lacks permission. The key in use does not carry the scope needed for this endpoint. Create a restricted key that includes it.
|
||||
Error E-4128 - Tax identifier invalid. The tax registration number does not match the format for its country. Check the prefix and the number of digits.
|
||||
Error E-4122 - Payment method not attached. The payment method is not attached to this customer. Attach it first, or pass it directly on a one-off charge.
|
||||
Error E-4123 - Duplicate customer email. A customer with this email address already exists and the account is configured to enforce unique emails. Reuse the existing customer.
|
||||
Error E-4126 - Coupon expired. The coupon passed its redemption end date. Create a new coupon or extend the validity in the dashboard.
|
||||
---------------------
|
||||
|
||||
Given the context information and no prior knowledge, answer the query.
|
||||
|
||||
Follow these rules:
|
||||
|
||||
1. If the answer is not in the context, just say that you don't know.
|
||||
2. Avoid statements like "Based on the context..." or "The provided information...".
|
||||
|
||||
Query: How do I fix E-4129?
|
||||
|
||||
Answer:
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
# RerankingPostProcessor edge cases
|
||||
|
||||
no documents in -> 0 out
|
||||
3 documents in, topN = 5 -> 3 out (nothing invented, nothing dropped)
|
||||
@@ -0,0 +1,6 @@
|
||||
# Leaving out token_type_ids
|
||||
|
||||
model inputs : [input_ids, attention_mask, token_type_ids]
|
||||
we sent : [input_ids, attention_mask]
|
||||
result : Error code - ORT_FAIL - message: Non-zero status code returned while running Gather node. Name:'/bert/embeddings/token_type_embeddings/Gather' Status Message: /onnxruntime_src/include/onnxruntime/core/framework/op_kernel_context.h:45 const T* onnxruntime::OpKernelContext::Input(int) const [with T = onnxruntime::Tensor] Missing Input: token_type_ids
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
# (passage, query) instead of (query, passage); top 20 reranked
|
||||
|
||||
(query, passage) hit@1 49/60 MRR 0.870
|
||||
(passage, query) hit@1 37/60 MRR 0.708
|
||||
no error either way: both orders return a number for every pair
|
||||
@@ -0,0 +1,8 @@
|
||||
# Raw cross-encoder scores for the gold passage and for the best wrong passage
|
||||
|
||||
gold passage score : min -11.5, median 2.1, max 9.0
|
||||
best wrong passage score : min -11.4, median -1.4, max 7.8
|
||||
questions whose best-scoring passage is below 0: 26 of 60
|
||||
a fixed cut-off of 0 would drop the gold passage for 26 of 60 questions
|
||||
a fixed cut-off of 2 would drop the gold passage for 30 of 60 questions
|
||||
a fixed cut-off of 4 would drop the gold passage for 38 of 60 questions
|
||||
@@ -0,0 +1,7 @@
|
||||
# The answer sits at the end of a long passage
|
||||
|
||||
question + passage = 224 tokens
|
||||
score with max length 512 : -1.50
|
||||
score with max length 64 : -11.03 (the answer was cut off before the model saw it)
|
||||
score of the answer alone : 2.46
|
||||
passages in this corpus longer than 256 tokens: 0 of 94
|
||||
@@ -0,0 +1,11 @@
|
||||
# Questions whose gold passage is not in the bi-encoder's top 5
|
||||
|
||||
gold question bi-encoder reranked top 50
|
||||
E-4106 What does error E-4106 mean? 28 3
|
||||
E-4118 We see E-4118 in the logs, what should we do? 8 3
|
||||
E-4129 How do I fix E-4129? 12 1
|
||||
E-4138 What causes E-4138 on the shop floor? 6 1
|
||||
G-01 What happens when a customer's monthly payment does not go ~ 6 11
|
||||
|
||||
5 questions have no answer passage in the top 5; a reranker over the top 5 cannot help them,
|
||||
a reranker over the top 50 brings 4 of them into the top 5
|
||||
Reference in New Issue
Block a user