asmhatre/spring-ai
1
0
Fork 0
Code Activity

Add hybrid-search module: vector, tsvector and BM25 fused with Reciprocal Rank Fusion in Java and SQL over one PostgreSQL table; quality by query type, held-out k check, plans and latency at 50k rows

Browse Source 
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0146opDKBYtADXvkc39zdiyw
This commit is contained in:
Claude
2026-10-22 18:04:49 +00:00
parent d7e68a14af
commit 2ccf1d8d19
26 changed files with 1508 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
hybrid-search/output/01-rrf-by-hand.txt
+34
View File
@@ -0,0 +1,34 @@
# Reciprocal Rank Fusion on two short rankings
vector ranking : [A, B, C, D]
keyword ranking: [C, A, E, B]
--- k = 1 ---
A score 0.83333 (vector rank 1, keyword rank 2)
C score 0.75000 (vector rank 3, keyword rank 1)
B score 0.53333 (vector rank 2, keyword rank 4)
E score 0.25000 (vector rank 0, keyword rank 3)
D score 0.20000 (vector rank 4, keyword rank 0)
--- k = 60 ---
A score 0.03252 (vector rank 1, keyword rank 2)
C score 0.03227 (vector rank 3, keyword rank 1)
B score 0.03175 (vector rank 2, keyword rank 4)
E score 0.01587 (vector rank 0, keyword rank 3)
D score 0.01563 (vector rank 4, keyword rank 0)
--- k = 1000 ---
A score 0.00200 (vector rank 1, keyword rank 2)
C score 0.00200 (vector rank 3, keyword rank 1)
B score 0.00199 (vector rank 2, keyword rank 4)
E score 0.00100 (vector rank 0, keyword rank 3)
D score 0.00100 (vector rank 4, keyword rank 0)
--- the arithmetic for A at k = 60 ---
rank 1 in the vector list: 1/(60+1) = 0.01639
rank 2 in the keyword list: 1/(60+2) = 0.01613
sum = 0.03252
--- an item in one list only ---
E is rank 3 in the keyword list and absent from the vector list: 1/(60+3) = 0.01587
D is rank 4 in the vector list and absent from the keyword list: 1/(60+4) = 0.01563
hybrid-search/output/02-quality-by-query-type.txt
+73
View File
@@ -0,0 +1,73 @@
# Six rankers on 60 labelled queries over 94 passages (all-MiniLM-L6-v2, PostgreSQL 16, pgvector 0.8.7)
fusion: RRF with k=60 over the top 20 of each list. hit@1 and hit@5: the answer passage is first / in the first five. MRR@10: mean of 1/rank.
--- exact error code (12 queries) ---
ranker hit@1 hit@5 MRR@10
vector (pgvector cosine) 2/12 8/12 0.374
keyword (tsvector, ts_rank_cd) 9/12 11/12 0.819
keyword (BM25, in Java) 9/12 12/12 0.875
hybrid: vector + tsvector, RRF 9/12 11/12 0.806
hybrid: vector + BM25, RRF 10/12 11/12 0.883
hybrid in one SQL statement 9/12 11/12 0.806
--- exact SKU or setting (12 queries) ---
ranker hit@1 hit@5 MRR@10
vector (pgvector cosine) 12/12 12/12 1.000
keyword (tsvector, ts_rank_cd) 12/12 12/12 1.000
keyword (BM25, in Java) 11/12 12/12 0.958
hybrid: vector + tsvector, RRF 12/12 12/12 1.000
hybrid: vector + BM25, RRF 12/12 12/12 1.000
hybrid in one SQL statement 12/12 12/12 1.000
--- paraphrase (24 queries) ---
ranker hit@1 hit@5 MRR@10
vector (pgvector cosine) 15/24 23/24 0.767
keyword (tsvector, ts_rank_cd) 5/24 11/24 0.296
keyword (BM25, in Java) 4/24 13/24 0.290
hybrid: vector + tsvector, RRF 13/24 20/24 0.663
hybrid: vector + BM25, RRF 11/24 21/24 0.626
hybrid in one SQL statement 13/24 20/24 0.663
--- code plus words (12 queries) ---
ranker hit@1 hit@5 MRR@10
vector (pgvector cosine) 12/12 12/12 1.000
keyword (tsvector, ts_rank_cd) 10/12 12/12 0.896
keyword (BM25, in Java) 11/12 12/12 0.958
hybrid: vector + tsvector, RRF 12/12 12/12 1.000
hybrid: vector + BM25, RRF 12/12 12/12 1.000
hybrid in one SQL statement 12/12 12/12 1.000
--- all 60 queries ---
ranker hit@1 hit@5 MRR@10
vector (pgvector cosine) 41/60 55/60 0.782
keyword (tsvector, ts_rank_cd) 36/60 46/60 0.662
keyword (BM25, in Java) 35/60 49/60 0.674
hybrid: vector + tsvector, RRF 46/60 55/60 0.826
hybrid: vector + BM25, RRF 45/60 56/60 0.827
hybrid in one SQL statement 46/60 55/60 0.826
--- the SQL version against the Java version ---
queries whose top 5 are identical, in the same order, from both implementations: 60 of 60
--- one query of each kind: the top 3 from each ranker ---
[exact error code] "What does error E-4106 mean?" -> answer is E-4106
vector : E-4108, E-4105, E-4114
tsvector : E-4106, E-4109, E-4139
BM25 : E-4109, E-4106, E-4140
hybrid : E-4108, E-4109, E-4111
[exact SKU or setting] "Does OP-TERM-5520 work when the internet is down?" -> answer is OP-TERM-5520
vector : OP-TERM-5520, OP-SUP-GOLD, G-30
tsvector : OP-TERM-5520, OP-TERM-6100, OP-TERM-5530
BM25 : OP-TERM-5520
hybrid : OP-TERM-5520, OP-SUP-GOLD, OP-TERM-5530
[paraphrase] "What happens when a customer's monthly payment does not go through?" -> answer is G-01
vector : G-15, G-04, G-16
tsvector : E-4110, E-4111, G-15
BM25 : G-04, G-06, orbit.receipts.locale
hybrid : G-15, G-16, G-04
[code plus words] "E-4106 keeps appearing in our Express webhook handler, is the body being parsed?" -> answer is E-4106
vector : E-4106, E-4107, E-4116
tsvector : E-4106, E-4102, E-4107
BM25 : E-4106, E-4102, E-4107
hybrid : E-4106, E-4107, E-4102
hybrid-search/output/03-rrf-parameters.txt
+26
View File
@@ -0,0 +1,26 @@
# RRF parameters on the 60 queries: k, candidates per list, list weights
--- k, with 20 candidates per list ---
k hit@1 hit@5 MRR@10
1 46/60 58/60 0.851
10 45/60 55/60 0.826
60 46/60 55/60 0.826
200 46/60 55/60 0.825
1000 46/60 55/60 0.825
--- candidates taken from each list, with k = 60 ---
cands hit@1 hit@5 MRR@10
3 41/60 57/60 0.802
5 43/60 59/60 0.828
10 45/60 57/60 0.823
20 46/60 55/60 0.826
50 46/60 55/60 0.829
--- weights (vector : keyword), k = 60, 20 candidates ---
weights hit@1 hit@5 MRR@10
1.0:0.0 41/60 55/60 0.782
1.0:0.5 45/60 56/60 0.822
1.0:1.0 46/60 55/60 0.826
0.5:1.0 42/60 54/60 0.781
0.0:1.0 37/60 50/60 0.692
hybrid-search/output/04-held-out-k.txt
+15
View File
@@ -0,0 +1,15 @@
# Choosing k on half of the queries and scoring it on the other half
candidates per list: 20. The 60 queries are split by position into two folds of 30; k comes from {1, 10, 60, 200}.
--- fold 1: train on 30, test on 30 ---
k chosen on the training half (best MRR@10 0.872): 1
held-out half, k=1 : 22/30 29/30 0.831
held-out half, k=60 : 24/30 28/30 0.859
held-out half, vector only: 21/30 30/30 0.831
--- fold 2: train on 30, test on 30 ---
k chosen on the training half (best MRR@10 0.859): 60
held-out half, k=60 : 22/30 27/30 0.794
held-out half, k=60 : 22/30 27/30 0.794
held-out half, vector only: 20/30 25/30 0.733
hybrid-search/output/05-retriever-and-advisor.txt
+43
View File
@@ -0,0 +1,43 @@
# HybridDocumentRetriever inside RetrievalAugmentationAdvisor (top 3 passages per question)
--- what the retriever returns for "Why am I getting E-4116?" ---
score 0.03279 vector rank 1 keyword rank 1 Error E-4116 - Charge already captured. A capture was attemp
score 0.03125 vector rank 4 keyword rank 4 Error E-4111 - Authentication challenge failed. The customer
score 0.02988 vector rank 9 keyword rank 5 Error E-4118 - Payout account not verified. Payouts are bloc
--- the prompt the model received ---
Context information is below.
---------------------
Error E-4116 - Charge already captured. A capture was attempted on a charge that has already been captured in full. Fetch the charge first and check its status instead of capturing blindly.
Error E-4111 - Authentication challenge failed. The customer abandoned or failed the 3-D Secure challenge. The payment cannot be completed with this card; offer another payment method.
Error E-4118 - Payout account not verified. Payouts are blocked because the bank account has not passed verification. Confirm the two micro-deposits shown in the dashboard to unlock transfers.
---------------------
Given the context information and no prior knowledge, answer the query.
Follow these rules:
1. If the answer is not in the context, just say that you don't know.
2. Avoid statements like "Based on the context..." or "The provided information...".
Query: Why am I getting E-4116?
Answer:
--- does the answer passage reach the prompt? the 12 exact-code questions, top 3 passages ---
E-4106 vector-only: false hybrid: false What does error E-4106 mean?
E-4116 vector-only: true hybrid: true Why am I getting E-4116?
E-4118 vector-only: false hybrid: true We see E-4118 in the logs, what should we do?
E-4120 vector-only: true hybrid: true Explain E-4120
E-4122 vector-only: false hybrid: true Support told me to look up E-4122
E-4126 vector-only: true hybrid: true What is E-4126?
E-4129 vector-only: false hybrid: true How do I fix E-4129?
E-4131 vector-only: false hybrid: true Our service started returning E-4131 this morning
E-4133 vector-only: false hybrid: true Meaning of E-4133
E-4135 vector-only: true hybrid: true Is E-4135 our fault?
E-4138 vector-only: false hybrid: true What causes E-4138 on the shop floor?
E-4140 vector-only: true hybrid: true Customer reports E-4140, is the money safe?
answer passage in the top 3: vector-only 5 of 12, hybrid 11 of 12
hybrid-search/output/06-scale-plans-and-latency.txt
+45
View File
@@ -0,0 +1,45 @@
# 94 rows against 50,094 rows: plans, index sizes, latency (PostgreSQL 16, pgvector 0.8.7, 2 vCPU)
filler rows inserted: 50000 in 7819 ms. HNSW index built in 94632 ms, GIN index in 292 ms.
rows in hybrid_scale: 50094
heap 98 MB, HNSW index 98 MB, GIN index 2248 kB
--- plan for the vector query at 50,094 rows (scan nodes only) ---
Limit (actual rows=20 loops=1)
-> Index Scan using hybrid_scale_embedding_idx on hybrid_scale (actual rows=20 loops=1)
--- plan for the keyword query at 50,094 rows (scan nodes only) ---
Limit (actual rows=20 loops=1)
-> Sort (actual rows=20 loops=1)
Sort Key: (ts_rank_cd(d.tsv, '''get'' | ''e'' | ''-4116'''::tsquery)) DESC
Sort Method: quicksort Memory: 26kB
-> Bitmap Heap Scan on hybrid_scale d (actual rows=40 loops=1)
-> Bitmap Index Scan on hybrid_scale_tsv_idx (actual rows=40 loops=1)
--- plan for the same keyword query at 94 rows ---
Limit (actual rows=20 loops=1)
-> Sort (actual rows=20 loops=1)
Sort Key: (ts_rank_cd(d.tsv, '''get'' | ''e'' | ''-4116'''::tsquery)) DESC
Sort Method: quicksort Memory: 26kB
-> Seq Scan on hybrid_docs d (actual rows=40 loops=1)
Rows Removed by Filter: 54
--- rows the keyword query has to rank, by query kind (50,094 rows) ---
exact error code average rows matching the OR query: 1547
exact SKU or setting average rows matching the OR query: 1970
paraphrase average rows matching the OR query: 3781
code plus words average rows matching the OR query: 5220
--- latency in milliseconds over 60 queries x 3 runs after 2 warm-up passes (query embedding excluded unless stated) ---
94 p50 94 p95 50k p50 50k p95
embedding the question (MiniLM) 7.4 10.4 (same for both table sizes)
vector, top 20 25.3 32.2 28.0 36.4
keyword, top 20 27.0 39.3 85.5 177.1
hybrid in Java (2 queries + fuse) 51.6 65.3 111.2 208.2
hybrid in one SQL statement 31.5 40.1 132.3 226.4
--- keyword query latency at 50,094 rows by query kind (p50 / p95 ms) ---
exact error code 51.1 / 124.0
exact SKU or setting 35.7 / 223.0
paraphrase 85.9 / 169.6
code plus words 121.2 / 214.9