# Six rankers on 60 labelled queries over 94 passages (all-MiniLM-L6-v2, PostgreSQL 16, pgvector 0.8.7)

fusion: RRF with k=60 over the top 20 of each list. hit@1 and hit@5: the answer passage is first / in the first five. MRR@10: mean of 1/rank.

--- exact error code (12 queries) ---
ranker                             hit@1   hit@5   MRR@10
vector (pgvector cosine)            2/12   8/12  0.374
keyword (tsvector, ts_rank_cd)      9/12  11/12  0.819
keyword (BM25, in Java)             9/12  12/12  0.875
hybrid: vector + tsvector, RRF      9/12  11/12  0.806
hybrid: vector + BM25, RRF         10/12  11/12  0.883
hybrid in one SQL statement         9/12  11/12  0.806

--- exact SKU or setting (12 queries) ---
ranker                             hit@1   hit@5   MRR@10
vector (pgvector cosine)           12/12  12/12  1.000
keyword (tsvector, ts_rank_cd)     12/12  12/12  1.000
keyword (BM25, in Java)            11/12  12/12  0.958
hybrid: vector + tsvector, RRF     12/12  12/12  1.000
hybrid: vector + BM25, RRF         12/12  12/12  1.000
hybrid in one SQL statement        12/12  12/12  1.000

--- paraphrase (24 queries) ---
ranker                             hit@1   hit@5   MRR@10
vector (pgvector cosine)           15/24  23/24  0.767
keyword (tsvector, ts_rank_cd)      5/24  11/24  0.296
keyword (BM25, in Java)             4/24  13/24  0.290
hybrid: vector + tsvector, RRF     13/24  20/24  0.663
hybrid: vector + BM25, RRF         11/24  21/24  0.626
hybrid in one SQL statement        13/24  20/24  0.663

--- code plus words (12 queries) ---
ranker                             hit@1   hit@5   MRR@10
vector (pgvector cosine)           12/12  12/12  1.000
keyword (tsvector, ts_rank_cd)     10/12  12/12  0.896
keyword (BM25, in Java)            11/12  12/12  0.958
hybrid: vector + tsvector, RRF     12/12  12/12  1.000
hybrid: vector + BM25, RRF         12/12  12/12  1.000
hybrid in one SQL statement        12/12  12/12  1.000

--- all 60 queries ---
ranker                             hit@1   hit@5   MRR@10
vector (pgvector cosine)           41/60  55/60  0.782
keyword (tsvector, ts_rank_cd)     36/60  46/60  0.662
keyword (BM25, in Java)            35/60  49/60  0.674
hybrid: vector + tsvector, RRF     46/60  55/60  0.826
hybrid: vector + BM25, RRF         45/60  56/60  0.827
hybrid in one SQL statement        46/60  55/60  0.826

--- the SQL version against the Java version ---
queries whose top 5 are identical, in the same order, from both implementations: 60 of 60

--- one query of each kind: the top 3 from each ranker ---
[exact error code] "What does error E-4106 mean?"  -> answer is E-4106
   vector   : E-4108, E-4105, E-4114
   tsvector : E-4106, E-4109, E-4139
   BM25     : E-4109, E-4106, E-4140
   hybrid   : E-4108, E-4109, E-4111
[exact SKU or setting] "Does OP-TERM-5520 work when the internet is down?"  -> answer is OP-TERM-5520
   vector   : OP-TERM-5520, OP-SUP-GOLD, G-30
   tsvector : OP-TERM-5520, OP-TERM-6100, OP-TERM-5530
   BM25     : OP-TERM-5520
   hybrid   : OP-TERM-5520, OP-SUP-GOLD, OP-TERM-5530
[paraphrase] "What happens when a customer's monthly payment does not go through?"  -> answer is G-01
   vector   : G-15, G-04, G-16
   tsvector : E-4110, E-4111, G-15
   BM25     : G-04, G-06, orbit.receipts.locale
   hybrid   : G-15, G-16, G-04
[code plus words] "E-4106 keeps appearing in our Express webhook handler, is the body being parsed?"  -> answer is E-4106
   vector   : E-4106, E-4107, E-4116
   tsvector : E-4106, E-4102, E-4107
   BM25     : E-4106, E-4102, E-4107
   hybrid   : E-4106, E-4107, E-4102
