Claude
1d4625a1c2
Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01B38FGKKam5SCGgwgduVAh3
66 lines
4.9 KiB
Markdown
66 lines
4.9 KiB
Markdown
# 4. Retrieval and reranking
|
|
|
|
prev: [3. Ingestion](03-ingestion.md) · [Index](../README.md) · next: [5. Generation and the faithfulness check](05-generation-and-the-faithfulness-check.md)
|
|
|
|
Tests: [`RetrievalTest`](../src/test/java/com/ankurm/rag/RetrievalTest.java),
|
|
[`TenantFilterTest`](../src/test/java/com/ankurm/rag/TenantFilterTest.java),
|
|
[`RerankTest`](../src/test/java/com/ankurm/rag/RerankTest.java). Wiring: [`RagConfig`](../src/main/java/com/ankurm/rag/config/RagConfig.java).
|
|
|
|
**All similarity scores in this chapter come from the hashing embedding model.** They show how the components behave, not what a
|
|
real model would score.
|
|
|
|
## Top-k, and the threshold that is not there by default
|
|
|
|
`VectorStoreDocumentRetriever` returns the `topK` nearest chunks. Its default similarity threshold accepts everything, and
|
|
[output 06](output/06-retrieval-threshold-and-prompt.txt) shows what follows. For a question about leave, a threshold of 0.0 returned all four
|
|
chunks (scores 0.59, 0.41, 0.25, 0.20); 0.3 returned two. For an off-topic question ("capital of Mongolia") the default still
|
|
returned all four chunks, each with score 0.0000, and all four went into the prompt: **the "nothing found" safety net never
|
|
fires while the threshold accepts everything.** With 0.3 nothing was retrieved.
|
|
|
|
Which threshold is right depends entirely on the embedding model and cannot be copied from here. Find yours by asking questions
|
|
you know the corpus cannot answer and looking at the top scores.
|
|
|
|
## The empty-context path
|
|
|
|
`ContextualQueryAugmenter` builds the prompt. With `allowEmptyContext(false)` (what `RagConfig` sets) and no chunks, the model
|
|
receives only "The user query is outside your knowledge base. Politely inform the user that you can't answer it."
|
|
With `allowEmptyContext(true)` the model receives the bare question and answers from whatever it was trained on, which for a
|
|
document-QA service is the failure you built RAG to avoid. Both prompts are in output 06.
|
|
`QuestionAnswerAdvisor` has no such switch: [output 11](output/11-question-answer-advisor.txt) shows it sending an empty context block instead.
|
|
|
|
## Tenant isolation
|
|
|
|
Put `tenant_id` in every chunk's metadata at ingestion (the controller does) and pass a filter at query time.
|
|
[Output 07](output/07-tenant-filter-and-injection.txt), on the in-memory store and on PostgreSQL alike:
|
|
|
|
- with no filter, the top three chunks for "How many days of annual leave" included one from the other tenant;
|
|
- `new FilterExpressionBuilder().eq("tenant_id", "acme").build()` returned only that tenant's chunks;
|
|
- a filter *string* built as `"tenant_id == '" + input + "'"` with `input = globex' || tenant_id == 'acme` parsed to
|
|
`tenant_id == 'globex' || tenant_id == 'acme'` and returned both tenants' chunks;
|
|
- the same hostile text handed to `eq("tenant_id", input)` returned nothing, with a single or a double quote in it, on both stores.
|
|
|
|
So: never build a filter string from a request value; pass the value to the builder. That is what
|
|
[`RagController`](../src/main/java/com/ankurm/rag/web/RagController.java) does. The two hostile strings tried are the ones in the
|
|
test; this is not a security audit of the filter parser.
|
|
|
|
**Not tested:** how the filter performs. In pgvector, an approximate (HNSW) index is scanned first and the `WHERE` is applied to what
|
|
it returns, so a selective filter can return fewer than `topK` rows; pgvector 0.8.0 added iterative index scans to address this
|
|
([pgvector README, "Filtering"](https://github.com/pgvector/pgvector#filtering)). This repository ran pgvector **0.6.0**, which predates them,
|
|
and never ran a filtered query against enough rows to see the effect. If one tenant is a small fraction of the table, test it.
|
|
|
|
## Reranking
|
|
|
|
Spring AI 2.0.1 has the hook (`DocumentPostProcessor`) and no reranker: `spring-ai-rag` contains no class with "rerank" in its
|
|
name ([output 12](output/12-api-facts.txt)). [`LlmReranker`](../src/main/java/com/ankurm/rag/query/LlmReranker.java) is ours: it asks the
|
|
chat model to rate each candidate 0-10 and keeps the best `topN` ([output 08](output/08-reranking.txt)).
|
|
|
|
- **Cost:** one model call per candidate. Four candidates made four calls; with `topK` 20 every question costs 20 rating calls before the answer.
|
|
- **Latency:** the calls run on virtual threads. With 200 ms per call standing in for the network, 20 calls took over 4 s one after another
|
|
and under 1 s through the reranker. That shows the structure works; a real API's rate limits are not modelled.
|
|
- **Failure:** the reply must be a bare integer. `"Score: 8"` failed to parse for every candidate, all scores became 0, and the survivors were the
|
|
vector-search order. Nothing errors; only the `rag.rerank.failures` counter moves. A reply of `" 9\n"` parsed fine. Watch the counter.
|
|
|
|
**Not tested:** whether reranking improves which chunks reach the prompt. The scripted rater is word overlap and agrees with the fake embeddings.
|
|
|
|
Next: [5. Generation and the faithfulness check](05-generation-and-the-faithfulness-check.md)
|