asmhatre/spring-ai
1
0
Fork 0
Code Actions Activity

Add rag module: Spring AI 2.0 RAG with pgvector, chunking, reranking and a faithfulness check

Browse Source 
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01B38FGKKam5SCGgwgduVAh3
This commit is contained in:
Claude
2026-09-21 19:09:05 +00:00
commit 1d4625a1c2
62 changed files with 3715 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
rag/docs/05-generation-and-the-faithfulness-check.md
+28
View File
@@ -0,0 +1,28 @@
# 5. Generation and the faithfulness check
prev: [4. Retrieval and reranking](04-retrieval-and-reranking.md) · [Index](../README.md) · next: [6. Observability and a production checklist](06-observability-and-production-checklist.md)
Source: [`RagQueryService`](../src/main/java/com/ankurm/rag/query/RagQueryService.java). Test:
[`FaithfulnessTest`](../src/test/java/com/ankurm/rag/FaithfulnessTest.java) ([output 09](output/09-faithfulness-check.txt)).
After the model answers, `RagQueryService` asks a second model call, `FactCheckingEvaluator`, whether the answer is supported by the
chunks that were in the prompt. The response reports one of three statuses:
| status | when | fact check runs? |
|---|---|---|
| `answered` | chunks retrieved and the judge said "yes" | yes |
| `ungrounded` | chunks retrieved and the judge said anything other than "yes" (case aside), or the judge call failed | yes |
| `no_context` | nothing retrieved | no |
Behaviours the test pins down:
- **The verdict is matched literally.** A judge reply of `Yes.` counted as *not* grounded; `YES` counted as grounded. The evaluator
compares against "yes" ignoring case, not ignoring punctuation. A real judge model that likes to add a full stop would fail
every answer; instruct it to answer with one word, and watch `rag_queries_total{status="ungrounded"}`.
- **A failing judge is not a passing judge.** If the judge call throws (a rate limit, a timeout) the answer is reported `ungrounded` and
`rag.faithfulness.judge_failures` increments. The answer text is still returned, so the caller decides what to do with an unchecked answer.
- **No chunks, no check.** With nothing to check against, the service skips the judge (zero fact-check calls in the output) and reports `no_context`.
- **The check is only as good as the judge.** The tests script the judge, so they prove the wiring and the four outcomes, not that a real model
spots an invented claim. Measure that on your own questions before you rely on it, and note that it doubles the model calls per question.
Next: [6. Observability and a production checklist](06-observability-and-production-checklist.md)