Add rag module: Spring AI 2.0 RAG with pgvector, chunking, reranking and a faithfulness check
Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01B38FGKKam5SCGgwgduVAh3
This commit is contained in:
@@ -0,0 +1,32 @@
|
||||
# Generation and the faithfulness check
|
||||
|
||||
|
||||
--- 1. the answer is in the chunks ---
|
||||
status=answered grounded=true sources=4
|
||||
answer: Full-time employees are entitled to 20 working days
|
||||
of annual leave per calendar year.
|
||||
|
||||
--- 2. the model answers with something the chunks do not say ---
|
||||
status=ungrounded grounded=false sources=4
|
||||
answer: Employees get 30 days of annual leave.
|
||||
the check the judge model was given:
|
||||
Evaluate whether or not the following claim is supported by the provided document.
|
||||
Respond with "yes" if the claim is supported, or "no" if it is not.
|
||||
|
||||
|
||||
--- 3. the judge says "Yes." instead of "yes" ---
|
||||
status=ungrounded grounded=false sources=4
|
||||
answer: Full-time employees are entitled to 20 working days
|
||||
of annual leave per calendar year.
|
||||
with the reply "YES": grounded=true
|
||||
|
||||
--- 4. the judge call itself fails ---
|
||||
status=ungrounded grounded=false sources=4
|
||||
answer: Full-time employees are entitled to 20 working days
|
||||
of annual leave per calendar year.
|
||||
rag.faithfulness.judge_failures = 1
|
||||
|
||||
--- 5. nothing is retrieved (threshold 0.3, off-topic question) ---
|
||||
status=no_context grounded=false sources=0
|
||||
answer: I don't have enough information in the provided documents.
|
||||
fact-check calls made: 0
|
||||
Reference in New Issue
Block a user