Claude
1d4625a1c2
Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01B38FGKKam5SCGgwgduVAh3
33 lines
1.2 KiB
Plaintext
33 lines
1.2 KiB
Plaintext
# Generation and the faithfulness check
|
|
|
|
|
|
--- 1. the answer is in the chunks ---
|
|
status=answered grounded=true sources=4
|
|
answer: Full-time employees are entitled to 20 working days
|
|
of annual leave per calendar year.
|
|
|
|
--- 2. the model answers with something the chunks do not say ---
|
|
status=ungrounded grounded=false sources=4
|
|
answer: Employees get 30 days of annual leave.
|
|
the check the judge model was given:
|
|
Evaluate whether or not the following claim is supported by the provided document.
|
|
Respond with "yes" if the claim is supported, or "no" if it is not.
|
|
|
|
|
|
--- 3. the judge says "Yes." instead of "yes" ---
|
|
status=ungrounded grounded=false sources=4
|
|
answer: Full-time employees are entitled to 20 working days
|
|
of annual leave per calendar year.
|
|
with the reply "YES": grounded=true
|
|
|
|
--- 4. the judge call itself fails ---
|
|
status=ungrounded grounded=false sources=4
|
|
answer: Full-time employees are entitled to 20 working days
|
|
of annual leave per calendar year.
|
|
rag.faithfulness.judge_failures = 1
|
|
|
|
--- 5. nothing is retrieved (threshold 0.3, off-topic question) ---
|
|
status=no_context grounded=false sources=0
|
|
answer: I don't have enough information in the provided documents.
|
|
fact-check calls made: 0
|