spring-ai/rag/docs/05-generation-and-the-faithfulness-check.md

5. Generation and the faithfulness check

prev: 4. Retrieval and reranking · Index · next: 6. Observability and a production checklist

Source: RagQueryService. Test: FaithfulnessTest (output 09).

After the model answers, RagQueryService asks a second model call, FactCheckingEvaluator, whether the answer is supported by the chunks that were in the prompt. The response reports one of three statuses:

status	when	fact check runs?
answered	chunks retrieved and the judge said "yes"	yes
ungrounded	chunks retrieved and the judge said anything other than "yes" (case aside), or the judge call failed	yes
no_context	nothing retrieved	no

Behaviours the test pins down:

• The verdict is matched literally. A judge reply of Yes. counted as not grounded; YES counted as grounded. The evaluator compares against "yes" ignoring case, not ignoring punctuation. A real judge model that likes to add a full stop would fail every answer; instruct it to answer with one word, and watch rag_queries_total{status="ungrounded"}.
• A failing judge is not a passing judge. If the judge call throws (a rate limit, a timeout) the answer is reported ungrounded and rag.faithfulness.judge_failures increments. The answer text is still returned, so the caller decides what to do with an unchecked answer.
• No chunks, no check. With nothing to check against, the service skips the judge (zero fact-check calls in the output) and reports no_context.
• The check is only as good as the judge. The tests script the judge, so they prove the wiring and the four outcomes, not that a real model spots an invented claim. Measure that on your own questions before you rely on it, and note that it doubles the model calls per question.

Next: 6. Observability and a production checklist