asmhatre/spring-ai
1
0
Fork 0
Code Activity

Add evaluation module: RelevancyEvaluator and FactCheckingEvaluator, golden dataset with a pass-rate gate, deterministic CI judge and simulated judge noise

Browse Source 
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01JXVi2GMQ7bR5EmbUFdDj7N
This commit is contained in:
Claude
2026-10-09 06:15:41 +00:00
parent 80db4eb8ac
commit 65f580c5bc
37 changed files with 1353 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
evaluation/output/05-graded-and-composite.txt
+16
View File
@@ -0,0 +1,16 @@
# GradedEvaluator (minimum grade 4) and CompositeEvaluator
judge reply pass score feedback
'5' true 1.0 grade 5 of 5
'4/5' true 0.8 grade 4 of 5
'Score: 4 - mostly right, misses the date' true 0.8 grade 4 of 5
'3' false 0.6 grade 3 of 5
'I would say excellent' false 0.0 judge did not return a grade: I would say excellent
'10' false 0.0 judge did not return a grade: 10
== CompositeEvaluator on the regressed 'extra-seat' answer ("12 USD" instead of "8 USD") ==
answer : Extra seats cost 12 USD per month.
pass : false
score : 0.5
feedback : contains-facts failed (missing: [8 USD]);
verdicts : {relevancy=true, contains-facts=false}