asmhatre/spring-ai
1
0
Fork 0
Code Activity

Add evaluation module: RelevancyEvaluator and FactCheckingEvaluator, golden dataset with a pass-rate gate, deterministic CI judge and simulated judge noise

Browse Source 
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01JXVi2GMQ7bR5EmbUFdDj7N
This commit is contained in:
Claude
2026-10-09 06:15:41 +00:00
parent 80db4eb8ac
commit 65f580c5bc
37 changed files with 1353 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
evaluation/output/03-golden-run.txt
+37
View File
@@ -0,0 +1,37 @@
# Golden dataset: 12 cases, deterministic judge, pass-rate gate at 0.90
== healthy build ==
case relevant grounded has facts passed
refund-annual true true true true
refund-monthly true true true true
storage-pro true true true true
support-hours true true true true
sso-plan true true true true
data-region true true true true
api-rate-limit true true true true
backup-retention true true true true
macos-agent true true true true
reset-link true true true true
extra-seat true true true true
trial-card true true true true
pass rate: 1.00 failing: []
== regressed build ==
case relevant grounded has facts passed
refund-annual false false false false
refund-monthly true true true true
storage-pro true true true true
support-hours true true true true
sso-plan false false false false
data-region true true true true
api-rate-limit true true true true
backup-retention true true true true
macos-agent true true true true
reset-link true true true true
extra-seat true true false false
trial-card true true true true
pass rate: 0.75 failing: [refund-annual, sso-plan, extra-seat]
== the gate ==
healthy build : requirePassRate(0.90) returned normally
regressed build : java.lang.AssertionError: pass rate 0.75 is below the 0.90 threshold; failing cases: [refund-annual, sso-plan, extra-seat]