Add evaluation module: RelevancyEvaluator and FactCheckingEvaluator, golden dataset with a pass-rate gate, deterministic CI judge and simulated judge noise

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01JXVi2GMQ7bR5EmbUFdDj7N
This commit is contained in:
Claude
2026-10-09 06:15:41 +00:00
parent 80db4eb8ac
commit 65f580c5bc
37 changed files with 1353 additions and 0 deletions
+17
View File
@@ -0,0 +1,17 @@
# Simulated judge noise: 200 runs of a 12-case suite per build
noise-free pass rate: healthy build 1.00, regressed build 0.75
== 3% of judge verdicts flipped (mean pass rate: healthy 0.947, regressed 0.706) ==
gate healthy build fails it regressed build fails it
every case must pass 96 of 200 runs 200 of 200 runs
pass rate >= 0.90 29 of 200 runs 200 of 200 runs
pass rate >= 0.85 29 of 200 runs 200 of 200 runs
pass rate >= 0.80 2 of 200 runs 200 of 200 runs
== 8% of judge verdicts flipped (mean pass rate: healthy 0.859, regressed 0.640) ==
gate healthy build fails it regressed build fails it
every case must pass 166 of 200 runs 200 of 200 runs
pass rate >= 0.90 102 of 200 runs 200 of 200 runs
pass rate >= 0.85 102 of 200 runs 200 of 200 runs
pass rate >= 0.80 47 of 200 runs 200 of 200 runs