Add evaluation module: RelevancyEvaluator and FactCheckingEvaluator, golden dataset with a pass-rate gate, deterministic CI judge and simulated judge noise
Co-Authored-By: Claude Sonnet 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01JXVi2GMQ7bR5EmbUFdDj7N
This commit is contained in:
@@ -0,0 +1,17 @@
|
||||
# Simulated judge noise: 200 runs of a 12-case suite per build
|
||||
|
||||
noise-free pass rate: healthy build 1.00, regressed build 0.75
|
||||
|
||||
== 3% of judge verdicts flipped (mean pass rate: healthy 0.947, regressed 0.706) ==
|
||||
gate healthy build fails it regressed build fails it
|
||||
every case must pass 96 of 200 runs 200 of 200 runs
|
||||
pass rate >= 0.90 29 of 200 runs 200 of 200 runs
|
||||
pass rate >= 0.85 29 of 200 runs 200 of 200 runs
|
||||
pass rate >= 0.80 2 of 200 runs 200 of 200 runs
|
||||
|
||||
== 8% of judge verdicts flipped (mean pass rate: healthy 0.859, regressed 0.640) ==
|
||||
gate healthy build fails it regressed build fails it
|
||||
every case must pass 166 of 200 runs 200 of 200 runs
|
||||
pass rate >= 0.90 102 of 200 runs 200 of 200 runs
|
||||
pass rate >= 0.85 102 of 200 runs 200 of 200 runs
|
||||
pass rate >= 0.80 47 of 200 runs 200 of 200 runs
|
||||
Reference in New Issue
Block a user