Add evaluation module: RelevancyEvaluator and FactCheckingEvaluator, golden dataset with a pass-rate gate, deterministic CI judge and simulated judge noise

Co-Authored-By: Claude Sonnet 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01JXVi2GMQ7bR5EmbUFdDj7N
2026-10-09 06:15:41 +00:00
parent 80db4eb8ac
commit 65f580c5bc
37 changed files with 1353 additions and 0 deletions
@@ -0,0 +1,42 @@
+# What RelevancyEvaluator and FactCheckingEvaluator send to the judge
+
+--- RelevancyEvaluator: prompt sent to the judge (1 message(s)) ---
+	Your task is to evaluate if the response for the query
+	is in line with the context information provided.
+
+	You have two options to answer. Either YES or NO.
+
+	Answer YES, if the response for the query
+	is in line with context information otherwise NO.
+
+	Query:
+	How long do I have to request a refund on an annual plan?
+
+	Response:
+	You can request a refund on an annual plan within 30 days of purchase.
+
+	Context:
+	Annual plans can be refunded within 30 days of purchase.
+	Monthly plans are not refundable.
+
+	Answer:
+
+--- RelevancyEvaluator: judge said "yes" ---
+pass=true score=1.0 feedback='' metadata={}
+
+--- FactCheckingEvaluator: prompt sent to the judge ---
+	Evaluate whether or not the following claim is supported by the provided document.
+	Respond with "yes" if the claim is supported, or "no" if it is not.
+
+	Document:
+	Annual plans can be refunded within 30 days of purchase.
+	Monthly plans are not refundable.
+
+	Claim:
+	You can request a refund on an annual plan within 30 days of purchase.
+
+--- FactCheckingEvaluator: judge said "yes" ---
+pass=true score=0.0 feedback='' metadata={}
+
+--- RelevancyEvaluator: judge said "no" ---
+pass=false score=0.0 feedback=''