asmhatreandClaude Sonnet 5.5
d51ef56468
Co-Authored-By: Claude Sonnet 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01G8ikz8xdWuTP5yun8DZ1hk
33 lines
2.8 KiB
Plaintext
33 lines
2.8 KiB
Plaintext
# Threshold sweep: 30 cached questions, 90 rewordings and 20 off-topic questions asked
|
|
|
|
threshold = the value given to SearchRequest.similarityThreshold (Spring AI score); cosine = 2 x threshold - 1.
|
|
A served answer is RIGHT if it belongs to the intent of the question asked, WRONG if it belongs to another intent.
|
|
An off-topic question has no right answer in the cache, so any hit on it is WRONG.
|
|
|
|
threshold | cosine | hit rate | right | wrong (reworded) | wrong (off-topic) | wrong / all hits | cached pairs of different intents that answer for each other
|
|
0.50 | 0.00 | 100% | 86/90 | 4/90 | 20/20 | 21.8% | 396 of 435
|
|
0.60 | 0.20 | 100% | 86/90 | 4/90 | 6/20 | 10.4% | 151 of 435
|
|
0.70 | 0.40 | 99% | 86/90 | 3/90 | 0/20 | 3.4% | 27 of 435
|
|
0.75 | 0.50 | 94% | 83/90 | 2/90 | 0/20 | 2.4% | 13 of 435
|
|
0.80 | 0.60 | 86% | 77/90 | 0/90 | 0/20 | 0.0% | 2 of 435
|
|
0.85 | 0.70 | 58% | 52/90 | 0/90 | 0/20 | 0.0% | 1 of 435
|
|
0.90 | 0.80 | 26% | 23/90 | 0/90 | 0/20 | 0.0% | 0 of 435
|
|
0.95 | 0.90 | 3% | 3/90 | 0/90 | 0/20 | 0.0% | 0 of 435
|
|
|
|
Questions that were answered WRONGLY at threshold 0.80 (asked -> served from): none among these 110 probes
|
|
|
|
Rewordings the cache MISSED at threshold 0.80 (best stored question and its score):
|
|
0.787 "Where can I end my membership plan?" (nearest: "How do I cancel my monthly subscription?")
|
|
0.746 "Can you tell me what qualifies for getting my money back?" (nearest: "Which items can be refunded?")
|
|
0.696 "Where is my package right now?" (nearest: "Can I pick up my order at a store?")
|
|
0.759 "I want to see the delivery status of my purchase" (nearest: "How can I track my order?")
|
|
0.752 "Has my returned parcel arrived at your warehouse yet?" (nearest: "How long will delivery take?")
|
|
0.790 "My device broke, how do I get it repaired under warranty?" (nearest: "How do I make a warranty claim?")
|
|
0.797 "How many days until my parcel gets here?" (nearest: "How long will delivery take?")
|
|
0.739 "Can I pay with a credit card or PayPal?" (nearest: "Which payment methods do you accept?")
|
|
0.798 "How can I pay for my order?" (nearest: "How can I track my order?")
|
|
0.769 "I need a receipt for my purchase, where do I find it?" (nearest: "Where can I download an invoice for my order?")
|
|
0.743 "Is in-store collection available?" (nearest: "Can I pick up my order at a store?")
|
|
0.753 "I would rather collect my purchase myself, is that an option?" (nearest: "Can I pick up my order at a store?")
|
|
0.703 "Do you offer click and collect?" (nearest: "Do you sell gift cards?")
|