asmhatreandClaude Sonnet 5.5
d51ef56468
Co-Authored-By: Claude Sonnet 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01G8ikz8xdWuTP5yun8DZ1hk
36 lines
2.4 KiB
Plaintext
36 lines
2.4 KiB
Plaintext
# What a similarity score looks like (all-MiniLM-L6-v2, cosine)
|
|
|
|
Plain cosine similarity (NOT the Spring AI score, see the end) of each later phrasing with the cached phrasing of the SAME question (90 pairs):
|
|
min 0.337 p10 0.537 median 0.729 p90 0.869 max 0.916
|
|
lowest: 0.337 Where is my package right now? -> How can I track my order?
|
|
|
|
Best score each of those phrasings gets against a cached question of a DIFFERENT intent (look-alikes like reset password / reset router):
|
|
min 0.209 p10 0.311 median 0.446 p90 0.563 max 0.633
|
|
highest: 0.633 Can you help me reset the password for my login? -> How do I reset two-factor authentication on my account?
|
|
|
|
Best score of 20 off-topic questions against any cached question:
|
|
min 0.096 p10 0.133 median 0.180 p90 0.225 max 0.289
|
|
|
|
Does Redis report the same number? "I forgot my password, how can I get back into my account?"
|
|
cosine computed in Java : 0.838726
|
|
Document.getScore() from Redis : 0.919363 (matched stored question: "How do I reset my account password?")
|
|
(1 + cosine) / 2 : 0.919363 <- the score is this, not the cosine
|
|
same question as the cached one -> score 1.000000
|
|
|
|
The closest pairs of DIFFERENT cached questions (Spring AI score; 435 pairs in all). Above the threshold, one answers for the other:
|
|
0.852 "How do I reset my account password?" / "How do I reset two-factor authentication on my account?"
|
|
0.824 "How do I cancel an order I just placed?" / "How do I cancel a return I already requested?"
|
|
0.796 "How do I cancel an order I just placed?" / "How do I cancel my monthly subscription?"
|
|
0.791 "How can I track my order?" / "How can I track the return I sent back?"
|
|
0.789 "How can I track my order?" / "Can I pick up my order at a store?"
|
|
0.779 "How long is the warranty on your laptops?" / "How do I make a warranty claim?"
|
|
0.775 "How long does a refund take to arrive?" / "How can I track the return I sent back?"
|
|
0.767 "How do I update the firmware on my headphones?" / "How do I pair my headphones over Bluetooth?"
|
|
|
|
Spring AI's similarityThreshold is applied to that score. Threshold -> the cosine it really means (cosine = 2 x score - 1):
|
|
similarityThreshold 0.70 = cosine 0.40
|
|
similarityThreshold 0.80 = cosine 0.60
|
|
similarityThreshold 0.85 = cosine 0.70
|
|
similarityThreshold 0.90 = cosine 0.80
|
|
similarityThreshold 0.95 = cosine 0.90
|