Add semantic-cache module: CallAdvisor over a Redis Stack vector index with a real in-process embedding model; threshold sweep (hit rate against wrong answers), 600-request replay with an illustrative cost model, tenant filter, expiry, look-alike and embedding-model-change traps; optional real local model
Co-Authored-By: Claude Sonnet 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01G8ikz8xdWuTP5yun8DZ1hk
This commit is contained in:
Claude Sonnet 5.5
@@ -0,0 +1,35 @@
|
||||
# What a similarity score looks like (all-MiniLM-L6-v2, cosine)
|
||||
|
||||
Plain cosine similarity (NOT the Spring AI score, see the end) of each later phrasing with the cached phrasing of the SAME question (90 pairs):
|
||||
min 0.337 p10 0.537 median 0.729 p90 0.869 max 0.916
|
||||
lowest: 0.337 Where is my package right now? -> How can I track my order?
|
||||
|
||||
Best score each of those phrasings gets against a cached question of a DIFFERENT intent (look-alikes like reset password / reset router):
|
||||
min 0.209 p10 0.311 median 0.446 p90 0.563 max 0.633
|
||||
highest: 0.633 Can you help me reset the password for my login? -> How do I reset two-factor authentication on my account?
|
||||
|
||||
Best score of 20 off-topic questions against any cached question:
|
||||
min 0.096 p10 0.133 median 0.180 p90 0.225 max 0.289
|
||||
|
||||
Does Redis report the same number? "I forgot my password, how can I get back into my account?"
|
||||
cosine computed in Java : 0.838726
|
||||
Document.getScore() from Redis : 0.919363 (matched stored question: "How do I reset my account password?")
|
||||
(1 + cosine) / 2 : 0.919363 <- the score is this, not the cosine
|
||||
same question as the cached one -> score 1.000000
|
||||
|
||||
The closest pairs of DIFFERENT cached questions (Spring AI score; 435 pairs in all). Above the threshold, one answers for the other:
|
||||
0.852 "How do I reset my account password?" / "How do I reset two-factor authentication on my account?"
|
||||
0.824 "How do I cancel an order I just placed?" / "How do I cancel a return I already requested?"
|
||||
0.796 "How do I cancel an order I just placed?" / "How do I cancel my monthly subscription?"
|
||||
0.791 "How can I track my order?" / "How can I track the return I sent back?"
|
||||
0.789 "How can I track my order?" / "Can I pick up my order at a store?"
|
||||
0.779 "How long is the warranty on your laptops?" / "How do I make a warranty claim?"
|
||||
0.775 "How long does a refund take to arrive?" / "How can I track the return I sent back?"
|
||||
0.767 "How do I update the firmware on my headphones?" / "How do I pair my headphones over Bluetooth?"
|
||||
|
||||
Spring AI's similarityThreshold is applied to that score. Threshold -> the cosine it really means (cosine = 2 x score - 1):
|
||||
similarityThreshold 0.70 = cosine 0.40
|
||||
similarityThreshold 0.80 = cosine 0.60
|
||||
similarityThreshold 0.85 = cosine 0.70
|
||||
similarityThreshold 0.90 = cosine 0.80
|
||||
similarityThreshold 0.95 = cosine 0.90
|
||||
@@ -0,0 +1,14 @@
|
||||
# SemanticCacheAdvisor in front of a ChatClient (threshold 0.80)
|
||||
|
||||
ask : How do I reset my account password?
|
||||
-> model call, model calls so far 1, answer starts "[reset-password]"
|
||||
ask : How do I reset my account password?
|
||||
-> CACHE HIT, score 1.000, model calls so far 1, answer starts "[reset-password]"
|
||||
ask : I forgot my password, how can I get back into my account?
|
||||
-> CACHE HIT, score 0.919, model calls so far 1, answer starts "[reset-password]"
|
||||
ask : How do I reset my router to factory settings?
|
||||
-> model call, model calls so far 2, answer starts "[reset-router]"
|
||||
ask : What is the capital of Australia?
|
||||
-> model call, model calls so far 3, answer starts "[ood-0]"
|
||||
|
||||
advisor counters: 2 hits, 3 misses; model received 3 calls for 5 questions; entries in Redis: 3
|
||||
@@ -0,0 +1,32 @@
|
||||
# Threshold sweep: 30 cached questions, 90 rewordings and 20 off-topic questions asked
|
||||
|
||||
threshold = the value given to SearchRequest.similarityThreshold (Spring AI score); cosine = 2 x threshold - 1.
|
||||
A served answer is RIGHT if it belongs to the intent of the question asked, WRONG if it belongs to another intent.
|
||||
An off-topic question has no right answer in the cache, so any hit on it is WRONG.
|
||||
|
||||
threshold | cosine | hit rate | right | wrong (reworded) | wrong (off-topic) | wrong / all hits | cached pairs of different intents that answer for each other
|
||||
0.50 | 0.00 | 100% | 86/90 | 4/90 | 20/20 | 21.8% | 396 of 435
|
||||
0.60 | 0.20 | 100% | 86/90 | 4/90 | 6/20 | 10.4% | 151 of 435
|
||||
0.70 | 0.40 | 99% | 86/90 | 3/90 | 0/20 | 3.4% | 27 of 435
|
||||
0.75 | 0.50 | 94% | 83/90 | 2/90 | 0/20 | 2.4% | 13 of 435
|
||||
0.80 | 0.60 | 86% | 77/90 | 0/90 | 0/20 | 0.0% | 2 of 435
|
||||
0.85 | 0.70 | 58% | 52/90 | 0/90 | 0/20 | 0.0% | 1 of 435
|
||||
0.90 | 0.80 | 26% | 23/90 | 0/90 | 0/20 | 0.0% | 0 of 435
|
||||
0.95 | 0.90 | 3% | 3/90 | 0/90 | 0/20 | 0.0% | 0 of 435
|
||||
|
||||
Questions that were answered WRONGLY at threshold 0.80 (asked -> served from): none among these 110 probes
|
||||
|
||||
Rewordings the cache MISSED at threshold 0.80 (best stored question and its score):
|
||||
0.787 "Where can I end my membership plan?" (nearest: "How do I cancel my monthly subscription?")
|
||||
0.746 "Can you tell me what qualifies for getting my money back?" (nearest: "Which items can be refunded?")
|
||||
0.696 "Where is my package right now?" (nearest: "Can I pick up my order at a store?")
|
||||
0.759 "I want to see the delivery status of my purchase" (nearest: "How can I track my order?")
|
||||
0.752 "Has my returned parcel arrived at your warehouse yet?" (nearest: "How long will delivery take?")
|
||||
0.790 "My device broke, how do I get it repaired under warranty?" (nearest: "How do I make a warranty claim?")
|
||||
0.797 "How many days until my parcel gets here?" (nearest: "How long will delivery take?")
|
||||
0.739 "Can I pay with a credit card or PayPal?" (nearest: "Which payment methods do you accept?")
|
||||
0.798 "How can I pay for my order?" (nearest: "How can I track my order?")
|
||||
0.769 "I need a receipt for my purchase, where do I find it?" (nearest: "Where can I download an invoice for my order?")
|
||||
0.743 "Is in-store collection available?" (nearest: "Can I pick up my order at a store?")
|
||||
0.753 "I would rather collect my purchase myself, is that an option?" (nearest: "Can I pick up my order at a store?")
|
||||
0.703 "Do you offer click and collect?" (nearest: "Do you sell gift cards?")
|
||||
@@ -0,0 +1,16 @@
|
||||
# 600 requests through the advisor, by threshold
|
||||
|
||||
Workload: 600 requests (seed 42), 30 intents asked with Zipf popularity in random phrasings, 15% off-topic.
|
||||
121 distinct texts, so a plain exact-match cache could answer at most 479 of 600 (80%).
|
||||
Cost model (ILLUSTRATIVE): $2.50 per million input tokens, $10.00 per million output tokens, tokens = characters / 4.
|
||||
|
||||
setup | model calls | hit rate | wrong | wrong of hits | model $ | saved
|
||||
no cache | 600 | - | - | - | 0.4564 | -
|
||||
cache, threshold 0.70 | 42 | 93.0% | 191 | 34.2% | 0.0316 | 93%
|
||||
cache, threshold 0.80 | 62 | 89.7% | 7 | 1.3% | 0.0468 | 90%
|
||||
cache, threshold 0.90 | 109 | 81.8% | 0 | 0.0% | 0.0827 | 82%
|
||||
cache, threshold 0.95 | 119 | 80.2% | 0 | 0.0% | 0.0904 | 80%
|
||||
cache, exact match only | 121 | 79.8% | 0 | 0.0% | 0.0919 | 80%
|
||||
|
||||
Average time to serve a cache hit (embed the question, search Redis, build the response): 4 ms at 0.70, 4 ms at 0.80, 3 ms at 0.90, 3 ms at 0.95
|
||||
Saved dollars exclude the cost of embedding (local model here, CPU only) and of running Redis.
|
||||
@@ -0,0 +1,10 @@
|
||||
# Pairs that read alike and mean different things (cosine similarity)
|
||||
|
||||
cached question | new question | cosine | Spring AI score = (1 + cosine) / 2
|
||||
How do I cancel my order? | How do I keep my order and not cancel it? | 0.938 | 0.969 <- served at threshold 0.90 and 0.80
|
||||
Is shipping free for orders over $50? | Is shipping free for orders over $500? | 0.848 | 0.924 <- served at threshold 0.90 and 0.80
|
||||
Do you ship to Canada? | Do you ship to Cuba? | 0.645 | 0.822 <- served at threshold 0.80
|
||||
What is the warranty on the laptop? | What is the warranty on the monitor? | 0.799 | 0.899 <- served at threshold 0.80
|
||||
Where is order 1001? | Where is order 2002? | 0.593 | 0.796
|
||||
Can I get a refund within 30 days? | Can I get a refund after 30 days? | 0.984 | 0.992 <- served at threshold 0.90 and 0.80
|
||||
I want to delete my account | I do not want to delete my account | 0.931 | 0.966 <- served at threshold 0.90 and 0.80
|
||||
@@ -0,0 +1,8 @@
|
||||
# Same question, two tenants
|
||||
|
||||
shop-a asks "How do I reset my account password?" -> model call (calls so far: 1)
|
||||
shop-b asks the identical question -> model call, no hit (model calls so far: 2)
|
||||
shop-a asks again -> cache hit
|
||||
entries in the index: 2 (one per tenant)
|
||||
The filter is a tag on the document and an expression on the search: tenant == 'shop-b'.
|
||||
nearest(shop-a, q) without asking as shop-b still finds: "How do I reset my account password?"
|
||||
@@ -0,0 +1,4 @@
|
||||
# Expiry with EXPIRE on the document's key
|
||||
|
||||
stored one entry, index holds 1 document(s); lookup -> hit
|
||||
EXPIRE sc:<id> 1, wait 1.5 s; index now holds 0 document(s); lookup -> miss
|
||||
@@ -0,0 +1,4 @@
|
||||
# An index built for 384 dimensions, written by a model with 8
|
||||
|
||||
index created by MiniLM (384 dimensions); a new application version configures an 8-dimension model against the same index name.
|
||||
add + search -> exception JedisDataException: Error parsing vector similarity query: query vector blob size (32) does not match index's expected size (1536).
|
||||
@@ -0,0 +1,5 @@
|
||||
# Where the milliseconds go on a lookup (110 questions, 30 entries cached, 2 vCPUs)
|
||||
|
||||
embed the question (MiniLM in-process) : median 2.4 ms, p95 3.6 ms
|
||||
Redis KNN search with the tenant filter : median 0.3 ms, p95 2.5 ms (lookup time minus one embedding)
|
||||
Timings drift run to run; the shape (embedding dominates, Redis is small) does not.
|
||||
@@ -0,0 +1,16 @@
|
||||
# A real local model (TinyLlama 1.1B, Q4_0, 60 tokens) behind the advisor, 2 vCPUs
|
||||
|
||||
first ask 2100 ms model "How do I reset my account password?"
|
||||
first ask 1953 ms model "How do I reset my router to factory settings?"
|
||||
first ask 5 ms CACHE "How do I reset two-factor authentication on my account?" <- served the answer to "How do I reset my account password?" (score 0.852), a DIFFERENT question
|
||||
first ask 1902 ms model "How do I cancel an order I just placed?"
|
||||
first ask 1920 ms model "How do I cancel my monthly subscription?"
|
||||
first ask 10 ms CACHE "How do I cancel a return I already requested?" <- served the answer to "How do I cancel an order I just placed?" (score 0.824), a DIFFERENT question
|
||||
reworded 17 ms CACHE "I forgot my password, how can I get back into my account?" <- How do I reset my account password?
|
||||
reworded 9 ms CACHE "What is the way to restore my router to its factory defaults?" <- How do I reset my router to factory settings?
|
||||
reworded 1941 ms model "I lost my phone, how can I reset my 2FA?"
|
||||
reworded 9 ms CACHE "I made an order by mistake, can I cancel it?" <- How do I cancel an order I just placed?
|
||||
reworded 6 ms CACHE "I want to stop my recurring subscription payments" <- How do I cancel my monthly subscription?
|
||||
reworded 2013 ms model "I changed my mind about sending the item back, can I withdraw the return?"
|
||||
|
||||
median model call 1920 ms; median cache hit 9 ms (4 of 6 rewordings hit); first-time questions wrongly answered from the cache: 2 of 6
|
||||
Reference in New Issue
Block a user