Add advanced-rag module: Easy RAG, query compression and expansion, routing, minScore, content injection, chat-memory cost and sources, each with a captured transcript
Co-Authored-By: Claude Sonnet 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_017ic4b85ApxQRK2Z8yuzC5r
This commit is contained in:
@@ -0,0 +1,35 @@
|
||||
Easy RAG ingestion of kb/ (6 files)
|
||||
segments stored: 6
|
||||
|
||||
Retrieved for "How long are webhooks retried?":
|
||||
0.873 webhooks.txt
|
||||
0.820 refund-policy.txt
|
||||
0.811 account-security.txt
|
||||
|
||||
User message the LLM actually received:
|
||||
How long are webhooks retried?
|
||||
|
||||
Answer using the following information:
|
||||
Acme Cloud Webhooks
|
||||
|
||||
Every webhook delivery is signed. The signature is sent in the X-Acme-Signature header and is an HMAC-SHA256 of the raw request body using your endpoint secret. Reject any delivery whose signature does not match.
|
||||
|
||||
If your endpoint does not answer with a 2xx status within 10 seconds, the delivery is retried up to 8 times with exponential backoff. The retries are spread over 24 hours. After the last failed attempt the endpoint is marked as failing and an email is sent to the organisation owner.
|
||||
|
||||
Deliveries can arrive more than once and out of order, so handlers must be idempotent. Use the event id to deduplicate.
|
||||
|
||||
Acme Cloud Refund Policy
|
||||
|
||||
Annual plans can be refunded in full within 30 days of purchase. Monthly plans can be refunded within 7 days of the first charge. After those windows, no refund is issued, but you can cancel at any time and keep access until the end of the paid period.
|
||||
|
||||
Once a refund is approved, it takes 5 to 10 business days to appear on your card statement. The time depends on your bank, not on Acme Cloud. Refunds always go back to the original payment method; we cannot pay them out to a different card.
|
||||
|
||||
If a refund does not show up after 10 business days, contact support with the invoice number so we can send you the acquirer reference number.
|
||||
|
||||
Acme Cloud Account Security
|
||||
|
||||
Password reset links expire after 30 minutes and can be used only once. If a link has expired, request a new one from the sign-in page.
|
||||
|
||||
Two-factor authentication is mandatory for organisation owners. Supported methods are authenticator apps and hardware security keys. SMS codes are not supported.
|
||||
|
||||
SSO via SAML 2.0 is available on the Business plan. After SSO is enforced, password sign-in is disabled for every member except the organisation owner, who keeps a break-glass login.
|
||||
@@ -0,0 +1,12 @@
|
||||
Turn 1: "What is the refund window for annual plans?"
|
||||
Turn 2 (follow-up): "How long does it take to show up?"
|
||||
|
||||
Naive RAG searches with the follow-up exactly as typed:
|
||||
0.821 hardware-returns.txt
|
||||
0.817 deployments.txt
|
||||
0.816 refund-policy.txt
|
||||
|
||||
Searching with a self-contained rewrite instead ("How long does a refund for an annual plan take to show up on my card?"):
|
||||
0.918 refund-policy.txt
|
||||
0.857 hardware-returns.txt
|
||||
0.795 deployments.txt
|
||||
@@ -0,0 +1,83 @@
|
||||
Rewriter LLM calls after turn 1: 0
|
||||
Rewriter LLM calls after turn 2: 1
|
||||
|
||||
Prompt CompressingQueryTransformer sent to the rewriter on turn 2:
|
||||
Read and understand the conversation between the User and the AI. Then, analyze the new query from the User. Identify all relevant details, terms, and context from both the conversation and the new query. Reformulate this query into a clear, concise, and self-contained format suitable for information retrieval.
|
||||
|
||||
Conversation:
|
||||
User: What is the refund window for annual plans?
|
||||
|
||||
Answer using the following information:
|
||||
Acme Cloud Refund Policy
|
||||
|
||||
Annual plans can be refunded in full within 30 days of purchase. Monthly plans can be refunded within 7 days of the first charge. After those windows, no refund is issued, but you can cancel at any time and keep access until the end of the paid period.
|
||||
|
||||
Once a refund is approved, it takes 5 to 10 business days to appear on your card statement. The time depends on your bank, not on Acme Cloud. Refunds always go back to the original payment method; we cannot pay them out to a different card.
|
||||
|
||||
If a refund does not show up after 10 business days, contact support with the invoice number so we can send you the acquirer reference number.
|
||||
|
||||
Acme Edge Box Hardware Returns
|
||||
|
||||
The Acme Edge Box can be returned within 45 days of delivery if it is in its original packaging. Request a return label from the support portal; the label is valid for 14 days.
|
||||
|
||||
After our warehouse receives the box, the refund is processed within 3 business days. It then follows the normal card timeline of 5 to 10 business days.
|
||||
|
||||
Boxes that arrive damaged or with missing power adapters are inspected first, which can add up to 5 business days.
|
||||
|
||||
Acme Cloud Deployments
|
||||
|
||||
A standard deployment takes about 4 minutes from push to live. The first 5 percent of traffic goes to the new version as a canary for 2 minutes before the rest is shifted.
|
||||
|
||||
A rollback takes about 30 seconds because the previous version stays warm for one hour after every deployment. After that hour, a rollback is a normal 4 minute deployment of the older build.
|
||||
|
||||
Deployments are blocked automatically when the error rate of the canary is more than 2 percent above the baseline.
|
||||
AI: (scripted answer)
|
||||
|
||||
User query: How long does it take to show up?
|
||||
|
||||
It is very important that you provide only reformulated query and nothing else! Do not prepend a query with anything!
|
||||
|
||||
User message the answering LLM received on turn 2:
|
||||
How long does it take to show up?
|
||||
|
||||
Answer using the following information:
|
||||
Acme Cloud Refund Policy
|
||||
|
||||
Annual plans can be refunded in full within 30 days of purchase. Monthly plans can be refunded within 7 days of the first charge. After those windows, no refund is issued, but you can cancel at any time and keep access until the end of the paid period.
|
||||
|
||||
Once a refund is approved, it takes 5 to 10 business days to appear on your card statement. The time depends on your bank, not on Acme Cloud. Refunds always go back to the original payment method; we cannot pay them out to a different card.
|
||||
|
||||
If a refund does not show up after 10 business days, contact support with the invoice number so we can send you the acquirer reference number.
|
||||
|
||||
Acme Edge Box Hardware Returns
|
||||
|
||||
The Acme Edge Box can be returned within 45 days of delivery if it is in its original packaging. Request a return label from the support portal; the label is valid for 14 days.
|
||||
|
||||
After our warehouse receives the box, the refund is processed within 3 business days. It then follows the normal card timeline of 5 to 10 business days.
|
||||
|
||||
Boxes that arrive damaged or with missing power adapters are inspected first, which can add up to 5 business days.
|
||||
|
||||
Acme Cloud Deployments
|
||||
|
||||
A standard deployment takes about 4 minutes from push to live. The first 5 percent of traffic goes to the new version as a canary for 2 minutes before the rest is shifted.
|
||||
|
||||
A rollback takes about 30 seconds because the previous version stays warm for one hour after every deployment. After that hour, a rollback is a normal 4 minute deployment of the older build.
|
||||
|
||||
Deployments are blocked automatically when the error rate of the canary is more than 2 percent above the baseline.
|
||||
|
||||
Rewriter prompt size in characters, turn 2 / 3 / 4: 2256 / 3998 / 5748
|
||||
|
||||
With storeRetrievedContentInChatMemory(false), turn 2 / 3 / 4: 565 / 616 / 675
|
||||
|
||||
Prompt sent to the rewriter on turn 2 with storeRetrievedContentInChatMemory(false):
|
||||
Read and understand the conversation between the User and the AI. Then, analyze the new query from the User. Identify all relevant details, terms, and context from both the conversation and the new query. Reformulate this query into a clear, concise, and self-contained format suitable for information retrieval.
|
||||
|
||||
Conversation:
|
||||
User: What is the refund window for annual plans?
|
||||
AI: (scripted answer)
|
||||
|
||||
User query: How long does it take to show up?
|
||||
|
||||
It is very important that you provide only reformulated query and nothing else! Do not prepend a query with anything!
|
||||
|
||||
User message the answering LLM still received on turn 4 (augmented: true)
|
||||
@@ -0,0 +1,62 @@
|
||||
=== Case 1: a query whose wording misses the document ===
|
||||
Original query: I shipped a bad release, can I undo it?
|
||||
Single-query top 3:
|
||||
0.807 hardware-returns.txt
|
||||
0.790 webhooks.txt
|
||||
0.780 refund-policy.txt
|
||||
|
||||
Queries produced: 3
|
||||
|
||||
"How can I revert a faulty release?"
|
||||
0.798 deployments.txt
|
||||
0.781 hardware-returns.txt
|
||||
0.779 refund-policy.txt
|
||||
|
||||
"Is it possible to roll back a bad deployment?"
|
||||
0.858 deployments.txt
|
||||
0.779 webhooks.txt
|
||||
0.776 refund-policy.txt
|
||||
|
||||
"What is the way to reverse a release that went wrong?"
|
||||
0.788 deployments.txt
|
||||
0.779 refund-policy.txt
|
||||
0.774 hardware-returns.txt
|
||||
|
||||
After DefaultContentAggregator (reciprocal rank fusion): 4 unique segments, in this order:
|
||||
deployments.txt
|
||||
refund-policy.txt
|
||||
hardware-returns.txt
|
||||
webhooks.txt
|
||||
|
||||
=== Case 2: a query that was already fine ===
|
||||
Original query: How do I get my money back?
|
||||
Single-query top 3:
|
||||
0.852 refund-policy.txt
|
||||
0.804 hardware-returns.txt
|
||||
0.763 webhooks.txt
|
||||
|
||||
Queries produced: 3
|
||||
|
||||
"What is the process for requesting a refund?"
|
||||
0.866 refund-policy.txt
|
||||
0.818 hardware-returns.txt
|
||||
0.787 webhooks.txt
|
||||
|
||||
"How can I be reimbursed for a purchase?"
|
||||
0.852 refund-policy.txt
|
||||
0.805 hardware-returns.txt
|
||||
0.764 webhooks.txt
|
||||
|
||||
"Steps to cancel and receive a refund"
|
||||
0.872 refund-policy.txt
|
||||
0.817 hardware-returns.txt
|
||||
0.797 webhooks.txt
|
||||
|
||||
After DefaultContentAggregator (reciprocal rank fusion): 3 unique segments, in this order:
|
||||
refund-policy.txt
|
||||
hardware-returns.txt
|
||||
webhooks.txt
|
||||
|
||||
Prompt ExpandingQueryTransformer sent to the expander:
|
||||
Generate 3 different versions of a provided user query. Each version should be worded differently, using synonyms or alternative sentence structures, but they should all retain the original meaning. These versions will be used to retrieve relevant documents. It is very important to provide each query version on a separate line, without enumerations, hyphens, or any additional formatting!
|
||||
User query: I shipped a bad release, can I undo it?
|
||||
@@ -0,0 +1,23 @@
|
||||
Query: How long are webhooks retried?
|
||||
|
||||
Searching the single merged store:
|
||||
0.873 webhooks.txt
|
||||
0.820 refund-policy.txt
|
||||
0.811 account-security.txt
|
||||
|
||||
Prompt LanguageModelQueryRouter sent to the router LLM:
|
||||
Based on the user query, determine the most suitable data source(s) to retrieve relevant information from the following options:
|
||||
1: Billing, refunds, hardware returns and account security for customers
|
||||
2: API rate limits, deployments and webhooks for developers
|
||||
It is very important that your answer consists of either a single number or multiple numbers separated by commas and nothing else!
|
||||
User query: How long are webhooks retried?
|
||||
|
||||
Router LLM answered "2" -> retrievers chosen: 1
|
||||
0.873 webhooks.txt
|
||||
0.802 deployments.txt
|
||||
0.786 rate-limits.txt
|
||||
|
||||
Router LLM answered "The engineering one, I think.":
|
||||
default fallback strategy -> retrievers chosen: 0
|
||||
ROUTE_TO_ALL -> retrievers chosen: 2
|
||||
FAIL -> java.lang.RuntimeException: java.lang.NumberFormatException: For input string: "The engineering one"
|
||||
@@ -0,0 +1,42 @@
|
||||
Default injector:
|
||||
How long are webhooks retried?
|
||||
|
||||
Answer using the following information:
|
||||
Acme Cloud Webhooks
|
||||
|
||||
Every webhook delivery is signed. The signature is sent in the X-Acme-Signature header and is an HMAC-SHA256 of the raw request body using your endpoint secret. Reject any delivery whose signature does not match.
|
||||
|
||||
If your endpoint does not answer with a 2xx status within 10 seconds, the delivery is retried up to 8 times with exponential backoff. The retries are spread over 24 hours. After the last failed attempt the endpoint is marked as failing and an email is sent to the organisation owner.
|
||||
|
||||
Deliveries can arrive more than once and out of order, so handlers must be idempotent. Use the event id to deduplicate.
|
||||
|
||||
Acme Cloud Refund Policy
|
||||
|
||||
Annual plans can be refunded in full within 30 days of purchase. Monthly plans can be refunded within 7 days of the first charge. After those windows, no refund is issued, but you can cancel at any time and keep access until the end of the paid period.
|
||||
|
||||
Once a refund is approved, it takes 5 to 10 business days to appear on your card statement. The time depends on your bank, not on Acme Cloud. Refunds always go back to the original payment method; we cannot pay them out to a different card.
|
||||
|
||||
If a refund does not show up after 10 business days, contact support with the invoice number so we can send you the acquirer reference number.
|
||||
|
||||
Custom template + metadataKeysToInclude(["file_name"]):
|
||||
How long are webhooks retried?
|
||||
|
||||
Answer ONLY from the sources below. If they do not contain the answer, say you do not know.
|
||||
|
||||
content: Acme Cloud Webhooks
|
||||
|
||||
Every webhook delivery is signed. The signature is sent in the X-Acme-Signature header and is an HMAC-SHA256 of the raw request body using your endpoint secret. Reject any delivery whose signature does not match.
|
||||
|
||||
If your endpoint does not answer with a 2xx status within 10 seconds, the delivery is retried up to 8 times with exponential backoff. The retries are spread over 24 hours. After the last failed attempt the endpoint is marked as failing and an email is sent to the organisation owner.
|
||||
|
||||
Deliveries can arrive more than once and out of order, so handlers must be idempotent. Use the event id to deduplicate.
|
||||
file_name: webhooks.txt
|
||||
|
||||
content: Acme Cloud Refund Policy
|
||||
|
||||
Annual plans can be refunded in full within 30 days of purchase. Monthly plans can be refunded within 7 days of the first charge. After those windows, no refund is issued, but you can cancel at any time and keep access until the end of the paid period.
|
||||
|
||||
Once a refund is approved, it takes 5 to 10 business days to appear on your card statement. The time depends on your bank, not on Acme Cloud. Refunds always go back to the original payment method; we cannot pay them out to a different card.
|
||||
|
||||
If a refund does not show up after 10 business days, contact support with the invoice number so we can send you the acquirer reference number.
|
||||
file_name: refund-policy.txt
|
||||
@@ -0,0 +1,51 @@
|
||||
Off-topic: What is the capital of France?
|
||||
minScore 0.0 (default):
|
||||
0.696 deployments.txt
|
||||
0.670 rate-limits.txt
|
||||
0.669 account-security.txt
|
||||
minScore 0.75:
|
||||
(nothing)
|
||||
|
||||
Message sent to the LLM with minScore 0.0:
|
||||
What is the capital of France?
|
||||
|
||||
Answer using the following information:
|
||||
Acme Cloud Deployments
|
||||
|
||||
A standard deployment takes about 4 minutes from push to live. The first 5 percent of traffic goes to the new version as a canary for 2 minutes before the rest is shifted.
|
||||
|
||||
A rollback takes about 30 seconds because the previous version stays warm for one hour after every deployment. After that hour, a rollback is a normal 4 minute deployment of the older build.
|
||||
|
||||
Deployments are blocked automatically when the error rate of the canary is more than 2 percent above the baseline.
|
||||
|
||||
Acme Cloud API Rate Limits
|
||||
|
||||
Every API key is limited to 600 requests per minute, with a burst allowance of 100 requests per second. Requests over the limit receive HTTP 429 with a Retry-After header giving the number of seconds to wait.
|
||||
|
||||
Limits are per key, not per organisation, so a noisy integration cannot starve the others. Batch endpoints count as one request regardless of how many items they contain.
|
||||
|
||||
To raise a limit, open a ticket with the key id and the peak traffic you expect. Increases are applied within one business day.
|
||||
|
||||
Acme Cloud Account Security
|
||||
|
||||
Password reset links expire after 30 minutes and can be used only once. If a link has expired, request a new one from the sign-in page.
|
||||
|
||||
Two-factor authentication is mandatory for organisation owners. Supported methods are authenticator apps and hardware security keys. SMS codes are not supported.
|
||||
|
||||
SSO via SAML 2.0 is available on the Business plan. After SSO is enforced, password sign-in is disabled for every member except the organisation owner, who keeps a break-glass login.
|
||||
|
||||
Message sent to the LLM with minScore 0.75:
|
||||
What is the capital of France?
|
||||
|
||||
On-topic: How long does a refund take to show up on my card?
|
||||
minScore 0.0:
|
||||
0.906 refund-policy.txt
|
||||
0.868 hardware-returns.txt
|
||||
0.782 webhooks.txt
|
||||
minScore 0.75:
|
||||
0.906 refund-policy.txt
|
||||
0.868 hardware-returns.txt
|
||||
0.782 webhooks.txt
|
||||
minScore 0.85:
|
||||
0.906 refund-policy.txt
|
||||
0.868 hardware-returns.txt
|
||||
@@ -0,0 +1,34 @@
|
||||
LLM calls per turn (compress + route + answer):
|
||||
turn 1 rewriter=0 router=1 answer=1
|
||||
turn 2 rewriter=1 router=1 answer=1
|
||||
|
||||
User message the answering LLM received on turn 2:
|
||||
How long does it take to show up?
|
||||
|
||||
Answer using the following information:
|
||||
content: Acme Cloud Refund Policy
|
||||
|
||||
Annual plans can be refunded in full within 30 days of purchase. Monthly plans can be refunded within 7 days of the first charge. After those windows, no refund is issued, but you can cancel at any time and keep access until the end of the paid period.
|
||||
|
||||
Once a refund is approved, it takes 5 to 10 business days to appear on your card statement. The time depends on your bank, not on Acme Cloud. Refunds always go back to the original payment method; we cannot pay them out to a different card.
|
||||
|
||||
If a refund does not show up after 10 business days, contact support with the invoice number so we can send you the acquirer reference number.
|
||||
file_name: refund-policy.txt
|
||||
|
||||
content: Acme Edge Box Hardware Returns
|
||||
|
||||
The Acme Edge Box can be returned within 45 days of delivery if it is in its original packaging. Request a return label from the support portal; the label is valid for 14 days.
|
||||
|
||||
After our warehouse receives the box, the refund is processed within 3 business days. It then follows the normal card timeline of 5 to 10 business days.
|
||||
|
||||
Boxes that arrive damaged or with missing power adapters are inspected first, which can add up to 5 business days.
|
||||
file_name: hardware-returns.txt
|
||||
|
||||
content: Acme Cloud Account Security
|
||||
|
||||
Password reset links expire after 30 minutes and can be used only once. If a link has expired, request a new one from the sign-in page.
|
||||
|
||||
Two-factor authentication is mandatory for organisation owners. Supported methods are authenticator apps and hardware security keys. SMS codes are not supported.
|
||||
|
||||
SSO via SAML 2.0 is available on the Business plan. After SSO is enforced, password sign-in is disabled for every member except the organisation owner, who keeps a break-glass login.
|
||||
file_name: account-security.txt
|
||||
@@ -0,0 +1,3 @@
|
||||
loadDocuments("kb") -> 0 documents
|
||||
loadDocumentsRecursively("kb") -> 6 documents
|
||||
ingest(flat, store) -> java.lang.IllegalArgumentException: segments cannot be null or empty
|
||||
@@ -0,0 +1,4 @@
|
||||
answer: (scripted answer)
|
||||
sources:
|
||||
0.873 webhooks.txt
|
||||
0.820 refund-policy.txt
|
||||
@@ -0,0 +1,4 @@
|
||||
EmbeddingStoreContentRetriever.DEFAULT_MAX_RESULTS = 3
|
||||
EmbeddingStoreContentRetriever.DEFAULT_MIN_SCORE = 0.0
|
||||
ExpandingQueryTransformer.DEFAULT_N = 3
|
||||
LanguageModelQueryRouter.FallbackStrategy = [DO_NOT_ROUTE, ROUTE_TO_ALL, FAIL]
|
||||
Reference in New Issue
Block a user