Add rag module: Spring AI 2.0 RAG with pgvector, chunking, reranking and a faithfulness check
Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01B38FGKKam5SCGgwgduVAh3
This commit is contained in:
Claude
@@ -0,0 +1,13 @@
|
||||
# The pipeline as Spring beans (the shipped configuration, fake models)
|
||||
|
||||
stage / role bean name actual class
|
||||
DocumentTransformer chunker TokenTextSplitter
|
||||
EmbeddingModel embeddingModel HashingEmbeddingModel
|
||||
VectorStore vectorStore PgVectorStore
|
||||
RetrievalAugmentationAdvisor retrievalAdvisor RetrievalAugmentationAdvisor
|
||||
LlmReranker reranker LlmReranker
|
||||
ChatModel chatModel FakeChatModel
|
||||
ChatClient ragChatClient DefaultChatClient
|
||||
FactCheckingEvaluator factChecker FactCheckingEvaluator
|
||||
IngestionService ingestionService IngestionService
|
||||
RagQueryService ragQueryService RagQueryService
|
||||
@@ -0,0 +1,27 @@
|
||||
# What TokenTextSplitter does to a 40-sentence document
|
||||
|
||||
document: 4268 characters, 880 cl100k_base tokens
|
||||
|
||||
--- new TokenTextSplitter() (defaults: 800 tokens, 350 min chars, 5 min length to embed) ---
|
||||
chunks: 2, sizes in tokens: [792, 88]
|
||||
|
||||
--- chunk size 100 tokens ---
|
||||
chunks: 10
|
||||
chunk 0: 88 tokens, 419 chars, starts "Sentence 1 describes t", ends "ry full-time employee."
|
||||
chunk 1: 88 tokens, 420 chars, starts "Sentence 5 describes t", ends "ry full-time employee."
|
||||
chunk 2: 88 tokens, 426 chars, starts "Sentence 9 describes t", ends "ry full-time employee."
|
||||
chunk 3: 88 tokens, 428 chars, starts "Sentence 13 describes ", ends "ry full-time employee."
|
||||
chunk 4: 88 tokens, 427 chars, starts "Sentence 17 describes ", ends "ry full-time employee."
|
||||
chunk 5: 88 tokens, 427 chars, starts "Sentence 21 describes ", ends "ry full-time employee."
|
||||
chunk 6: 88 tokens, 428 chars, starts "Sentence 25 describes ", ends "ry full-time employee."
|
||||
chunk 7: 88 tokens, 428 chars, starts "Sentence 29 describes ", ends "ry full-time employee."
|
||||
chunk 8: 88 tokens, 428 chars, starts "Sentence 33 describes ", ends "ry full-time employee."
|
||||
chunk 9: 88 tokens, 427 chars, starts "Sentence 37 describes ", ends "ry full-time employee."
|
||||
metadata of chunk 1 (parent_document_id, a random UUID, left out): {chunk_index=1, page_number=1, source_file=long.txt, total_chunks=10}
|
||||
|
||||
--- is there any overlap between neighbouring chunks? ---
|
||||
boundaries where the first 30 characters of a chunk already appear in the chunk before it: 0 of 9
|
||||
|
||||
--- minChunkSizeChars: where the cut lands ---
|
||||
minChunkSizeChars=350: 10 chunks, 9 of the first 9 end on a full stop
|
||||
minChunkSizeChars= 50: 10 chunks, 9 of the first 9 end on a full stop
|
||||
@@ -0,0 +1,31 @@
|
||||
# RecursiveChunker and SemanticChunker
|
||||
|
||||
|
||||
--- RecursiveChunker(maxChars=400, overlapChars=80) ---
|
||||
chunks: 14, longest: 395 characters
|
||||
chunk 0: 314 chars, starts "Sentence 1 describes the", ends "very full-time employee."
|
||||
chunk 1: 388 chars, starts "for item 3 and states th", ends "very full-time employee."
|
||||
chunk 2: 387 chars, starts "for item 6 and states th", ends "very full-time employee."
|
||||
chunk 3: 394 chars, starts "for item 9 and states th", ends "very full-time employee."
|
||||
chunk 4: 394 chars, starts "for item 12 and states t", ends "very full-time employee."
|
||||
chunk 5: 395 chars, starts "for item 15 and states t", ends "very full-time employee."
|
||||
chunk 6: 395 chars, starts "for item 18 and states t", ends "very full-time employee."
|
||||
chunk 7: 394 chars, starts "for item 21 and states t", ends "very full-time employee."
|
||||
chunk 8: 395 chars, starts "for item 24 and states t", ends "very full-time employee."
|
||||
chunk 9: 394 chars, starts "for item 27 and states t", ends "very full-time employee."
|
||||
chunk 10: 395 chars, starts "for item 30 and states t", ends "very full-time employee."
|
||||
chunk 11: 395 chars, starts "for item 33 and states t", ends "very full-time employee."
|
||||
chunk 12: 394 chars, starts "for item 36 and states t", ends "very full-time employee."
|
||||
chunk 13: 180 chars, starts "for item 39 and states t", ends "very full-time employee."
|
||||
metadata of chunk 2: {chunk_index=2, chunk_total=14, page_number=1, source_file=long.txt}
|
||||
boundaries where the next chunk opens with words the previous one ended with: 13 of 13
|
||||
|
||||
--- one 900-character word-salad with no separators falls back to a hard cut ---
|
||||
chunks: [400, 400, 100]
|
||||
|
||||
--- SemanticChunker(distance 0.9) on three topics, four sentences each ---
|
||||
chunks: 3
|
||||
[0] Annual leave is twenty days per year. Unused annual leave carries over until March. Leave requests go through the HR portal. Annual leave accrues monthly.
|
||||
[1] Expenses need a receipt above fifty euros. Expense claims must be filed within thirty days. Receipts for expenses are uploaded as photos. Approved expenses are paid with salary.
|
||||
[2] Remote work is allowed two days per week. Managers agree the remote work days. Remote work needs a quiet workspace. Remote work days are recorded in the calendar.
|
||||
texts sent to the embedding model: 12 (12 sentences, one batched call)
|
||||
@@ -0,0 +1,27 @@
|
||||
# PagePdfDocumentReader: what one PDF page becomes
|
||||
|
||||
pages in the PDF: 4, documents read: 4
|
||||
page document metadata: {page_number=1}
|
||||
page document metadata: {page_number=2}
|
||||
page document metadata: {page_number=3}
|
||||
page document metadata: {page_number=4}
|
||||
|
||||
--- page 2 as read (single spaces as dots, runs of 4+ as [n spaces], line ends as a pilcrow) ---
|
||||
¶
|
||||
[12 spaces]4.1··Annual···Leave···Entitlement.···Full-time···employees[5 spaces]are··entitled·to·20··working···days[104 spaces]¶
|
||||
[12 spaces]of·annual···leave··per··calendar···year.··Part-time···employees[5 spaces]receive···leave··pro··rata.[107 spaces]¶
|
||||
¶
|
||||
[12 spaces]4.2··Leave···Carryover.···Unused[5 spaces]annual···leave··may···be··carried··over··for·a·maximum[6 spaces]of·5·days[96 spaces]¶
|
||||
[12 spaces]into·the··next··calendar···year··and···must··be··used···by·31··March.[133 spaces]¶
|
||||
|
||||
|
||||
--- page 2 after IngestionService.tidy ---
|
||||
4.1·Annual·Leave·Entitlement.·Full-time·employees·are·entitled·to·20·working·days¶
|
||||
of·annual·leave·per·calendar·year.·Part-time·employees·receive·leave·pro·rata.¶
|
||||
¶
|
||||
4.2·Leave·Carryover.·Unused·annual·leave·may·be·carried·over·for·a·maximum·of·5·days¶
|
||||
into·the·next·calendar·year·and·must·be·used·by·31·March.
|
||||
|
||||
--- size of page 2 ---
|
||||
as read : 862 characters, longest run of spaces 133
|
||||
tidied : 304 characters, longest run of spaces 1
|
||||
@@ -0,0 +1,14 @@
|
||||
# Ingesting the same handbook more than once (real PostgreSQL + pgvector)
|
||||
|
||||
rows are counted with: select count(*) from vector_store
|
||||
1. first upload -> status=ingested chunksWritten=4 chunksReplaced=0 | rows in table=4, texts embedded so far=4
|
||||
2. same bytes again -> status=skipped chunksWritten=0 chunksReplaced=0 | rows in table=4, texts embedded so far=4
|
||||
3. page 2 edited (20 -> 22 days) -> status=updated chunksWritten=4 chunksReplaced=4 | rows in table=4, texts embedded so far=8
|
||||
rows still saying "20 working days": 0, rows saying "22 working days": 1
|
||||
4. same text, exported again -> status=updated chunksWritten=4 chunksReplaced=4 | rows in table=4, texts embedded so far=12
|
||||
the two PDFs have identical text and different bytes: true
|
||||
the file hash is a hash of bytes, so a re-export counts as a change and is re-embedded
|
||||
|
||||
--- the naive version: vectorStore.add() on every upload, nothing remembered ---
|
||||
the same 4 chunks added on two more uploads: rows in table 4 -> 12
|
||||
rows saying "22 working days" now: 3
|
||||
@@ -0,0 +1,61 @@
|
||||
# Retrieval: threshold, prompt shape and the empty-context path
|
||||
|
||||
the store holds the 4 pages of one handbook as 4 chunks; topK = 20
|
||||
|
||||
--- question: "How many days of annual leave do employees get?" ---
|
||||
similarityThreshold 0.0 (the default): 4 chunk(s)
|
||||
score 0.5891 page 2 "4.1 Annual Leave Entitlement. Full-time empl..."
|
||||
score 0.4077 page 3 "4.3 Requesting Leave. All leave requests mus..."
|
||||
score 0.2535 page 1 "Acme Employee Handbook 2026 3.5 Probationar..."
|
||||
score 0.2023 page 4 "6.1 Expenses. Receipts are required for ever..."
|
||||
similarityThreshold 0.3: 2 chunk(s)
|
||||
score 0.5891 page 2 "4.1 Annual Leave Entitlement. Full-time empl..."
|
||||
score 0.4077 page 3 "4.3 Requesting Leave. All leave requests mus..."
|
||||
|
||||
--- question: "What is the capital of Mongolia?" ---
|
||||
similarityThreshold 0.0 (the default): 4 chunk(s)
|
||||
score 0.0000 page 1 "Acme Employee Handbook 2026 3.5 Probationar..."
|
||||
score 0.0000 page 2 "4.1 Annual Leave Entitlement. Full-time empl..."
|
||||
score 0.0000 page 3 "4.3 Requesting Leave. All leave requests mus..."
|
||||
score 0.0000 page 4 "6.1 Expenses. Receipts are required for ever..."
|
||||
similarityThreshold 0.3: 0 chunk(s)
|
||||
|
||||
--- the prompt the model receives (threshold 0.3, question about leave) ---
|
||||
Context information is below.
|
||||
|
||||
---------------------
|
||||
4.1 Annual Leave Entitlement. Full-time employees are entitled to 20 working days
|
||||
of annual leave per calendar year. Part-time employees receive leave pro rata.
|
||||
|
||||
4.2 Leave Carryover. Unused annual leave may be carried over for a maximum of 5 days
|
||||
into the next calendar year and must be used by 31 March.
|
||||
4.3 Requesting Leave. All leave requests must be submitted through the HR portal
|
||||
at least two weeks in advance for absences longer than three days.
|
||||
|
||||
4.7 Sick Leave. Sick leave is separate from annual leave and is not deducted from it.
|
||||
A medical certificate is required after three consecutive days.
|
||||
---------------------
|
||||
|
||||
Given the context information and no prior knowledge, answer the query.
|
||||
|
||||
Follow these rules:
|
||||
|
||||
1. If the answer is not in the context, just say that you don't know.
|
||||
2. Avoid statements like "Based on the context..." or "The provided information...".
|
||||
|
||||
Query: How many days of annual leave do employees get?
|
||||
|
||||
Answer:
|
||||
|
||||
|
||||
--- nothing retrieved, allowEmptyContext(false): the prompt the model receives ---
|
||||
The user query is outside your knowledge base.
|
||||
Politely inform the user that you can't answer it.
|
||||
|
||||
|
||||
--- nothing retrieved, allowEmptyContext(true): the prompt the model receives ---
|
||||
What is the capital of Mongolia?
|
||||
|
||||
--- off-topic question, default threshold 0.0, allowEmptyContext(false) ---
|
||||
chunks placed in the prompt: 4 of 4
|
||||
the empty-context safety net fired: false
|
||||
@@ -0,0 +1,42 @@
|
||||
# Tenant isolation: metadata filters, and a filter built from user input
|
||||
|
||||
two tenants each upload a handbook with a section 4.1 on annual leave
|
||||
acme says 20 working days, globex says 25
|
||||
|
||||
--- SimpleVectorStore (in memory) ---
|
||||
no filter, top 3: 3 chunk(s)
|
||||
tenant=acme page=2 "4.1 Annual Leave Entitlement. Full-time ..."
|
||||
tenant=acme page=3 "4.3 Requesting Leave. All leave requests..."
|
||||
tenant=globex page=1 "Globex Staff Manual 2026 4.1 Annual Lea..."
|
||||
eq("tenant_id", "acme") built with FilterExpressionBuilder, top 3: 3 chunk(s)
|
||||
tenant=acme page=2 "4.1 Annual Leave Entitlement. Full-time ..."
|
||||
tenant=acme page=3 "4.3 Requesting Leave. All leave requests..."
|
||||
tenant=acme page=1 "Acme Employee Handbook 2026 3.5 Probati..."
|
||||
filter string built by concatenation: tenant_id == 'globex' || tenant_id == 'acme'
|
||||
result, top 5: 5 chunk(s)
|
||||
tenant=acme page=2 "4.1 Annual Leave Entitlement. Full-time ..."
|
||||
tenant=acme page=3 "4.3 Requesting Leave. All leave requests..."
|
||||
tenant=globex page=1 "Globex Staff Manual 2026 4.1 Annual Lea..."
|
||||
tenant=acme page=1 "Acme Employee Handbook 2026 3.5 Probati..."
|
||||
tenant=acme page=4 "6.1 Expenses. Receipts are required for ..."
|
||||
same text passed to FilterExpressionBuilder.eq(), top 5: 0 chunk(s)
|
||||
double-quote variant passed to FilterExpressionBuilder.eq(), top 5: 0 chunk(s)
|
||||
|
||||
--- PgVectorStore (PostgreSQL + pgvector) ---
|
||||
no filter, top 3: 3 chunk(s)
|
||||
tenant=acme page=2 "4.1 Annual Leave Entitlement. Full-time ..."
|
||||
tenant=acme page=3 "4.3 Requesting Leave. All leave requests..."
|
||||
tenant=globex page=1 "Globex Staff Manual 2026 4.1 Annual Lea..."
|
||||
eq("tenant_id", "acme") built with FilterExpressionBuilder, top 3: 3 chunk(s)
|
||||
tenant=acme page=2 "4.1 Annual Leave Entitlement. Full-time ..."
|
||||
tenant=acme page=3 "4.3 Requesting Leave. All leave requests..."
|
||||
tenant=acme page=1 "Acme Employee Handbook 2026 3.5 Probati..."
|
||||
filter string built by concatenation: tenant_id == 'globex' || tenant_id == 'acme'
|
||||
result, top 5: 5 chunk(s)
|
||||
tenant=acme page=2 "4.1 Annual Leave Entitlement. Full-time ..."
|
||||
tenant=acme page=3 "4.3 Requesting Leave. All leave requests..."
|
||||
tenant=globex page=1 "Globex Staff Manual 2026 4.1 Annual Lea..."
|
||||
tenant=acme page=1 "Acme Employee Handbook 2026 3.5 Probati..."
|
||||
tenant=acme page=4 "6.1 Expenses. Receipts are required for ..."
|
||||
same text passed to FilterExpressionBuilder.eq(), top 5: 0 chunk(s)
|
||||
double-quote variant passed to FilterExpressionBuilder.eq(), top 5: 0 chunk(s)
|
||||
@@ -0,0 +1,29 @@
|
||||
# LLM reranking: calls, order, latency and failure
|
||||
|
||||
|
||||
--- one model call per candidate; only topN survive (topN = 2) ---
|
||||
candidates in: 4, model calls made: 4, chunks out: 2
|
||||
order from the vector search, best first:
|
||||
similarity 0.5891 page 2
|
||||
similarity 0.4077 page 3
|
||||
similarity 0.2535 page 1
|
||||
similarity 0.2023 page 4
|
||||
order after reranking, best first:
|
||||
rerank_score 8 page 2
|
||||
rerank_score 6 page 3
|
||||
the rating prompt for the page 2 candidate (calls run concurrently, so pick it by content):
|
||||
Rate how well the PASSAGE helps answer the QUESTION, from 0 (irrelevant) to 10 (answers it).
|
||||
Reply with a single integer and nothing else.
|
||||
QUESTION: How many days of annual leave do employees get?
|
||||
PASSAGE: 4.1 Annual Leave Entitlement. Full-time employees are entitled to 20 working days
|
||||
|
||||
--- latency: 20 candidates, each rating call takes 200 ms (a Thread.sleep in the fake model) ---
|
||||
the 20 calls one after another take 4000 ms or more: true
|
||||
LlmReranker, one virtual thread per candidate, takes under 1000 ms: true
|
||||
a real API adds its own rate limits, which this test cannot show
|
||||
|
||||
--- failure: the model does not reply with a bare integer ---
|
||||
reply "Score: 8" for every candidate -> failures counted: 4 of 4
|
||||
scores assigned: [0, 0]
|
||||
pages kept, in order: [2, 3] (the vector-search order, because every score is 0)
|
||||
reply " 9\n" (padded) -> score 9
|
||||
@@ -0,0 +1,32 @@
|
||||
# Generation and the faithfulness check
|
||||
|
||||
|
||||
--- 1. the answer is in the chunks ---
|
||||
status=answered grounded=true sources=4
|
||||
answer: Full-time employees are entitled to 20 working days
|
||||
of annual leave per calendar year.
|
||||
|
||||
--- 2. the model answers with something the chunks do not say ---
|
||||
status=ungrounded grounded=false sources=4
|
||||
answer: Employees get 30 days of annual leave.
|
||||
the check the judge model was given:
|
||||
Evaluate whether or not the following claim is supported by the provided document.
|
||||
Respond with "yes" if the claim is supported, or "no" if it is not.
|
||||
|
||||
|
||||
--- 3. the judge says "Yes." instead of "yes" ---
|
||||
status=ungrounded grounded=false sources=4
|
||||
answer: Full-time employees are entitled to 20 working days
|
||||
of annual leave per calendar year.
|
||||
with the reply "YES": grounded=true
|
||||
|
||||
--- 4. the judge call itself fails ---
|
||||
status=ungrounded grounded=false sources=4
|
||||
answer: Full-time employees are entitled to 20 working days
|
||||
of annual leave per calendar year.
|
||||
rag.faithfulness.judge_failures = 1
|
||||
|
||||
--- 5. nothing is retrieved (threshold 0.3, off-topic question) ---
|
||||
status=no_context grounded=false sources=0
|
||||
answer: I don't have enough information in the provided documents.
|
||||
fact-check calls made: 0
|
||||
@@ -0,0 +1,26 @@
|
||||
# End to end: HTTP, real PostgreSQL + pgvector, the shipped application.yml
|
||||
|
||||
schema from init.sql: [document_chunks_embedding_idx, document_chunks_pkey]
|
||||
|
||||
--- POST /api/ingest (acme, then globex, then acme again) ---
|
||||
{"filename":"acme-handbook.pdf","status":"ingested","chunksWritten":4,"chunksReplaced":0}
|
||||
{"filename":"globex-manual.pdf","status":"ingested","chunksWritten":1,"chunksReplaced":0}
|
||||
{"filename":"acme-handbook.pdf","status":"skipped","chunksWritten":0,"chunksReplaced":0}
|
||||
|
||||
--- what is in the table ---
|
||||
rows: 5
|
||||
rows per tenant: [acme=4, globex=1]
|
||||
metadata of one row: {"doc_type": "general", "tenant_id": "globex", "chunk_index": 0, "page_number": 1, "source_file": "globex-manual.pdf", "total_chunks": 1}
|
||||
|
||||
--- POST /api/query ---
|
||||
tenantId acme: {"answer":"Full-time employees are entitled to 20 working days\nof annual leave per calendar year.","sources":[{"file":"acme-handbook.pdf","page":2,"rerankScore":8,"preview":"4.1 Annual Leave Entitlement. Full-time employees are entitled to 20 working day"},{"file":"acme-handbook.pdf","page":3,"rerankScore":6,"preview":"4.3 Requesting Leave. All leave requests must be submitted through the HR portal"},{"file":"acme-handbook.pdf","page":1,"rerankScore":6,"preview":"Acme Employee Handbook 2026\n\n3.5 Probationary Period. During the three month pro"},{"file":"acme-handbook.pdf","page":4,"rerankScore":4,"preview":"6.1 Expenses. Receipts are required for every expense above 50 euros.\nClaims mus"}],"grounded":true,"status":"answered"}
|
||||
tenantId globex: {"answer":"Full-time staff are entitled to 25 working days\nof annual leave per calendar year.","sources":[{"file":"globex-manual.pdf","page":1,"rerankScore":6,"preview":"Globex Staff Manual 2026\n\n4.1 Annual Leave Entitlement. Full-time staff are enti"}],"grounded":true,"status":"answered"}
|
||||
|
||||
--- GET /actuator/prometheus (only the rag_ series; the timer's sum and max are left out because they change every run) ---
|
||||
rag_chunks_ingested_total 5.0
|
||||
rag_context_chunks_count 2
|
||||
rag_context_chunks_sum 5.0
|
||||
rag_context_chunks_max 4.0
|
||||
rag_ingestion_skipped_total 1.0
|
||||
rag_queries_total{status="answered"} 2.0
|
||||
rag_query_duration_seconds_count{status="answered"} 2
|
||||
@@ -0,0 +1,39 @@
|
||||
# QuestionAnswerAdvisor: the smallest RAG
|
||||
|
||||
only the prompts are recorded: what a real model would reply is not something this repository tests
|
||||
|
||||
--- the prompt the model received ---
|
||||
How many days of annual leave do employees get?
|
||||
|
||||
Context information is below, surrounded by ---------------------
|
||||
|
||||
---------------------
|
||||
4.1 Annual Leave Entitlement. Full-time employees are entitled to 20 working days
|
||||
of annual leave per calendar year. Part-time employees receive leave pro rata.
|
||||
|
||||
4.2 Leave Carryover. Unused annual leave may be carried over for a maximum of 5 days
|
||||
into the next calendar year and must be used by 31 March.
|
||||
4.3 Requesting Leave. All leave requests must be submitted through the HR portal
|
||||
at least two weeks in advance for absences longer than three days.
|
||||
|
||||
4.7 Sick Leave. Sick leave is separate from annual leave and is not deducted from it.
|
||||
A medical certificate is required after three consecutive days.
|
||||
---------------------
|
||||
|
||||
Given the context and provided history information and not prior knowledge,
|
||||
reply to the user comment. If the answer is not in the context, inform
|
||||
the user that you can't answer the question.
|
||||
|
||||
|
||||
--- the prompt for an off-topic question (nothing passes the threshold) ---
|
||||
What is the capital of Mongolia?
|
||||
|
||||
Context information is below, surrounded by ---------------------
|
||||
|
||||
---------------------
|
||||
---------------------
|
||||
|
||||
Given the context and provided history information and not prior knowledge,
|
||||
reply to the user comment. If the answer is not in the context, inform
|
||||
the user that you can't answer the question.
|
||||
|
||||
@@ -0,0 +1,176 @@
|
||||
# Spring AI API facts, read with javap from the jars this build resolves
|
||||
|
||||
spring-ai.version: 2.0.1
|
||||
|
||||
## classes in org/springframework/ai/rag (top level, from spring-ai-rag)
|
||||
Query
|
||||
advisor/RetrievalAugmentationAdvisor
|
||||
advisor/package-info
|
||||
generation/augmentation/ContextualQueryAugmenter
|
||||
generation/augmentation/QueryAugmenter
|
||||
generation/augmentation/package-info
|
||||
generation/package-info
|
||||
package-info
|
||||
postretrieval/document/DocumentPostProcessor
|
||||
postretrieval/document/package-info
|
||||
postretrieval/package-info
|
||||
preretrieval/package-info
|
||||
preretrieval/query/expansion/MultiQueryExpander
|
||||
preretrieval/query/expansion/QueryExpander
|
||||
preretrieval/query/expansion/package-info
|
||||
preretrieval/query/transformation/CompressionQueryTransformer
|
||||
preretrieval/query/transformation/QueryTransformer
|
||||
preretrieval/query/transformation/RewriteQueryTransformer
|
||||
preretrieval/query/transformation/TranslationQueryTransformer
|
||||
preretrieval/query/transformation/package-info
|
||||
retrieval/join/ConcatenationDocumentJoiner
|
||||
retrieval/join/DocumentJoiner
|
||||
retrieval/join/package-info
|
||||
retrieval/search/DocumentRetriever
|
||||
retrieval/search/VectorStoreDocumentRetriever
|
||||
retrieval/search/package-info
|
||||
util/PromptAssert
|
||||
util/package-info
|
||||
|
||||
## classes anywhere in spring-ai-rag whose name contains "rerank": 0
|
||||
|
||||
## classes in any jar on this project's classpath whose name contains "SemanticSearchCache" or "SemanticCache": 0
|
||||
|
||||
## org.springframework.ai.transformer.splitter.TokenTextSplitter
|
||||
public class org.springframework.ai.transformer.splitter.TokenTextSplitter extends org.springframework.ai.transformer.splitter.TextSplitter {
|
||||
public org.springframework.ai.transformer.splitter.TokenTextSplitter();
|
||||
public org.springframework.ai.transformer.splitter.TokenTextSplitter(boolean);
|
||||
public org.springframework.ai.transformer.splitter.TokenTextSplitter(com.knuddels.jtokkit.api.EncodingType);
|
||||
public org.springframework.ai.transformer.splitter.TokenTextSplitter(com.knuddels.jtokkit.api.EncodingType, boolean);
|
||||
public org.springframework.ai.transformer.splitter.TokenTextSplitter(int, int, int, int, boolean, List<Character>);
|
||||
public static org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder builder();
|
||||
}
|
||||
|
||||
## org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder
|
||||
public final class org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder {
|
||||
public org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder withEncodingType(com.knuddels.jtokkit.api.EncodingType);
|
||||
public org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder withChunkSize(int);
|
||||
public org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder withMinChunkSizeChars(int);
|
||||
public org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder withMinChunkLengthToEmbed(int);
|
||||
public org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder withMaxNumChunks(int);
|
||||
public org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder withKeepSeparator(boolean);
|
||||
public org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder withPunctuationMarks(List<Character>);
|
||||
public org.springframework.ai.transformer.splitter.TokenTextSplitter build();
|
||||
}
|
||||
|
||||
## org.springframework.ai.reader.pdf.PagePdfDocumentReader
|
||||
public class org.springframework.ai.reader.pdf.PagePdfDocumentReader implements org.springframework.ai.document.DocumentReader {
|
||||
public static final String METADATA_START_PAGE_NUMBER;
|
||||
public static final String METADATA_END_PAGE_NUMBER;
|
||||
public static final String METADATA_FILE_NAME;
|
||||
public org.springframework.ai.reader.pdf.PagePdfDocumentReader(String);
|
||||
public org.springframework.ai.reader.pdf.PagePdfDocumentReader(org.springframework.core.io.Resource);
|
||||
public org.springframework.ai.reader.pdf.PagePdfDocumentReader(String, org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig);
|
||||
public org.springframework.ai.reader.pdf.PagePdfDocumentReader(org.springframework.core.io.Resource, org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig);
|
||||
public List<org.springframework.ai.document.Document> get();
|
||||
public Object get();
|
||||
}
|
||||
|
||||
## org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig$Builder
|
||||
public final class org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig$Builder {
|
||||
public org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig$Builder withPageExtractedTextFormatter(org.springframework.ai.reader.ExtractedTextFormatter);
|
||||
public org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig$Builder withPagesPerDocument(int);
|
||||
public org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig$Builder withPageTopMargin(int);
|
||||
public org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig$Builder withPageBottomMargin(int);
|
||||
public org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig$Builder withReversedParagraphPosition(boolean);
|
||||
public org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig$Builder addPageRange(int, int);
|
||||
public org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig build();
|
||||
}
|
||||
|
||||
## org.springframework.ai.rag.postretrieval.document.DocumentPostProcessor
|
||||
public interface org.springframework.ai.rag.postretrieval.document.DocumentPostProcessor extends function.BiFunction<org.springframework.ai.rag.Query, List<org.springframework.ai.document.Document>, List<org.springframework.ai.document.Document>> {
|
||||
public abstract List<org.springframework.ai.document.Document> process(org.springframework.ai.rag.Query, List<org.springframework.ai.document.Document>);
|
||||
public default List<org.springframework.ai.document.Document> apply(org.springframework.ai.rag.Query, List<org.springframework.ai.document.Document>);
|
||||
public default Object apply(Object, Object);
|
||||
}
|
||||
|
||||
## org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever
|
||||
public final class org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever implements org.springframework.ai.rag.retrieval.search.DocumentRetriever {
|
||||
public static final String FILTER_EXPRESSION;
|
||||
public org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever(org.springframework.ai.vectorstore.VectorStore, Double, Integer, function.Supplier<org.springframework.ai.vectorstore.filter.Filter$Expression>);
|
||||
public List<org.springframework.ai.document.Document> retrieve(org.springframework.ai.rag.Query);
|
||||
public static org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever$Builder builder();
|
||||
}
|
||||
|
||||
## org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever$Builder
|
||||
public final class org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever$Builder {
|
||||
public org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever$Builder vectorStore(org.springframework.ai.vectorstore.VectorStore);
|
||||
public org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever$Builder similarityThreshold(Double);
|
||||
public org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever$Builder topK(Integer);
|
||||
public org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever$Builder filterExpression(org.springframework.ai.vectorstore.filter.Filter$Expression);
|
||||
public org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever$Builder filterExpression(function.Supplier<org.springframework.ai.vectorstore.filter.Filter$Expression>);
|
||||
public org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever build();
|
||||
}
|
||||
|
||||
## org.springframework.ai.rag.generation.augmentation.ContextualQueryAugmenter$Builder
|
||||
public final class org.springframework.ai.rag.generation.augmentation.ContextualQueryAugmenter$Builder {
|
||||
public org.springframework.ai.rag.generation.augmentation.ContextualQueryAugmenter$Builder();
|
||||
public org.springframework.ai.rag.generation.augmentation.ContextualQueryAugmenter$Builder promptTemplate(org.springframework.ai.chat.prompt.PromptTemplate);
|
||||
public org.springframework.ai.rag.generation.augmentation.ContextualQueryAugmenter$Builder emptyContextPromptTemplate(org.springframework.ai.chat.prompt.PromptTemplate);
|
||||
public org.springframework.ai.rag.generation.augmentation.ContextualQueryAugmenter$Builder allowEmptyContext(Boolean);
|
||||
public org.springframework.ai.rag.generation.augmentation.ContextualQueryAugmenter$Builder documentFormatter(function.Function<List<org.springframework.ai.document.Document>, String>);
|
||||
public org.springframework.ai.rag.generation.augmentation.ContextualQueryAugmenter build();
|
||||
}
|
||||
|
||||
## org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder
|
||||
public final class org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder {
|
||||
public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder queryTransformers(List<org.springframework.ai.rag.preretrieval.query.transformation.QueryTransformer>);
|
||||
public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder queryTransformers(org.springframework.ai.rag.preretrieval.query.transformation.QueryTransformer...);
|
||||
public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder queryExpander(org.springframework.ai.rag.preretrieval.query.expansion.QueryExpander);
|
||||
public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder documentRetriever(org.springframework.ai.rag.retrieval.search.DocumentRetriever);
|
||||
public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder documentJoiner(org.springframework.ai.rag.retrieval.join.DocumentJoiner);
|
||||
public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder documentPostProcessors(List<org.springframework.ai.rag.postretrieval.document.DocumentPostProcessor>);
|
||||
public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder documentPostProcessors(org.springframework.ai.rag.postretrieval.document.DocumentPostProcessor...);
|
||||
public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder queryAugmenter(org.springframework.ai.rag.generation.augmentation.QueryAugmenter);
|
||||
public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder taskExecutor(org.springframework.core.task.TaskExecutor);
|
||||
public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder scheduler(reactor.core.scheduler.Scheduler);
|
||||
public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder order(Integer);
|
||||
public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor build();
|
||||
}
|
||||
|
||||
## org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor$Builder
|
||||
public final class org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor$Builder {
|
||||
public org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor$Builder promptTemplate(org.springframework.ai.chat.prompt.PromptTemplate);
|
||||
public org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor$Builder searchRequest(org.springframework.ai.vectorstore.SearchRequest);
|
||||
public org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor$Builder protectFromBlocking(boolean);
|
||||
public org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor$Builder scheduler(reactor.core.scheduler.Scheduler);
|
||||
public org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor$Builder order(int);
|
||||
public org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor build();
|
||||
}
|
||||
|
||||
## org.springframework.ai.chat.evaluation.FactCheckingEvaluator
|
||||
public class org.springframework.ai.chat.evaluation.FactCheckingEvaluator implements org.springframework.ai.evaluation.Evaluator {
|
||||
public static org.springframework.ai.chat.evaluation.FactCheckingEvaluator forBespokeMinicheck(org.springframework.ai.chat.client.ChatClient$Builder);
|
||||
public org.springframework.ai.evaluation.EvaluationResponse evaluate(org.springframework.ai.evaluation.EvaluationRequest);
|
||||
public static org.springframework.ai.chat.evaluation.FactCheckingEvaluator$Builder builder(org.springframework.ai.chat.client.ChatClient$Builder);
|
||||
}
|
||||
|
||||
## org.springframework.ai.vectorstore.VectorStore
|
||||
public interface org.springframework.ai.vectorstore.VectorStore extends org.springframework.ai.document.DocumentWriter,org.springframework.ai.vectorstore.VectorStoreRetriever {
|
||||
public default String getName();
|
||||
public abstract void add(List<org.springframework.ai.document.Document>);
|
||||
public default void accept(List<org.springframework.ai.document.Document>);
|
||||
public abstract void delete(List<String>);
|
||||
public abstract void delete(org.springframework.ai.vectorstore.filter.Filter$Expression);
|
||||
public default void delete(String);
|
||||
public default <T> Optional<T> getNativeClient();
|
||||
public default void accept(Object);
|
||||
}
|
||||
|
||||
## org.springframework.ai.vectorstore.SearchRequest$Builder
|
||||
public final class org.springframework.ai.vectorstore.SearchRequest$Builder {
|
||||
public org.springframework.ai.vectorstore.SearchRequest$Builder();
|
||||
public org.springframework.ai.vectorstore.SearchRequest$Builder query(String);
|
||||
public org.springframework.ai.vectorstore.SearchRequest$Builder topK(int);
|
||||
public org.springframework.ai.vectorstore.SearchRequest$Builder similarityThreshold(double);
|
||||
public org.springframework.ai.vectorstore.SearchRequest$Builder similarityThresholdAll();
|
||||
public org.springframework.ai.vectorstore.SearchRequest$Builder filterExpression(org.springframework.ai.vectorstore.filter.Filter$Expression);
|
||||
public org.springframework.ai.vectorstore.SearchRequest$Builder filterExpression(String);
|
||||
public org.springframework.ai.vectorstore.SearchRequest build();
|
||||
}
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
# Dependency tree, filtered
|
||||
|
||||
## Spring Boot and Spring AI versions
|
||||
spring-boot-starter-parent 4.1.1
|
||||
spring-ai-bom 2.0.1
|
||||
|
||||
## where spring-jdbc comes from (it is not under any Spring AI artifact)
|
||||
41:+- org.springframework.boot:spring-boot-starter-jdbc:jar:4.1.1:compile
|
||||
42:| +- org.springframework.boot:spring-boot-jdbc:jar:4.1.1:compile
|
||||
47:| | \- org.springframework:spring-jdbc:jar:7.0.9:compile
|
||||
48:| \- com.zaxxer:HikariCP:jar:7.0.2:compile
|
||||
|
||||
## what the pgvector starter brings
|
||||
+- org.springframework.ai:spring-ai-starter-vector-store-pgvector:jar:2.0.1:compile
|
||||
| +- org.springframework.ai:spring-ai-autoconfigure-vector-store-pgvector:jar:2.0.1:compile
|
||||
| +- org.springframework.ai:spring-ai-autoconfigure-vector-store-observation:jar:2.0.1:compile
|
||||
| \- org.springframework.ai:spring-ai-pgvector-store:jar:2.0.1:compile
|
||||
| +- org.postgresql:postgresql:jar:42.7.13:compile
|
||||
| | \- org.checkerframework:checker-qual:jar:3.55.1:runtime
|
||||
| \- com.pgvector:pgvector:jar:0.1.6:compile
|
||||
+- org.springframework.ai:spring-ai-rag:jar:2.0.1:compile
|
||||
|
||||
## every Spring AI artifact on the classpath
|
||||
org.springframework.ai:spring-ai-autoconfigure-model-chat-client:jar:2.0.1
|
||||
org.springframework.ai:spring-ai-autoconfigure-model-chat-memory:jar:2.0.1
|
||||
org.springframework.ai:spring-ai-autoconfigure-model-chat-observation:jar:2.0.1
|
||||
org.springframework.ai:spring-ai-autoconfigure-model-embedding-observation:jar:2.0.1
|
||||
org.springframework.ai:spring-ai-autoconfigure-model-image-observation:jar:2.0.1
|
||||
org.springframework.ai:spring-ai-autoconfigure-model-openai:jar:2.0.1
|
||||
org.springframework.ai:spring-ai-autoconfigure-model-tool:jar:2.0.1
|
||||
org.springframework.ai:spring-ai-autoconfigure-retry:jar:2.0.1
|
||||
org.springframework.ai:spring-ai-autoconfigure-vector-store-observation:jar:2.0.1
|
||||
org.springframework.ai:spring-ai-autoconfigure-vector-store-pgvector:jar:2.0.1
|
||||
org.springframework.ai:spring-ai-client-chat:jar:2.0.1
|
||||
org.springframework.ai:spring-ai-commons:jar:2.0.1
|
||||
org.springframework.ai:spring-ai-model:jar:2.0.1
|
||||
org.springframework.ai:spring-ai-openai:jar:2.0.1
|
||||
org.springframework.ai:spring-ai-pdf-document-reader:jar:2.0.1
|
||||
org.springframework.ai:spring-ai-pgvector-store:jar:2.0.1
|
||||
org.springframework.ai:spring-ai-rag:jar:2.0.1
|
||||
org.springframework.ai:spring-ai-starter-model-openai:jar:2.0.1
|
||||
org.springframework.ai:spring-ai-starter-vector-store-pgvector:jar:2.0.1
|
||||
org.springframework.ai:spring-ai-template-st:jar:2.0.1
|
||||
org.springframework.ai:spring-ai-vector-store-advisor:jar:2.0.1
|
||||
org.springframework.ai:spring-ai-vector-store:jar:2.0.1
|
||||
@@ -0,0 +1,37 @@
|
||||
# The 1.x article's code, against 1.1.0 and 2.0.1
|
||||
|
||||
## The starter artifact ids in the 1.x article: latest version ever published
|
||||
spring-ai-openai-spring-boot-starter latest: 1.0.0-M6
|
||||
spring-ai-pgvector-store-spring-boot-starter latest: 1.0.0-M6
|
||||
the ids that replaced them:
|
||||
spring-ai-starter-model-openai latest: 2.0.1
|
||||
spring-ai-starter-vector-store-pgvector latest: 2.0.1
|
||||
|
||||
## mvn validate on the article's dependency block with spring-ai-bom 1.1.0
|
||||
'dependencies.dependency.version' for org.springframework.ai:spring-ai-openai-spring-boot-starter:jar is missing. @ line 31, column 17
|
||||
'dependencies.dependency.version' for org.springframework.ai:spring-ai-pgvector-store-spring-boot-starter:jar is missing. @ line 35, column 17
|
||||
|
||||
## mvn validate on the article's dependency block with spring-ai-bom 2.0.1
|
||||
'dependencies.dependency.version' for org.springframework.ai:spring-ai-openai-spring-boot-starter:jar is missing. @ line 31, column 17
|
||||
'dependencies.dependency.version' for org.springframework.ai:spring-ai-pgvector-store-spring-boot-starter:jar is missing. @ line 35, column 17
|
||||
|
||||
## javac legacy-1x/src/LegacyIngestion.java against Spring AI 1.1.0
|
||||
legacy-1x/src/LegacyIngestion.java:17: error: incompatible types: ExtractedTextFormatter is not a functional interface
|
||||
.withPageExtractedTextFormatter(text -> text.replaceAll("s{3,}", " "))
|
||||
^
|
||||
|
||||
## javac legacy-1x/src/LegacyIngestion.java against Spring AI 2.0.1
|
||||
legacy-1x/src/LegacyIngestion.java:17: error: incompatible types: ExtractedTextFormatter is not a functional interface
|
||||
.withPageExtractedTextFormatter(text -> text.replaceAll("s{3,}", " "))
|
||||
^
|
||||
legacy-1x/src/LegacyIngestion.java:22: error: no suitable constructor found for TokenTextSplitter(int,int,int,int,boolean)
|
||||
TokenTextSplitter splitter = new TokenTextSplitter(512, 128, 5, 10_000, true);
|
||||
^
|
||||
|
||||
## the 1.x configuration keys in the metadata of spring-ai-autoconfigure-model-openai
|
||||
1.1.0 spring.ai.openai.chat.options.model current
|
||||
1.1.0 spring.ai.openai.chat.options.temperature current
|
||||
1.1.0 spring.ai.openai.embedding.options.model current
|
||||
2.0.1 spring.ai.openai.chat.options.model deprecated, use spring.ai.openai.chat.model
|
||||
2.0.1 spring.ai.openai.chat.options.temperature deprecated, use spring.ai.openai.chat.temperature
|
||||
2.0.1 spring.ai.openai.embedding.options.model deprecated, use spring.ai.openai.embedding.model
|
||||
@@ -0,0 +1,21 @@
|
||||
# spring.ai.* keys in application.yml against the jars' configuration metadata
|
||||
|
||||
property names in the jars' metadata: 1528
|
||||
|
||||
--- keys in the shipped application.yml ---
|
||||
spring.ai.openai.api-key current
|
||||
spring.ai.openai.chat.model current
|
||||
spring.ai.openai.chat.temperature current
|
||||
spring.ai.openai.embedding.model current
|
||||
spring.ai.vectorstore.pgvector.dimensions current
|
||||
spring.ai.vectorstore.pgvector.distance-type current
|
||||
spring.ai.vectorstore.pgvector.index-type current
|
||||
spring.ai.vectorstore.pgvector.initialize-schema current
|
||||
spring.ai.vectorstore.pgvector.schema-name current
|
||||
spring.ai.vectorstore.pgvector.table-name current
|
||||
|
||||
--- keys written the 1.x way ---
|
||||
spring.ai.openai.chat.options.model DEPRECATED, use spring.ai.openai.chat.model
|
||||
spring.ai.openai.chat.options.temperature DEPRECATED, use spring.ai.openai.chat.temperature
|
||||
spring.ai.openai.embedding.options.model DEPRECATED, use spring.ai.openai.embedding.model
|
||||
spring.ai.openai.chat.optoins.model UNKNOWN
|
||||
Reference in New Issue
Block a user