asmhatre/spring-ai
1
0
Fork 0
Code Actions Activity

Add rag module: Spring AI 2.0 RAG with pgvector, chunking, reranking and a faithfulness check

Browse Source 
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01B38FGKKam5SCGgwgduVAh3
This commit is contained in:
Claude
2026-09-21 19:09:05 +00:00
commit 1d4625a1c2
62 changed files with 3715 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
rag/docs/output/02-token-text-splitter.txt
+27
View File
@@ -0,0 +1,27 @@
# What TokenTextSplitter does to a 40-sentence document
document: 4268 characters, 880 cl100k_base tokens
--- new TokenTextSplitter() (defaults: 800 tokens, 350 min chars, 5 min length to embed) ---
chunks: 2, sizes in tokens: [792, 88]
--- chunk size 100 tokens ---
chunks: 10
chunk 0: 88 tokens, 419 chars, starts "Sentence 1 describes t", ends "ry full-time employee."
chunk 1: 88 tokens, 420 chars, starts "Sentence 5 describes t", ends "ry full-time employee."
chunk 2: 88 tokens, 426 chars, starts "Sentence 9 describes t", ends "ry full-time employee."
chunk 3: 88 tokens, 428 chars, starts "Sentence 13 describes ", ends "ry full-time employee."
chunk 4: 88 tokens, 427 chars, starts "Sentence 17 describes ", ends "ry full-time employee."
chunk 5: 88 tokens, 427 chars, starts "Sentence 21 describes ", ends "ry full-time employee."
chunk 6: 88 tokens, 428 chars, starts "Sentence 25 describes ", ends "ry full-time employee."
chunk 7: 88 tokens, 428 chars, starts "Sentence 29 describes ", ends "ry full-time employee."
chunk 8: 88 tokens, 428 chars, starts "Sentence 33 describes ", ends "ry full-time employee."
chunk 9: 88 tokens, 427 chars, starts "Sentence 37 describes ", ends "ry full-time employee."
metadata of chunk 1 (parent_document_id, a random UUID, left out): {chunk_index=1, page_number=1, source_file=long.txt, total_chunks=10}
--- is there any overlap between neighbouring chunks? ---
boundaries where the first 30 characters of a chunk already appear in the chunk before it: 0 of 9
--- minChunkSizeChars: where the cut lands ---
minChunkSizeChars=350: 10 chunks, 9 of the first 9 end on a full stop
minChunkSizeChars= 50: 10 chunks, 9 of the first 9 end on a full stop