asmhatre/spring-ai
1
0
Fork 0
Code Activity

Add rag-ingestion module: PDF, Word and HTML readers and chunking strategies compared by retrieval hit rate; metadata in the embedding, content-hash idempotent ingestion, re-embedding cost of an edit

Browse Source 
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0146opDKBYtADXvkc39zdiyw
This commit is contained in:
Claude
2026-10-22 18:04:49 +00:00
parent c13195295d
commit d7e68a14af
35 changed files with 1699 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
rag-ingestion/output/05-format-parity.txt
+15
View File
@@ -0,0 +1,15 @@
# One runbook, three formats, TokenTextSplitter(128), 37 questions
reader and configuration chunks meanToks hit@1 hit@3 hit@5 cut top3Toks
PDF PagePdfDocumentReader, each page split alone 36 106.6 24/37 32/37 33/37 1 333
PDF pages joined first, footer still there 34 112.9 26/37 33/37 34/37 1 338
PDF pages joined, pageBottomMargin=60 34 110.5 27/37 33/37 34/37 1 330
PDF pages joined, margin 60, whitespace collapsed 19 117.7 27/37 36/37 37/37 0 348
DOCX TikaDocumentReader 19 114.1 31/37 35/37 36/37 0 335
HTML JsoupDocumentReader, default selector (body) 19 114.8 30/37 34/37 36/37 0 342
HTML JsoupDocumentReader, selector=article 19 113.8 29/37 34/37 36/37 0 342
--- tokens the splitter had to cut, by reader ---
PDF pages as read : 30358 characters, 3933 tokens
PDF whitespace collapsed: 10655 characters, 2251 tokens
DOCX : 10657 characters, 2172 tokens