asmhatre/spring-ai
1
0
Fork 0
Code Activity

Add rag-ingestion module: PDF, Word and HTML readers and chunking strategies compared by retrieval hit rate; metadata in the embedding, content-hash idempotent ingestion, re-embedding cost of an edit

Browse Source 
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0146opDKBYtADXvkc39zdiyw
This commit is contained in:
Claude
2026-10-22 18:04:49 +00:00
parent c13195295d
commit d7e68a14af
35 changed files with 1699 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
rag-ingestion/output/01-readers.txt
+38
View File
@@ -0,0 +1,38 @@
# What each reader returns for the same 36-paragraph runbook
corpus: 10665 characters of body text (title, 12 headings, 36 paragraphs)
--- PagePdfDocumentReader, default config ---
Documents returned: 4 (one per PDF page)
metadata of page 1: {file_name=runbook.pdf, page_number=1}
pages whose text contains the running footer: 4 of 4
characters as read: 30358, after collapsing runs of whitespace: 10812 (the corpus itself is 10665)
page 1 ends with: " Helios Operations Runbook - Confidential - Page 1"
page 2 starts with: "they receive every page but are not expected to act, and they write up what"
paragraphs that no single page contains whole, because a page break cuts them: 2 of 36
--- PagePdfDocumentReader with pageBottomMargin ---
pageBottomMargin= 30: footer on 4 of 4 pages, answer fragments still in the text: 37 of 37
pageBottomMargin= 60: footer on 0 of 4 pages, answer fragments still in the text: 37 of 37
pageBottomMargin=100: footer on 0 of 4 pages, answer fragments still in the text: 36 of 37
no margin: footer on 4 of 4 pages, answer fragments still in the text: 37 of 37
--- TikaDocumentReader on the .docx ---
Documents returned: 1
metadata: {source=runbook.docx}
characters: 10657, contains the footer text: true
starts with: "Helios Cloud Platform Operations Runbook\n1. Backups and Restore\nEvery production databas"
blank-line-separated blocks in the text: 1, newline-separated lines: 50
answer fragments in the text: 37 of 37
--- TikaDocumentReader on the same PDF and HTML ---
.pdf -> 1 Document, 10861 characters, metadata keys [source]
.html -> 1 Document, 10764 characters, contains "Cookie settings": true, contains tracking script text: false
--- JsoupDocumentReader on the .html ---
default config: 1 Document, 10712 characters, contains nav link text "Status": true, contains "Cookie settings": true
default metadata: {description=How Helios runs production., title=Helios Cloud Platform Operations Runbook}
selector=article: 1 Document, 10615 characters, nav text: false, footer text: false
selector=section, groupByElement=true: 12 Documents
first section Document: 1188 characters, starts "1. Backups and Restore Every production database is snapshot"
with metadataTag("description") and additionalMetadata: {description=How Helios runs production., format=html, title=Helios Cloud Platform Operations Runbook}
rag-ingestion/output/02-chunk-size-sweep.txt
+23
View File
@@ -0,0 +1,23 @@
# TokenTextSplitter chunk size against retrieval hit rate (all-MiniLM-L6-v2, 37 questions)
source: runbook.docx read by TikaDocumentReader, 2172 cl100k tokens, 37 questions, embedding model all-MiniLM-L6-v2 (max 256 WordPiece tokens)
hit@k: the question's answer is inside one of the k best chunks. cut: answers that no chunk contains whole.
top3Toks: tokens in the three best chunks, the context a prompt would carry per question.
--- defaults except chunkSize (minChunkSizeChars=350) ---
chunkSize chunks meanToks hit@1 hit@3 hit@5 cut top3Toks
32 68 31.9 22/37 29/37 30/37 3 96
64 34 63.9 27/37 33/37 33/37 2 191
128 19 114.1 31/37 35/37 36/37 0 335
256 9 241.1 24/37 31/37 36/37 0 720
512 5 434.0 21/37 33/37 37/37 0 1278
800 3 723.3 17/37 37/37 37/37 0 2170
--- same sizes with minChunkSizeChars=10 ---
chunkSize chunks meanToks hit@1 hit@3 hit@5 cut top3Toks
32 88 24.6 31/37 32/37 32/37 0 76
64 42 51.7 27/37 32/37 34/37 1 160
128 19 114.1 31/37 35/37 36/37 0 335
256 9 241.1 24/37 31/37 36/37 0 720
512 5 434.0 21/37 33/37 37/37 0 1278
800 3 723.3 17/37 37/37 37/37 0 2170
rag-ingestion/output/03-chunking-strategies.txt
+18
View File
@@ -0,0 +1,18 @@
# Chunking strategies on the same runbook (all-MiniLM-L6-v2, 37 questions)
strategy chunks meanToks hit@1 hit@3 hit@5 cut top3Toks
TokenTextSplitter 128 tokens 19 114.1 31/37 35/37 36/37 0 335
TokenTextSplitter 64 tokens, minChars=10 42 51.7 27/37 32/37 34/37 1 160
SentenceSplitter 128 tokens, no overlap 20 108.5 24/37 33/37 34/37 0 341
SentenceSplitter 128 tokens, 1 sentence overlap 24 113.5 30/37 33/37 36/37 0 337
SentenceSplitter 64 tokens, no overlap 41 52.8 30/37 34/37 34/37 0 161
one chunk per paragraph 37 56.4 34/37 36/37 36/37 0 176
one chunk per paragraph + section heading 37 62.7 32/37 35/37 36/37 0 194
one chunk per section (JsoupDocumentReader) 12 178.7 31/37 37/37 37/37 0 529
--- 37 chunks from 36 paragraphs ---
last paragraph chunk: section=12. Decommissioning, text="Helios Operations Runbook - Confidential"
--- what a paragraph chunk with its heading looks like ---
metadata: {chunk_index=2, section=1. Backups and Restore, source=runbook.docx}
text: 1. Backups and Restore\nPoint-in-time recovery is available for the last 7 days. To request a recovery, the service owne
rag-ingestion/output/04-metadata-in-the-embedding.txt
+14
View File
@@ -0,0 +1,14 @@
# Metadata and the text that gets embedded
chunk metadata: {chunk_index=0, content_hash=86f5d5577cede265, format=docx, section=1. Backups and Restore, source=runbook.docx}
--- the text each model would embed for the same chunk ---
MetadataMode.NONE : \n\nEvery production database is snapshotted once a day at 01:30 UTC. Daily snapshots are kept for 35 days, af
MetadataMode.EMBED: chunk_index: 0\nformat: docx\nsection: 1. Backups and Restore\nsource: runbook.docx\ncontent_hash: 86f5d5577ce
--- hit rate, one chunk per paragraph, 37 questions ---
configuration chunks meanToks hit@1 hit@3 hit@5 cut top3Toks
MetadataMode.NONE 37 56.4 34/37 36/37 36/37 0 176
MetadataMode.EMBED, every key embedded 37 56.4 29/37 36/37 36/37 0 175
MetadataMode.EMBED, four keys in excludedEmbedMetadataKeys 37 56.4 31/37 35/37 36/37 0 174
embedded text with the four keys excluded: section: 1. Backups and Restore\n\nEvery production database is snapshotted once a day at 01:30 UTC. Daily sna
rag-ingestion/output/05-format-parity.txt
+15
View File
@@ -0,0 +1,15 @@
# One runbook, three formats, TokenTextSplitter(128), 37 questions
reader and configuration chunks meanToks hit@1 hit@3 hit@5 cut top3Toks
PDF PagePdfDocumentReader, each page split alone 36 106.6 24/37 32/37 33/37 1 333
PDF pages joined first, footer still there 34 112.9 26/37 33/37 34/37 1 338
PDF pages joined, pageBottomMargin=60 34 110.5 27/37 33/37 34/37 1 330
PDF pages joined, margin 60, whitespace collapsed 19 117.7 27/37 36/37 37/37 0 348
DOCX TikaDocumentReader 19 114.1 31/37 35/37 36/37 0 335
HTML JsoupDocumentReader, default selector (body) 19 114.8 30/37 34/37 36/37 0 342
HTML JsoupDocumentReader, selector=article 19 113.8 29/37 34/37 36/37 0 342
--- tokens the splitter had to cut, by reader ---
PDF pages as read : 30358 characters, 3933 tokens
PDF whitespace collapsed: 10655 characters, 2251 tokens
DOCX : 10657 characters, 2172 tokens
rag-ingestion/output/06-idempotent-reingestion.txt
+18
View File
@@ -0,0 +1,18 @@
# Re-ingesting runbook.docx into pgvector 0.8.7 (all-MiniLM-L6-v2, 384 dims)
--- a naive loop: read, split, add, with random chunk ids ---
run 1: rows in table = 19, texts embedded so far = 19
run 2: rows in table = 38, texts embedded so far = 38
run 3: rows in table = 57, texts embedded so far = 57
rows saying "kept for 35 days": 3
--- TokenTextSplitter(128): windows of tokens ---
1. first ingest of v1 chunks=19 embedded=19 unchanged= 0 deleted= 0 | rows=19, texts embedded in total=19
2. same file again chunks=19 embedded= 0 unchanged=19 deleted= 0 | rows=19, texts embedded in total=19
3. same file a third time chunks=19 embedded= 0 unchanged=19 deleted= 0 | rows=19, texts embedded in total=19
4. v2: one paragraph edited chunks=19 embedded= 1 unchanged=18 deleted= 1 | rows=19, texts embedded in total=20
rows saying "kept for 35 days": 0, rows saying "kept for 45 days": 1
best match for the snapshot question now contains "45 days": true
5. v3: section 12 deleted from the file chunks=17 embedded= 1 unchanged=16 deleted= 3 | rows=17, texts embedded in total=22
rows saying "180 days in advance" (a fact only section 12 holds): 0
rag-ingestion/output/07-paragraph-chunks-reingestion.txt
+12
View File
@@ -0,0 +1,12 @@
# Same sequence with one chunk per paragraph
--- HeadingAwareChunker: one chunk per paragraph ---
1. first ingest of v1 chunks=37 embedded=37 unchanged= 0 deleted= 0 | rows=37, texts embedded in total=37
2. same file again chunks=37 embedded= 0 unchanged=37 deleted= 0 | rows=37, texts embedded in total=37
3. same file a third time chunks=37 embedded= 0 unchanged=37 deleted= 0 | rows=37, texts embedded in total=37
4. v2: one paragraph edited chunks=37 embedded= 1 unchanged=36 deleted= 1 | rows=37, texts embedded in total=38
rows saying "kept for 35 days": 0, rows saying "kept for 45 days": 1
best match for the snapshot question now contains "45 days": true
5. v3: section 12 deleted from the file chunks=34 embedded= 0 unchanged=34 deleted= 3 | rows=34, texts embedded in total=39
rows saying "180 days in advance" (a fact only section 12 holds): 0
rag-ingestion/output/08-edit-cost.txt
+9
View File
@@ -0,0 +1,9 @@
# Chunks to embed again after appending one sentence to one paragraph (all 36 paragraphs tried)
splitter chunks mean median max
TokenTextSplitter 128 tokens 19 5.0 2 17
TokenTextSplitter 256 tokens 9 2.5 2 10
SentenceSplitter 128, no overlap 19 2.4 2 8
SentenceSplitter 128, 1 sentence overlap 24 3.1 2 13
one chunk per paragraph 36 1.0 1 1
columns: chunks in the unedited file; mean, median and max chunks re-embedded over the 36 edits
rag-ingestion/output/09-metadata-filter.txt
+29
View File
@@ -0,0 +1,29 @@
# Filtering retrieval by the metadata written at ingestion
rows in table: 73 (runbook.docx and runbook.html, one chunk per paragraph)
JsoupDocumentReader selector=article: 1 line(s) of text -> 0 chunks from the paragraph chunker
selector="article h1, article h2, article p" with separator "\n": 49 lines -> 36 chunks
rows per format: [docx=37, html=36]
question: "How long is data kept?"
--- no filter, top 5 ---
0.640 html 12. Decommissioning After a customer contract ends, the customer's data is del
0.640 docx 12. Decommissioning After a customer contract ends, the customer's data is del
0.588 html 1. Backups and Restore Every production database is snapshotted once a day at 01:
0.588 docx 1. Backups and Restore Every production database is snapshotted once a day at 01:
0.569 html 9. Logging and Monitoring Metrics are stored at 10 second resolution for 7 days, the
(5 results)
--- filter: section == '9. Logging and Monitoring' ---
0.569 docx 9. Logging and Monitoring Metrics are stored at 10 second resolution for 7 days, the
0.569 html 9. Logging and Monitoring Metrics are stored at 10 second resolution for 7 days, the
0.514 docx 9. Logging and Monitoring Application logs are searchable for 90 days in the hot tie
0.514 html 9. Logging and Monitoring Application logs are searchable for 90 days in the hot tie
0.188 docx 9. Logging and Monitoring To keep paging humane, each on-call shift has a noise budg
(5 results)
--- filter: format == 'html' && section == '1. Backups and Restore' ---
0.588 html 1. Backups and Restore Every production database is snapshotted once a day at 01:
0.468 html 1. Backups and Restore Point-in-time recovery is available for the last 7 days. T
0.254 html 1. Backups and Restore A backup that has never been restored is only a hope. On t
(3 results)