Add rag-ingestion module: PDF, Word and HTML readers and chunking strategies compared by retrieval hit rate; metadata in the embedding, content-hash idempotent ingestion, re-embedding cost of an edit

Co-Authored-By: Claude Sonnet 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_0146opDKBYtADXvkc39zdiyw
2026-10-22 18:04:49 +00:00
parent c13195295d
commit d7e68a14af
35 changed files with 1699 additions and 0 deletions
@@ -0,0 +1,38 @@
+# What each reader returns for the same 36-paragraph runbook
+
+corpus: 10665 characters of body text (title, 12 headings, 36 paragraphs)
+
+--- PagePdfDocumentReader, default config ---
+Documents returned: 4 (one per PDF page)
+metadata of page 1: {file_name=runbook.pdf, page_number=1}
+pages whose text contains the running footer: 4 of 4
+characters as read: 30358, after collapsing runs of whitespace: 10812 (the corpus itself is 10665)
+page 1 ends with: "               Helios  Operations  Runbook -  Confidential  -  Page  1"
+page 2 starts with: "they receive every page but are not expected to act, and they write up what"
+paragraphs that no single page contains whole, because a page break cuts them: 2 of 36
+
+--- PagePdfDocumentReader with pageBottomMargin ---
+pageBottomMargin= 30: footer on 4 of 4 pages, answer fragments still in the text: 37 of 37
+pageBottomMargin= 60: footer on 0 of 4 pages, answer fragments still in the text: 37 of 37
+pageBottomMargin=100: footer on 0 of 4 pages, answer fragments still in the text: 36 of 37
+no margin:            footer on 4 of 4 pages, answer fragments still in the text: 37 of 37
+
+--- TikaDocumentReader on the .docx ---
+Documents returned: 1
+metadata: {source=runbook.docx}
+characters: 10657, contains the footer text: true
+starts with: "Helios Cloud Platform Operations Runbook\n1. Backups and Restore\nEvery production databas"
+blank-line-separated blocks in the text: 1, newline-separated lines: 50
+answer fragments in the text: 37 of 37
+
+--- TikaDocumentReader on the same PDF and HTML ---
+.pdf  -> 1 Document, 10861 characters, metadata keys [source]
+.html -> 1 Document, 10764 characters, contains "Cookie settings": true, contains tracking script text: false
+
+--- JsoupDocumentReader on the .html ---
+default config: 1 Document, 10712 characters, contains nav link text "Status": true, contains "Cookie settings": true
+default metadata: {description=How Helios runs production., title=Helios Cloud Platform Operations Runbook}
+selector=article: 1 Document, 10615 characters, nav text: false, footer text: false
+selector=section, groupByElement=true: 12 Documents
+first section Document: 1188 characters, starts "1. Backups and Restore Every production database is snapshot"
+with metadataTag("description") and additionalMetadata: {description=How Helios runs production., format=html, title=Helios Cloud Platform Operations Runbook}