Add rag-ingestion module: PDF, Word and HTML readers and chunking strategies compared by retrieval hit rate; metadata in the embedding, content-hash idempotent ingestion, re-embedding cost of an edit

Co-Authored-By: Claude Sonnet 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_0146opDKBYtADXvkc39zdiyw
2026-10-22 18:04:49 +00:00
parent c13195295d
commit d7e68a14af
35 changed files with 1699 additions and 0 deletions
@@ -0,0 +1,18 @@
+# Chunking strategies on the same runbook (all-MiniLM-L6-v2, 37 questions)
+
+strategy                                     chunks  meanToks  hit@1  hit@3  hit@5   cut  top3Toks
+TokenTextSplitter 128 tokens                     19     114.1  31/37  35/37  36/37     0        335
+TokenTextSplitter 64 tokens, minChars=10         42      51.7  27/37  32/37  34/37     1        160
+SentenceSplitter 128 tokens, no overlap          20     108.5  24/37  33/37  34/37     0        341
+SentenceSplitter 128 tokens, 1 sentence overlap     24     113.5  30/37  33/37  36/37     0        337
+SentenceSplitter 64 tokens, no overlap           41      52.8  30/37  34/37  34/37     0        161
+one chunk per paragraph                          37      56.4  34/37  36/37  36/37     0        176
+one chunk per paragraph + section heading        37      62.7  32/37  35/37  36/37     0        194
+one chunk per section (JsoupDocumentReader)      12     178.7  31/37  37/37  37/37     0        529
+
+--- 37 chunks from 36 paragraphs ---
+last paragraph chunk: section=12. Decommissioning, text="Helios Operations Runbook - Confidential"
+
+--- what a paragraph chunk with its heading looks like ---
+metadata: {chunk_index=2, section=1. Backups and Restore, source=runbook.docx}
+text: 1. Backups and Restore\nPoint-in-time recovery is available for the last 7 days. To request a recovery, the service owne