Add rag-ingestion module: PDF, Word and HTML readers and chunking strategies compared by retrieval hit rate; metadata in the embedding, content-hash idempotent ingestion, re-embedding cost of an edit

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0146opDKBYtADXvkc39zdiyw

rag-ingestion/output/02-chunk-size-sweep.txt

@@ -0,0 +1,23 @@
+# TokenTextSplitter chunk size against retrieval hit rate (all-MiniLM-L6-v2, 37 questions)
+
+source: runbook.docx read by TikaDocumentReader, 2172 cl100k tokens, 37 questions, embedding model all-MiniLM-L6-v2 (max 256 WordPiece tokens)
+hit@k: the question's answer is inside one of the k best chunks. cut: answers that no chunk contains whole.
+top3Toks: tokens in the three best chunks, the context a prompt would carry per question.
+
+--- defaults except chunkSize (minChunkSizeChars=350) ---
+chunkSize  chunks  meanToks  hit@1  hit@3  hit@5   cut  top3Toks
+       32      68      31.9  22/37  29/37  30/37     3         96
+       64      34      63.9  27/37  33/37  33/37     2        191
+      128      19     114.1  31/37  35/37  36/37     0        335
+      256       9     241.1  24/37  31/37  36/37     0        720
+      512       5     434.0  21/37  33/37  37/37     0       1278
+      800       3     723.3  17/37  37/37  37/37     0       2170
+
+--- same sizes with minChunkSizeChars=10 ---
+chunkSize  chunks  meanToks  hit@1  hit@3  hit@5   cut  top3Toks
+       32      88      24.6  31/37  32/37  32/37     0         76
+       64      42      51.7  27/37  32/37  34/37     1        160
+      128      19     114.1  31/37  35/37  36/37     0        335
+      256       9     241.1  24/37  31/37  36/37     0        720
+      512       5     434.0  21/37  33/37  37/37     0       1278
+      800       3     723.3  17/37  37/37  37/37     0       2170