Add rag-ingestion module: PDF, Word and HTML readers and chunking strategies compared by retrieval hit rate; metadata in the embedding, content-hash idempotent ingestion, re-embedding cost of an edit

Co-Authored-By: Claude Sonnet 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_0146opDKBYtADXvkc39zdiyw
2026-10-22 18:04:49 +00:00
parent c13195295d
commit d7e68a14af
35 changed files with 1699 additions and 0 deletions
@@ -0,0 +1,12 @@
+# Same sequence with one chunk per paragraph
+
+
+--- HeadingAwareChunker: one chunk per paragraph ---
+1. first ingest of v1                    chunks=37 embedded=37 unchanged= 0 deleted= 0 | rows=37, texts embedded in total=37
+2. same file again                       chunks=37 embedded= 0 unchanged=37 deleted= 0 | rows=37, texts embedded in total=37
+3. same file a third time                chunks=37 embedded= 0 unchanged=37 deleted= 0 | rows=37, texts embedded in total=37
+4. v2: one paragraph edited              chunks=37 embedded= 1 unchanged=36 deleted= 1 | rows=37, texts embedded in total=38
+   rows saying "kept for 35 days": 0, rows saying "kept for 45 days": 1
+   best match for the snapshot question now contains "45 days": true
+5. v3: section 12 deleted from the file  chunks=34 embedded= 0 unchanged=34 deleted= 3 | rows=34, texts embedded in total=39
+   rows saying "180 days in advance" (a fact only section 12 holds): 0