asmhatre/spring-ai
1
0
Fork 0
Code Activity

Add rag-ingestion module: PDF, Word and HTML readers and chunking strategies compared by retrieval hit rate; metadata in the embedding, content-hash idempotent ingestion, re-embedding cost of an edit

Browse Source 
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0146opDKBYtADXvkc39zdiyw
This commit is contained in:
Claude
2026-10-22 18:04:49 +00:00
parent c13195295d
commit d7e68a14af
35 changed files with 1699 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
rag-ingestion/output/06-idempotent-reingestion.txt
+18
View File
@@ -0,0 +1,18 @@
# Re-ingesting runbook.docx into pgvector 0.8.7 (all-MiniLM-L6-v2, 384 dims)
--- a naive loop: read, split, add, with random chunk ids ---
run 1: rows in table = 19, texts embedded so far = 19
run 2: rows in table = 38, texts embedded so far = 38
run 3: rows in table = 57, texts embedded so far = 57
rows saying "kept for 35 days": 3
--- TokenTextSplitter(128): windows of tokens ---
1. first ingest of v1 chunks=19 embedded=19 unchanged= 0 deleted= 0 | rows=19, texts embedded in total=19
2. same file again chunks=19 embedded= 0 unchanged=19 deleted= 0 | rows=19, texts embedded in total=19
3. same file a third time chunks=19 embedded= 0 unchanged=19 deleted= 0 | rows=19, texts embedded in total=19
4. v2: one paragraph edited chunks=19 embedded= 1 unchanged=18 deleted= 1 | rows=19, texts embedded in total=20
rows saying "kept for 35 days": 0, rows saying "kept for 45 days": 1
best match for the snapshot question now contains "45 days": true
5. v3: section 12 deleted from the file chunks=17 embedded= 1 unchanged=16 deleted= 3 | rows=17, texts embedded in total=22
rows saying "180 days in advance" (a fact only section 12 holds): 0