asmhatre/spring-ai
1
0
Fork 0
Code Activity

Add rag-ingestion module: PDF, Word and HTML readers and chunking strategies compared by retrieval hit rate; metadata in the embedding, content-hash idempotent ingestion, re-embedding cost of an edit

Browse Source 
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0146opDKBYtADXvkc39zdiyw
This commit is contained in:
Claude
2026-10-22 18:04:49 +00:00
parent c13195295d
commit d7e68a14af
35 changed files with 1699 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
rag-ingestion/output/04-metadata-in-the-embedding.txt
+14
View File
@@ -0,0 +1,14 @@
# Metadata and the text that gets embedded
chunk metadata: {chunk_index=0, content_hash=86f5d5577cede265, format=docx, section=1. Backups and Restore, source=runbook.docx}
--- the text each model would embed for the same chunk ---
MetadataMode.NONE : \n\nEvery production database is snapshotted once a day at 01:30 UTC. Daily snapshots are kept for 35 days, af
MetadataMode.EMBED: chunk_index: 0\nformat: docx\nsection: 1. Backups and Restore\nsource: runbook.docx\ncontent_hash: 86f5d5577ce
--- hit rate, one chunk per paragraph, 37 questions ---
configuration chunks meanToks hit@1 hit@3 hit@5 cut top3Toks
MetadataMode.NONE 37 56.4 34/37 36/37 36/37 0 176
MetadataMode.EMBED, every key embedded 37 56.4 29/37 36/37 36/37 0 175
MetadataMode.EMBED, four keys in excludedEmbedMetadataKeys 37 56.4 31/37 35/37 36/37 0 174
embedded text with the four keys excluded: section: 1. Backups and Restore\n\nEvery production database is snapshotted once a day at 01:30 UTC. Daily sna