Add rag module: Spring AI 2.0 RAG with pgvector, chunking, reranking and a faithfulness check

Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01B38FGKKam5SCGgwgduVAh3
2026-09-21 19:09:05 +00:00
commit 1d4625a1c2
62 changed files with 3715 additions and 0 deletions
@@ -0,0 +1,58 @@
+# 3. Ingestion
+
+prev: [2. Chunking](02-chunking.md) · [Index](../README.md) · next: [4. Retrieval and reranking](04-retrieval-and-reranking.md)
+
+Source: [`IngestionService`](../src/main/java/com/ankurm/rag/ingest/IngestionService.java),
+[`IngestionTracker`](../src/main/java/com/ankurm/rag/ingest/IngestionTracker.java).
+Test: [`IngestionTest`](../src/test/java/com/ankurm/rag/IngestionTest.java).
+
+## What a PDF page becomes
+
+`PagePdfDocumentReader` with `withPagesPerDocument(1)` returns one `Document` per page, and the only metadata it adds
+is `page_number` ([output 04](output/04-pdf-pages-and-metadata.txt)). That is the number a citation needs, so keep one
+page per document until you have chosen a chunker; chunks inherit their page's metadata.
+
+The text is padded. In the sample PDF, page 2 came back as 862 characters with a run of 133 spaces at the end of a
+line, and 304 characters after `IngestionService.tidy` collapsed the padding. Padding costs tokens and changes what gets
+embedded. **The sample PDFs are generated by PDFBox inside this repository**, so the exact padding is an artefact of that
+generator and the reader; real PDFs pad differently. Look at what your own files produce before assuming it matches.
+
+Every chunk also gets `source_file`, `source_hash` and whatever the caller passes (the HTTP endpoint adds `tenant_id`
+and `doc_type`). Chunks written by `TokenTextSplitter` additionally carry `parent_document_id`, `chunk_index`
+and `total_chunks`. One stored row, as the end-to-end test read it back from PostgreSQL:
+
+```
+{"doc_type": "general", "tenant_id": "globex", "chunk_index": 0, "page_number": 1, "source_file": "globex-manual.pdf", "total_chunks": 1}
+```
+
+([output 10](output/10-end-to-end.txt); `source_hash` and `parent_document_id` were left out of that query.)
+
+## Uploading the same file twice
+
+[Output 05](output/05-ingestion-idempotency.txt), on real PostgreSQL, counting rows with `select count(*)`:
+
+| step | what happened | rows | texts embedded |
+|---|---|---|---|
+| first upload | `ingested`, 4 chunks | 4 | 4 |
+| same bytes again | `skipped` | 4 | 4 (nothing spent) |
+| page 2 edited | `updated`, 4 old chunks deleted first | 4 | 8 |
+| same text exported again | `updated` | 4 | 12 |
+| *naive:* `vectorStore.add()` on every upload, twice more | rows tripled | 12 | – |
+
+The naive version is what many tutorials show: each upload writes new chunk ids, so the same page is stored three times
+and retrieval returns duplicates that crowd out other passages. The service instead remembers, per file name, the SHA-256 of the
+bytes and the ids of the chunks it wrote, skips identical bytes, and on a change deletes the old chunk **by id** before adding.
+After the edit, zero rows still said "20 working days".
+
+The last row is a limit, not a feature: the hash is over the *file's bytes*. Two PDFs with identical text but different bytes
+(the PDFBox generator in the tests produces different bytes on every run for the same text; I did not check which field differs) count as a change and are re-embedded. Hashing the extracted
+text instead would avoid that, at the cost of reading the file first.
+
+## What the tracker does not do
+
+It lives in memory. A restart forgets every file, and the next upload of each one is re-embedded and its old chunks
+are **not** deleted, because the tracker no longer knows their ids. Two fixes: persist the tracker in a table, or store
+`source_file` in metadata (already done) and delete by filter, `vectorStore.delete(new FilterExpressionBuilder().eq("source_file", name).build())`.
+The second is not exercised by any test here.
+
+Next: [4. Retrieval and reranking](04-retrieval-and-reranking.md)