spring-ai/rag/docs/03-ingestion.md

3. Ingestion

prev: 2. Chunking · Index · next: 4. Retrieval and reranking

Source: IngestionService, IngestionTracker. Test: IngestionTest.

What a PDF page becomes

PagePdfDocumentReader with withPagesPerDocument(1) returns one Document per page, and the only metadata it adds is page_number (output 04). That is the number a citation needs, so keep one page per document until you have chosen a chunker; chunks inherit their page's metadata.

The text is padded. In the sample PDF, page 2 came back as 862 characters with a run of 133 spaces at the end of a line, and 304 characters after IngestionService.tidy collapsed the padding. Padding costs tokens and changes what gets embedded. The sample PDFs are generated by PDFBox inside this repository, so the exact padding is an artefact of that generator and the reader; real PDFs pad differently. Look at what your own files produce before assuming it matches.

Every chunk also gets source_file, source_hash and whatever the caller passes (the HTTP endpoint adds tenant_id and doc_type). Chunks written by TokenTextSplitter additionally carry parent_document_id, chunk_index and total_chunks. One stored row, as the end-to-end test read it back from PostgreSQL:

{"doc_type": "general", "tenant_id": "globex", "chunk_index": 0, "page_number": 1, "source_file": "globex-manual.pdf", "total_chunks": 1}

(output 10; source_hash and parent_document_id were left out of that query.)

Uploading the same file twice

Output 05, on real PostgreSQL, counting rows with select count(*):

step	what happened	rows	texts embedded
first upload	ingested, 4 chunks	4	4
same bytes again	skipped	4	4 (nothing spent)
page 2 edited	updated, 4 old chunks deleted first	4	8
same text exported again	updated	4	12
naive: vectorStore.add() on every upload, twice more	rows tripled	12	–

The naive version is what many tutorials show: each upload writes new chunk ids, so the same page is stored three times and retrieval returns duplicates that crowd out other passages. The service instead remembers, per file name, the SHA-256 of the bytes and the ids of the chunks it wrote, skips identical bytes, and on a change deletes the old chunk by id before adding. After the edit, zero rows still said "20 working days".

The last row is a limit, not a feature: the hash is over the file's bytes. Two PDFs with identical text but different bytes (the PDFBox generator in the tests produces different bytes on every run for the same text; I did not check which field differs) count as a change and are re-embedded. Hashing the extracted text instead would avoid that, at the cost of reading the file first.

What the tracker does not do

It lives in memory. A restart forgets every file, and the next upload of each one is re-embedded and its old chunks are not deleted, because the tracker no longer knows their ids. Two fixes: persist the tracker in a table, or store source_file in metadata (already done) and delete by filter, vectorStore.delete(new FilterExpressionBuilder().eq("source_file", name).build()). The second is not exercised by any test here.

Next: 4. Retrieval and reranking