Claude
1d4625a1c2
Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01B38FGKKam5SCGgwgduVAh3
59 lines
3.6 KiB
Markdown
59 lines
3.6 KiB
Markdown
# 3. Ingestion
|
||
|
||
prev: [2. Chunking](02-chunking.md) · [Index](../README.md) · next: [4. Retrieval and reranking](04-retrieval-and-reranking.md)
|
||
|
||
Source: [`IngestionService`](../src/main/java/com/ankurm/rag/ingest/IngestionService.java),
|
||
[`IngestionTracker`](../src/main/java/com/ankurm/rag/ingest/IngestionTracker.java).
|
||
Test: [`IngestionTest`](../src/test/java/com/ankurm/rag/IngestionTest.java).
|
||
|
||
## What a PDF page becomes
|
||
|
||
`PagePdfDocumentReader` with `withPagesPerDocument(1)` returns one `Document` per page, and the only metadata it adds
|
||
is `page_number` ([output 04](output/04-pdf-pages-and-metadata.txt)). That is the number a citation needs, so keep one
|
||
page per document until you have chosen a chunker; chunks inherit their page's metadata.
|
||
|
||
The text is padded. In the sample PDF, page 2 came back as 862 characters with a run of 133 spaces at the end of a
|
||
line, and 304 characters after `IngestionService.tidy` collapsed the padding. Padding costs tokens and changes what gets
|
||
embedded. **The sample PDFs are generated by PDFBox inside this repository**, so the exact padding is an artefact of that
|
||
generator and the reader; real PDFs pad differently. Look at what your own files produce before assuming it matches.
|
||
|
||
Every chunk also gets `source_file`, `source_hash` and whatever the caller passes (the HTTP endpoint adds `tenant_id`
|
||
and `doc_type`). Chunks written by `TokenTextSplitter` additionally carry `parent_document_id`, `chunk_index`
|
||
and `total_chunks`. One stored row, as the end-to-end test read it back from PostgreSQL:
|
||
|
||
```
|
||
{"doc_type": "general", "tenant_id": "globex", "chunk_index": 0, "page_number": 1, "source_file": "globex-manual.pdf", "total_chunks": 1}
|
||
```
|
||
|
||
([output 10](output/10-end-to-end.txt); `source_hash` and `parent_document_id` were left out of that query.)
|
||
|
||
## Uploading the same file twice
|
||
|
||
[Output 05](output/05-ingestion-idempotency.txt), on real PostgreSQL, counting rows with `select count(*)`:
|
||
|
||
| step | what happened | rows | texts embedded |
|
||
|---|---|---|---|
|
||
| first upload | `ingested`, 4 chunks | 4 | 4 |
|
||
| same bytes again | `skipped` | 4 | 4 (nothing spent) |
|
||
| page 2 edited | `updated`, 4 old chunks deleted first | 4 | 8 |
|
||
| same text exported again | `updated` | 4 | 12 |
|
||
| *naive:* `vectorStore.add()` on every upload, twice more | rows tripled | 12 | – |
|
||
|
||
The naive version is what many tutorials show: each upload writes new chunk ids, so the same page is stored three times
|
||
and retrieval returns duplicates that crowd out other passages. The service instead remembers, per file name, the SHA-256 of the
|
||
bytes and the ids of the chunks it wrote, skips identical bytes, and on a change deletes the old chunk **by id** before adding.
|
||
After the edit, zero rows still said "20 working days".
|
||
|
||
The last row is a limit, not a feature: the hash is over the *file's bytes*. Two PDFs with identical text but different bytes
|
||
(the PDFBox generator in the tests produces different bytes on every run for the same text; I did not check which field differs) count as a change and are re-embedded. Hashing the extracted
|
||
text instead would avoid that, at the cost of reading the file first.
|
||
|
||
## What the tracker does not do
|
||
|
||
It lives in memory. A restart forgets every file, and the next upload of each one is re-embedded and its old chunks
|
||
are **not** deleted, because the tracker no longer knows their ids. Two fixes: persist the tracker in a table, or store
|
||
`source_file` in metadata (already done) and delete by filter, `vectorStore.delete(new FilterExpressionBuilder().eq("source_file", name).build())`.
|
||
The second is not exercised by any test here.
|
||
|
||
Next: [4. Retrieval and reranking](04-retrieval-and-reranking.md)
|