Every retrieval-augmented generation (RAG) demo starts with the same line: load some files, split them, put them in a vector store. It looks like a detail. It is the part that decides whether the right paragraph is even available when a user asks a question, and it has more knobs than the demos admit: which reader for which file type, how big a chunk, whether to cut on tokens or on sentences, what to attach as metadata, and what happens when the same file is loaded a second time.
This article builds a small ingestion pipeline for PDF, Word and HTML files with Spring AI, then answers the knob questions with a measurement instead of an opinion: one fixed set of 37 questions, one local embedding model, and a count of how often the paragraph that holds the answer comes back. The numbers are small and the document is synthetic, so treat them as a demonstration of how to measure, not as a verdict on your documents.
Versions this was written and run against. Spring Boot 4.1.2, Spring AI 2.0.2 (the latest GA release; 2.1.0-RC1 is a release candidate and was not used) and Java 25 (Temurin 25.0.5). Embeddings come fromall-MiniLM-L6-v2throughspring-ai-transformers, running in-process, so no API key is needed. The vector store is pgvector 0.8.7 on PostgreSQL 16.15 for the ingestion tests and Spring AI'sSimpleVectorStorefor the measurements. All the code is in therag-ingestionmodule of asmhatre/spring-ai, and every console block below is quoted from a file underoutput/, written by the test that produced it.
Ingestion is four steps, and each one can quietly lose the answer
Before any question is asked, a file has to become rows in a vector store. Spring AI gives each step its own interface, which is the reason you can swap one without touching the rest.
A Document in Spring AI is just text plus a map of metadata. A reader turns a file into one or more of them. A splitter (a DocumentTransformer) turns long documents into chunks. The vector store embeds each chunk, which means turning its text into a list of 384 numbers, and stores it next to the text. When a user asks something, the question is embedded the same way and the store returns the chunks whose numbers are closest. If the paragraph with the answer was cut in half, or buried under a page footer, or never stored, no later step can recover it. For the part after retrieval, see Production-grade RAG with Spring AI; for choosing the store itself, Choosing a Vector Store for Spring AI.
The test: one runbook, three file formats, 37 questions
To compare choices fairly you need something that stays still. The test document is a made-up operations runbook for an imaginary cloud company (title, 12 numbered sections, 36 paragraphs, about 10,700 characters). It is written to disk as a PDF, a Word file and an HTML page with the same text, plus a running footer on every page and navigation links in the HTML, because real files have those.
There are 37 questions, each with a short answer fragment that appears in exactly one place, for example a question about how long daily snapshots are kept whose answer contains “35 days”. A question counts as a hit@k when one of the best k chunks contains its answer fragment. So hit@1 is “the right chunk came back first”, and hit@5 is “it was somewhere in the top five”. Every table below is these counts out of 37. The fragments and the corpus are in Corpus.java.
Going deeper: how the measurement works, and what it cannot tell you
Chunks are embedded with all-MiniLM-L6-v2 (384 dimensions, input truncated at 256 word pieces) and stored in a SimpleVectorStore, whose similarity search is exact, so there is no approximate index to add noise. The scoring code is Eval.java. Two columns appear in the tables that are not hit rates: cut is the number of answers that no chunk contains whole, and top3Toks is how many tokens (counted with the cl100k encoding) the three best chunks add up to, which is roughly what you would paste into a prompt per question.
Limits worth stating up front. The corpus is synthetic and every fact is stated once, so real documents with repetition will behave differently. With 37 questions, a difference of two or three questions is within what a rephrased question could flip. And hit@k rewards big chunks as k grows, because a 700-token chunk contains the answer more often than a 30-token one; that is why top3Toks is printed next to it.
Step one: readers return different shapes for the same words
Spring AI ships a reader per family of files. Three cover most business documents: PagePdfDocumentReader (PDFBox underneath) for PDFs, TikaDocumentReader (Apache Tika) for Word and almost anything else, and JsoupDocumentReader for HTML. They all implement DocumentReader, so the calling code is the same, but what comes back is not.
static DocumentReader reader(Path file, String format) {
FileSystemResource resource = new FileSystemResource(file);
return switch (format) {
case "pdf" -> new PagePdfDocumentReader(resource,
PdfDocumentReaderConfig.builder().withPageBottomMargin(60).build());
case "html", "htm" -> new JsoupDocumentReader(resource, JsoupDocumentReaderConfig.builder()
.selector("article h1, article h2, article p").separator("\n").build());
default -> new TikaDocumentReader(resource);
};
}That switch is from IngestionPipeline.java. Each of the three non-default settings in it was added because of something the output below shows.
PDF: one Document per page, and the footer comes along
--- PagePdfDocumentReader, default config ---
Documents returned: 4 (one per PDF page)
metadata of page 1: {file_name=runbook.pdf, page_number=1}
pages whose text contains the running footer: 4 of 4
characters as read: 30358, after collapsing runs of whitespace: 10812 (the corpus itself is 10665)
page 1 ends with: " Helios Operations Runbook - Confidential - Page 1"
page 2 starts with: "they receive every page but are not expected to act, and they write up what"
paragraphs that no single page contains whole, because a page break cuts them: 2 of 36The reader returns four Documents, one per page, with only the file name and page number as metadata. Two things stand out. The running footer (“Helios Operations Runbook – Confidential – Page 1”) is in the text of all four pages, so it would be embedded and retrievable like any other sentence. And the text is full of padding: 30,358 characters as read against 10,812 once runs of whitespace are collapsed. A splitter that counts tokens is spending most of its budget on spaces. Page breaks also cut 2 of the 36 paragraphs in two.
--- PagePdfDocumentReader with pageBottomMargin ---
pageBottomMargin= 30: footer on 4 of 4 pages, answer fragments still in the text: 37 of 37
pageBottomMargin= 60: footer on 0 of 4 pages, answer fragments still in the text: 37 of 37
pageBottomMargin=100: footer on 0 of 4 pages, answer fragments still in the text: 36 of 37
no margin: footer on 4 of 4 pages, answer fragments still in the text: 37 of 37pageBottomMargin tells the reader to ignore text in the bottom strip of each page. At 30 points the footer survives, at 60 it is gone and all 37 answer fragments are still present, and at 100 the margin starts eating real text (36 of 37). The right value is a property of your documents, and the safe way to find it is the one used here: count a few known facts before and after.
Word through Tika: one Document, footer included, no structure
--- TikaDocumentReader on the .docx ---
Documents returned: 1
metadata: {source=runbook.docx}
characters: 10657, contains the footer text: true
starts with: "Helios Cloud Platform Operations Runbook\n1. Backups and Restore\nEvery production databas"
blank-line-separated blocks in the text: 1, newline-separated lines: 50
answer fragments in the text: 37 of 37Tika returns the whole Word file as a single Document whose only metadata is the source. The text keeps one line per paragraph (50 lines), which matters later, but it also includes the footer text, which in Word lives outside the body. It is not removed by anything in the reader and, as the strategies section shows, it ends up as a chunk of its own.
HTML through Jsoup: you choose what counts as content
--- JsoupDocumentReader on the .html ---
default config: 1 Document, 10712 characters, contains nav link text "Status": true, contains "Cookie settings": true
default metadata: {description=How Helios runs production., title=Helios Cloud Platform Operations Runbook}
selector=article: 1 Document, 10615 characters, nav text: false, footer text: false
selector=section, groupByElement=true: 12 Documents
first section Document: 1188 characters, starts "1. Backups and Restore Every production database is snapshot"
with metadataTag("description") and additionalMetadata: {description=How Helios runs production., format=html, title=Helios Cloud Platform Operations Runbook}With no configuration Jsoup reads the whole body, so the navigation links and the cookie banner text are in the output. Setting selector to article removes them. Setting it to section with groupByElement gives twelve Documents, one per section, which is a free first level of chunking. The same page through Tika also keeps “Cookie settings” but, usefully, drops the text of the tracking script.
Going deeper: the Jsoup text is flattened to one line, and that matters for a paragraph chunker
Jsoup joins the selected elements with the separator. With selector=article and the default settings the result was a single line of text, so a chunker that works line by line produced 0 chunks. The pipeline therefore selects article h1, article h2, article p with "\n" as the separator, which gave 49 lines and 36 chunks. This is why IngestionPipeline.java throws when there is text but no chunks: a splitter that does not understand the reader's output returns an empty list without complaining, and an ingestion job that writes zero rows and reports success is a quiet way to lose a document.
JsoupDocumentReader selector=article: 1 line(s) of text -> 0 chunks from the paragraph chunker
selector="article h1, article h2, article p" with separator "\n": 49 lines -> 36 chunks
rows per format: [docx=37, html=36]
question: "How long is data kept?"Going deeper: reader dependencies and what else is on the menu
The module depends on spring-ai-pdf-document-reader, spring-ai-tika-document-reader and spring-ai-jsoup-document-reader, all managed by the Spring AI 2.0.2 BOM. Spring AI also has a ParagraphPdfDocumentReader that splits by the PDF's table of contents; it needs a PDF that has one, so it was not tried on this generated file. Tika can read the PDF and HTML too (it returned one Document of 10,861 characters for the PDF and 10,764 for the HTML here), which is convenient for a single code path but gives up the per-page metadata and the selector.
- Going deeper, if you want it: the reader tests are in
ReadersTest.java; the file writers that build the three formats are inFixtures.java.
Step two: chunk size is a trade, and the middle won
A chunk has to be small enough that its embedding is about one thing, and large enough to contain an answer whole. TokenTextSplitter cuts by token count; its chunkSize defaults to 800. Here is the same Word file cut at six sizes.
--- defaults except chunkSize (minChunkSizeChars=350) ---
chunkSize chunks meanToks hit@1 hit@3 hit@5 cut top3Toks
32 68 31.9 22/37 29/37 30/37 3 96
64 34 63.9 27/37 33/37 33/37 2 191
128 19 114.1 31/37 35/37 36/37 0 335
256 9 241.1 24/37 31/37 36/37 0 720
512 5 434.0 21/37 33/37 37/37 0 1278
800 3 723.3 17/37 37/37 37/37 0 2170Read the table from the top. At 32 tokens a chunk is a fragment: 68 chunks, and three answers are cut across a boundary so no chunk holds them. As size grows, hit@1 climbs to 31 at 128 tokens (19 chunks) and then falls, to 17 at 800. Large chunks mix several topics, so the embedding is a blur and the right one is often second or third. Hit@3 and hit@5 tell the opposite story, reaching 37 at 800, but with only three chunks in the whole document, “top three” is the entire document. That is the trap in reading hit@k alone, and the top3Toks column shows what it costs: 335 tokens of context per question at 128, 2,170 at 800.
What to take from it: if the model sees only the top few chunks, hit@1 and hit@3 at a modest context cost are the numbers that matter, and in this document that points to 128 tokens (about 114 on average). A different corpus will have a different best size; the point is that it is a measured number and not the library default.
Going deeper: the 350-character minimum silently changes small chunk sizes
TokenTextSplitter has a second knob, minChunkSizeChars, which defaults to 350 and tells the splitter not to end a chunk before it holds that many characters. As I read the splitter's behaviour, it tries to end each chunk at the last sentence punctuation inside the window, but only if that point is past the minimum. A 32-token window is shorter than 350 characters, so it never ends on punctuation and every chunk is a raw 32-token slice (mean 31.9 in the table above). Setting the minimum to 10 lets it end at sentence boundaries, and the result changes a lot at the small sizes:
--- same sizes with minChunkSizeChars=10 ---
chunkSize chunks meanToks hit@1 hit@3 hit@5 cut top3Toks
32 88 24.6 31/37 32/37 32/37 0 76
64 42 51.7 27/37 32/37 34/37 1 160
128 19 114.1 31/37 35/37 36/37 0 335
256 9 241.1 24/37 31/37 36/37 0 720
512 5 434.0 21/37 33/37 37/37 0 1278
800 3 723.3 17/37 37/37 37/37 0 2170At size 32, chunks go from 68 to 88, the number of answers cut falls from 3 to 0 and hit@1 jumps from 22 to 31. From 128 up the two tables are identical. The lesson is not that 32 is good; it is that a chunk-size experiment which leaves the other parameter at its default can measure the default instead.
Step three: cut on the document's own joints instead of on a token count
A token window knows nothing about sentences or paragraphs. This document has natural joints, so I tried four other ways to cut it, all measured on the same 37 questions. SentenceSplitter (in this module, SentenceSplitter.java) packs whole sentences up to a token target and can repeat the last sentence or two at the start of the next chunk. HeadingAwareChunker (HeadingAwareChunker.java) makes one chunk per paragraph and records the section heading as metadata. And the section-per-Document output from Jsoup is a ready-made twelve-chunk strategy.
strategy chunks meanToks hit@1 hit@3 hit@5 cut top3Toks
TokenTextSplitter 128 tokens 19 114.1 31/37 35/37 36/37 0 335
TokenTextSplitter 64 tokens, minChars=10 42 51.7 27/37 32/37 34/37 1 160
SentenceSplitter 128 tokens, no overlap 20 108.5 24/37 33/37 34/37 0 341
SentenceSplitter 128 tokens, 1 sentence overlap 24 113.5 30/37 33/37 36/37 0 337
SentenceSplitter 64 tokens, no overlap 41 52.8 30/37 34/37 34/37 0 161
one chunk per paragraph 37 56.4 34/37 36/37 36/37 0 176
one chunk per paragraph + section heading 37 62.7 32/37 35/37 36/37 0 194
one chunk per section (JsoupDocumentReader) 12 178.7 31/37 37/37 37/37 0 529One chunk per paragraph is the best at hit@1 (34 of 37) with the smallest top-three context (176 tokens). The Word-file token window at 128 tokens gets 31. A sentence splitter with no overlap does worse than the token window at hit@1 (24), and one sentence of overlap brings it back to 30, which says the loss came from chunk boundaries falling between a question's wording and its answer. One chunk per section reaches 37 at hit@3, again because a section is big (529 tokens for the top three).
I expected that putting the section heading in front of each paragraph would help, because the heading adds context a lone paragraph lacks. It did not, here: 32 at hit@1 against 34 without. With 37 questions that is a difference of two, so the honest reading is “no measurable gain on this corpus”, not “headings hurt”.
A footer became a chunk. The last of the 37 paragraph chunks is not a paragraph:text="Helios Operations Runbook - Confidential"under section 12. The Word footer came through Tika as a line of text and the chunker took it for the last paragraph. It is harmless for these questions and a nuisance in production, where it could come back for a query that happens to mention “Confidential”. Strip repeating lines (headers, footers, page numbers) before chunking, or cut them by position aspageBottomMargindoes for PDFs.
Going deeper: the paragraph chunker, and why it needs line-shaped text
The paragraph chunker works only if the reader preserves line breaks. It reads the text line by line, treats a short numbered line with no full stop (matching ^\d+\. [^.]{1,60}$) as a heading and everything else as a paragraph. That is a heuristic written for this document's shape, which is fine for a demonstration and wrong for a document whose headings are not numbered. For your files the same idea needs a different detector, or a reader that exposes the real structure (Jsoup's section groups, or a Word-aware reader), and the point of the test is that you can check the detector the same way: count your known facts before and after.
// from HeadingAwareChunker.apply: one chunk per paragraph, tagged with its section
if (HEADING.matcher(line).matches()) { section = line; continue; }
Map<String, Object> metadata = new LinkedHashMap<>(doc.getMetadata());
metadata.put("section", section);
metadata.put("chunk_index", index++);
out.add(Document.builder().text(prefixHeading ? section + "\n" + line : line).metadata(metadata).build());Step four: metadata is stored with the chunk, and may also be embedded
Every Document carries a metadata map: file name, page, section, whatever you add. It is useful for filtering and for showing a citation. There is a subtlety that is easy to miss: when the vector store computes an embedding it asks the Document for its “embed” text, and by default that text includes the metadata. The same chunk looks like this to the two modes:
--- the text each model would embed for the same chunk ---
MetadataMode.NONE : \n\nEvery production database is snapshotted once a day at 01:30 UTC. Daily snapshots are kept for 35 days, af
MetadataMode.EMBED: chunk_index: 0\nformat: docx\nsection: 1. Backups and Restore\nsource: runbook.docx\ncontent_hash: 86f5d5577ceThe output is cut at 110 characters per line by the test, but the shape is visible: MetadataMode.EMBED puts chunk_index, format, section, source and a content hash in front of the paragraph. Does that help retrieval?
--- hit rate, one chunk per paragraph, 37 questions ---
configuration chunks meanToks hit@1 hit@3 hit@5 cut top3Toks
MetadataMode.NONE 37 56.4 34/37 36/37 36/37 0 176
MetadataMode.EMBED, every key embedded 37 56.4 29/37 36/37 36/37 0 175
MetadataMode.EMBED, four keys in excludedEmbedMetadataKeys 37 56.4 31/37 35/37 36/37 0 174
embedded text with the four keys excluded: section: 1. Backups and Restore\n\nEvery production database is snapshotted once a day at 01:30 UTC. Daily snaEmbedding every key cost five first-place hits (34 down to 29), because a hash and an index number are noise that every chunk shares. Excluding four of the keys and leaving the section heading in (DefaultContentFormatter.builder().withExcludedEmbedMetadataKeys(...)) recovers most of it (31) but still does not beat plain text. On this corpus the best setting is the simplest. If you want the heading in the embedding, put it in the text on purpose, as the previous section did; if you only want it for filtering, keep it out of the embedding.
Going deeper: how the exclusion is wired
Metadata keys listed in the content formatter's excludedEmbedMetadataKeys are left out of the text used for embedding but stay in the map and in the stored row. The test builds the formatter with source, format, chunk_index and content_hash excluded and calls setContentFormatter on each Document; see MetadataEmbeddingTest.java. The same MetadataMode choice exists on the vector store's side, so check which one your store uses when you add Documents.
Does the file format change what you get back?
Same text, three files, the same TokenTextSplitter at 128 tokens. Only the reader and its settings change:
reader and configuration chunks meanToks hit@1 hit@3 hit@5 cut top3Toks
PDF PagePdfDocumentReader, each page split alone 36 106.6 24/37 32/37 33/37 1 333
PDF pages joined first, footer still there 34 112.9 26/37 33/37 34/37 1 338
PDF pages joined, pageBottomMargin=60 34 110.5 27/37 33/37 34/37 1 330
PDF pages joined, margin 60, whitespace collapsed 19 117.7 27/37 36/37 37/37 0 348
DOCX TikaDocumentReader 19 114.1 31/37 35/37 36/37 0 335
HTML JsoupDocumentReader, default selector (body) 19 114.8 30/37 34/37 36/37 0 342
HTML JsoupDocumentReader, selector=article 19 113.8 29/37 34/37 36/37 0 342Word and HTML land together (31, 30 and 29 at hit@1). The PDF is worse when read as the library hands it to you: 24 when each page is split alone, 26 and 27 after joining the pages. The line that stands out is the last PDF row: the same pages, with whitespace collapsed, give hit@3 36 and hit@5 37, the best of the 128-token runs. The reason is in the last block of that file:
--- tokens the splitter had to cut, by reader ---
PDF pages as read : 30358 characters, 3933 tokens
PDF whitespace collapsed: 10655 characters, 2251 tokens
DOCX : 10657 characters, 2172 tokensThe PDF text is 3,933 tokens against about 2,200 for the same words in the other formats. The extra tokens are whitespace padding, which is not harmless: it shifts every chunk boundary and it fills the embedding model's 256-token window with spaces. A one-line replaceAll("\\s+", " ") after reading a PDF is the cheapest improvement in this article. It also has a price: collapsing whitespace flattens line breaks, which a line-oriented chunker like the paragraph chunker needs, so use it for window-based splitting and not before a paragraph chunker.
Loading the same file twice must not double your index
The loop in most tutorials is read, split, vectorStore.add(chunks). Run it twice and every chunk is stored twice with a fresh random id, and queries start returning duplicates. This is what that does to a real pgvector table:
--- a naive loop: read, split, add, with random chunk ids ---
run 1: rows in table = 19, texts embedded so far = 19
run 2: rows in table = 38, texts embedded so far = 38
run 3: rows in table = 57, texts embedded so far = 57
rows saying "kept for 35 days": 3Nineteen rows became 38 and then 57, and three rows say “kept for 35 days”. If that fact later changes to 45 days, the old rows are still in the table, still being retrieved, and now contradict the new one. The fix has two parts, and both are in MetadataEnricher.java and IngestionPipeline.java.
String hash = sha256(chunk.getText()).substring(0, 16);
// ...
String id = UUID.nameUUIDFromBytes((source + "#" + hash).getBytes(StandardCharsets.UTF_8)).toString();An id built from the source name and a hash of the chunk text means the same chunk of the same file always gets the same id, and a chunk whose text changed gets a new one. The pipeline then reads the ids already stored for that source, skips the ones it already has, adds the new ones and deletes the ones that are gone. The result with the 128-token splitter:
--- TokenTextSplitter(128): windows of tokens ---
1. first ingest of v1 chunks=19 embedded=19 unchanged= 0 deleted= 0 | rows=19, texts embedded in total=19
2. same file again chunks=19 embedded= 0 unchanged=19 deleted= 0 | rows=19, texts embedded in total=19
3. same file a third time chunks=19 embedded= 0 unchanged=19 deleted= 0 | rows=19, texts embedded in total=19
4. v2: one paragraph edited chunks=19 embedded= 1 unchanged=18 deleted= 1 | rows=19, texts embedded in total=20
rows saying "kept for 35 days": 0, rows saying "kept for 45 days": 1
best match for the snapshot question now contains "45 days": true
5. v3: section 12 deleted from the file chunks=17 embedded= 1 unchanged=16 deleted= 3 | rows=17, texts embedded in total=22
rows saying "180 days in advance" (a fact only section 12 holds): 0Re-running the same file embeds nothing. Editing one paragraph embeds one chunk and deletes one. Deleting section 12 from the file removes its rows, and a fact that only that section holds is no longer retrievable. All of this is a “diff by id”, not an update in place, which is why no stale “35 days” row survives.
Notice the last step with the token splitter: deleting a section embedded one chunk, because the windows after the deletion shifted and one boundary now falls somewhere new. With one chunk per paragraph it is cleaner:
--- HeadingAwareChunker: one chunk per paragraph ---
1. first ingest of v1 chunks=37 embedded=37 unchanged= 0 deleted= 0 | rows=37, texts embedded in total=37
2. same file again chunks=37 embedded= 0 unchanged=37 deleted= 0 | rows=37, texts embedded in total=37
3. same file a third time chunks=37 embedded= 0 unchanged=37 deleted= 0 | rows=37, texts embedded in total=37
4. v2: one paragraph edited chunks=37 embedded= 1 unchanged=36 deleted= 1 | rows=37, texts embedded in total=38
rows saying "kept for 35 days": 0, rows saying "kept for 45 days": 1
best match for the snapshot question now contains "45 days": true
5. v3: section 12 deleted from the file chunks=34 embedded= 0 unchanged=34 deleted= 3 | rows=34, texts embedded in total=39
rows saying "180 days in advance" (a fact only section 12 holds): 0Here the same deletion embeds nothing and deletes three rows. How much a single edit costs depends on the chunker, and it was measured for every one of the 36 paragraphs (append a sentence, count the chunks whose text changes):
splitter chunks mean median max
TokenTextSplitter 128 tokens 19 5.0 2 17
TokenTextSplitter 256 tokens 9 2.5 2 10
SentenceSplitter 128, no overlap 19 2.4 2 8
SentenceSplitter 128, 1 sentence overlap 24 3.1 2 13
one chunk per paragraph 36 1.0 1 1
columns: chunks in the unedited file; mean, median and max chunks re-embedded over the 36 editsWith a 128-token window an edit re-embeds 5.0 chunks on average, and in the worst case 17 of the 19, because a longer paragraph pushes every later window along. The paragraph chunker always re-embeds exactly one. Overlap makes it worse (3.1 against 2.4 for the sentence splitter). If your documents are edited often and embedding costs real money, this column may matter more than hit@1.
Going deeper: what this does not protect you from
Ids from content hashes make ingestion repeatable, but they do not coordinate concurrent ingests of the same source (two jobs can both read the stored ids before either writes; I did not test that case). They also treat a renamed file as a new source, so the old rows stay until you delete them by the old name. And MetadataEnricher hashes only the chunk text, so a change to metadata alone (a new section label) does not change the id and will not be re-embedded; include the metadata in the hash if that matters to you.
Metadata earns its place at query time
The previous sections kept metadata out of the embedding. Its real job is filtering, and it is also what makes a table that holds several versions of the same document usable. After loading both runbook.docx and runbook.html into one table (73 rows), a question returns every hit twice, once per format:
--- no filter, top 5 ---
0.640 html 12. Decommissioning After a customer contract ends, the customer's data is del
0.640 docx 12. Decommissioning After a customer contract ends, the customer's data is del
0.588 html 1. Backups and Restore Every production database is snapshotted once a day at 01:
0.588 docx 1. Backups and Restore Every production database is snapshotted once a day at 01:
0.569 html 9. Logging and Monitoring Metrics are stored at 10 second resolution for 7 days, the
(5 results)The pgvector store takes a filter expression on the metadata written at ingestion. Filtering to one section, or to one format and one section, removes the duplicates and the unrelated sections:
--- filter: format == 'html' && section == '1. Backups and Restore' ---
0.588 html 1. Backups and Restore Every production database is snapshotted once a day at 01:
0.468 html 1. Backups and Restore Point-in-time recovery is available for the last 7 days. T
0.254 html 1. Backups and Restore A backup that has never been restored is only a hope. On t
(3 results)This is the reason to write source, format and section at ingestion even though none of them helps the embedding: they let you answer “only from the current handbook” or “only from the Backups section” without a second index.
Should you build this yourself?
An honest answer. If your files are ordinary prose and you are happy with a token window, the shortest pipeline isTikaDocumentReaderandTokenTextSplitterand it is a reasonable start. The parts worth adding early are the cheap ones the measurements point to: collapse PDF whitespace, cut footers by position, and give chunks deterministic ids so a re-run is harmless. The parts that need evidence before you invest are chunking strategy and chunk size, and the way to get evidence is a labelled question set, which is a day of work for a small corpus. If your documents are mostly tables, scanned images or heavily structured, none of these readers will do well and you need a layout-aware extraction step first. Last, this was one synthetic document and one small embedding model; a larger model with a longer input window changes the chunk-size result, and only your own questions can tell you by how much.
Further reading
- Production-grade RAG with Spring AI and the complete example for what happens after retrieval.
- Choosing a Vector Store for Spring AI if pgvector is not the store you will use.
- Semantic Caching for LLM Calls in Spring Boot for caching answers to repeated questions.
- Testing LLM Apps in Java for evaluating answers, once the right chunks are retrieved.
- The companion code: rag-ingestion. Run
scripts/run-all.shto regenerate every file underoutput/.
Sources: Spring AI reference, ETL pipeline.
No Comments yet!