Add rag module: Spring AI 2.0 RAG with pgvector, chunking, reranking and a faithfulness check

Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01B38FGKKam5SCGgwgduVAh3
2026-09-21 19:09:05 +00:00
commit 1d4625a1c2
62 changed files with 3715 additions and 0 deletions
@@ -0,0 +1,63 @@
+# 1. The shape of a RAG pipeline
+
+[Index](../README.md) · next: [2. Chunking](02-chunking.md)
+
+A language model only knows what it was trained on and what is in the prompt. Retrieval-augmented generation
+(RAG) puts the right few paragraphs of *your* documents into the prompt, so the model can answer from them.
+Everything else in this repository is a decision about which paragraphs, how they get there, and what to do
+when the answer is not in them.
+
+## Two phases, six stages
+
+```
+ ingest (once per file)                      query (once per question)
+ ──────────────────────                      ─────────────────────────
+ PDF ─► read pages ─► chunk ─► embed ─► store      question ─► embed ─► search ─► rerank ─► prompt ─► model ─► check
+        IngestionService        VectorStore                    └────── RetrievalAugmentationAdvisor ──────┘  RagQueryService
+```
+
+| Stage | Class here | Spring AI part it stands on |
+|---|---|---|
+| read | [`IngestionService`](../src/main/java/com/ankurm/rag/ingest/IngestionService.java) | `PagePdfDocumentReader` |
+| chunk | `DocumentTransformer chunker` in [`RagConfig`](../src/main/java/com/ankurm/rag/config/RagConfig.java) | `TokenTextSplitter` (or [`RecursiveChunker`](../src/main/java/com/ankurm/rag/chunk/RecursiveChunker.java), [`SemanticChunker`](../src/main/java/com/ankurm/rag/chunk/SemanticChunker.java)) |
+| embed + store | `VectorStore` bean | `PgVectorStore`, built by the pgvector starter |
+| search | `retrievalAdvisor` bean | `VectorStoreDocumentRetriever` |
+| rerank | [`LlmReranker`](../src/main/java/com/ankurm/rag/query/LlmReranker.java) | the `DocumentPostProcessor` hook (Spring AI ships no reranker) |
+| prompt | `retrievalAdvisor` bean | `ContextualQueryAugmenter` |
+| model + check | [`RagQueryService`](../src/main/java/com/ankurm/rag/query/RagQueryService.java) | `ChatClient`, `FactCheckingEvaluator` |
+
+The beans as the application actually wires them are in [output 01](output/01-pipeline-beans.txt).
+
+## The smallest thing that works
+
+One advisor, no other class of ours:
+
+```java
+ChatClient.builder(chatModel)
+    .defaultAdvisors(QuestionAnswerAdvisor.builder(vectorStore)
+        .searchRequest(SearchRequest.builder().topK(2).similarityThreshold(0.3).build())
+        .build())
+    .build();
+```
+
+That is [`SimpleAdvisorTest`](../src/test/java/com/ankurm/rag/SimpleAdvisorTest.java). [Output 11](output/11-question-answer-advisor.txt)
+shows the two prompts it produces. For a question the store can answer, the chunks are pasted between two lines of
+dashes. For a question nothing matches, the prompt still has the same template with an **empty** block between the
+dashes, and the model is left to follow the template's last sentence ("if the answer is not in the context, inform
+the user that you can't answer"). Nothing in code decides that nothing was found. That is the first thing the
+larger pipeline adds: see [chapter 4](04-retrieval-and-reranking.md).
+
+## What this repository does not test
+
+The chat model and the embedding model are scripted stand-ins
+([`FakeChatModel`](../src/test/java/com/ankurm/rag/support/FakeChatModel.java),
+[`HashingEmbeddingModel`](../src/test/java/com/ankurm/rag/support/HashingEmbeddingModel.java)). That is deliberate:
+anyone can run every test with no API key, and what is asserted is what Spring AI *sends* and how this code
+*handles* what comes back. It also means **nothing here shows how a real model answers, how a real embedding
+model scores similarity, whether reranking improves answers, or how good a real judge is at spotting an
+invented claim.** Similarity scores and thresholds in the outputs belong to the hashing model; do not carry them over.
+
+The vector store is real: every test that touches persistence runs against PostgreSQL with pgvector
+(`scripts/pg-up.sh` or `docker-compose.yml`), and the end-to-end test starts the whole application over HTTP.
+
+Next: [2. Chunking](02-chunking.md)
@@ -0,0 +1,59 @@
+# 2. Chunking
+
+prev: [1. The shape of a RAG pipeline](01-the-shape-of-a-rag-pipeline.md) · [Index](../README.md) · next: [3. Ingestion](03-ingestion.md)
+
+A chunk is the unit that gets embedded, stored, retrieved and pasted into the prompt. Cut too big and the
+prompt fills with text unrelated to the question; cut too small and an answer is split across two chunks.
+
+This chapter measures what the chunkers do to one fixed document, not which one retrieves best. **There is no
+retrieval-quality benchmark in this repository**, so nothing here says which strategy is best for your
+documents. The fixture is forty numbered sentences, 4,268 characters, 880 `cl100k_base` tokens.
+
+## `TokenTextSplitter` in Spring AI 2.0.1
+
+Transcript: [output 02](output/02-token-text-splitter.txt). Test: [`ChunkingTest`](../src/test/java/com/ankurm/rag/ChunkingTest.java).
+
+- **The defaults are large.** `new TokenTextSplitter()` (800-token chunks) turned the 880-token fixture into
+  two chunks of 792 and 88 tokens. A short handbook page becomes one chunk.
+- **It prefers sentence ends.** With `withChunkSize(100)` the result was ten chunks of 88 tokens, every one ending
+  on a full stop; it did not cut at exactly 100 tokens.
+- **It has no overlap.** Zero of nine neighbouring boundaries repeated any text. A sentence that straddles a cut
+  belongs to one chunk only.
+- **`withMinChunkSizeChars` did nothing on this fixture.** 350 and 50 produced identical ten-chunk results. Reading
+  the bytecode of the splitter: when a chunk exceeds the token limit it is cut back to its last sentence-ending
+  mark, but only if that mark is more than `minChunkSizeChars` characters into the chunk. On text with a full stop
+  every ~100 characters the mark is always far enough in, so neither value matters. Do not read this as "the
+  setting is useless", only as "this document does not exercise it".
+- **The constructor changed.** The 1.x five-argument constructor `new TokenTextSplitter(512, 128, 5, 10_000, true)`
+  does not exist in 2.0.1; use the builder. See [output 14](output/14-legacy-1x.txt) and [output 12](output/12-api-facts.txt).
+
+## `RecursiveChunker` (ours)
+
+Transcript: [output 03](output/03-recursive-and-semantic-chunkers.txt). Source: [`RecursiveChunker`](../src/main/java/com/ankurm/rag/chunk/RecursiveChunker.java).
+
+Cuts at the coarsest boundary that fits (blank line, line break, sentence end, space, and only then mid-word),
+and repeats the last `overlapChars` characters at the start of the next chunk. With `(400, 80)` the fixture became
+14 chunks, longest 395 characters, and all 13 boundaries carried overlap. Sizes are characters, not tokens. A
+900-character string with no separators is cut hard at `[400, 400, 100]`.
+
+The overlap starts at a word boundary, so a chunk often *opens mid-sentence* ("for item 3 and states th..."). That is
+the price of overlap; whether it helps retrieval was not measured.
+
+## `SemanticChunker` (ours)
+
+Same transcript. Source: [`SemanticChunker`](../src/main/java/com/ankurm/rag/chunk/SemanticChunker.java).
+
+Embeds every sentence in **one batched call** and starts a new chunk where the cosine distance between neighbours exceeds a
+threshold. On twelve sentences about three topics it produced three chunks, one per topic, and sent 12 texts to the
+embedding model. That result is from the hashing model, which is good at exactly this (topics share words). With a
+real embedding model the right threshold is different and must be found on your own text; the value 0.9 has no meaning
+outside this test. It costs an embedding call per sentence at ingestion time.
+
+## Choosing
+
+Start with `TokenTextSplitter` and a size you set on purpose. Move to the recursive chunker when a fact is being cut
+in half at the boundary and you can see it in real retrievals. Consider semantic chunking only when documents mix
+topics without headings and you can afford the extra embedding calls. Whichever you use, change it by replacing the
+one `DocumentTransformer` bean; nothing else in the pipeline notices.
+
+prev: [1](01-the-shape-of-a-rag-pipeline.md) · next: [3. Ingestion](03-ingestion.md)
@@ -0,0 +1,58 @@
+# 3. Ingestion
+
+prev: [2. Chunking](02-chunking.md) · [Index](../README.md) · next: [4. Retrieval and reranking](04-retrieval-and-reranking.md)
+
+Source: [`IngestionService`](../src/main/java/com/ankurm/rag/ingest/IngestionService.java),
+[`IngestionTracker`](../src/main/java/com/ankurm/rag/ingest/IngestionTracker.java).
+Test: [`IngestionTest`](../src/test/java/com/ankurm/rag/IngestionTest.java).
+
+## What a PDF page becomes
+
+`PagePdfDocumentReader` with `withPagesPerDocument(1)` returns one `Document` per page, and the only metadata it adds
+is `page_number` ([output 04](output/04-pdf-pages-and-metadata.txt)). That is the number a citation needs, so keep one
+page per document until you have chosen a chunker; chunks inherit their page's metadata.
+
+The text is padded. In the sample PDF, page 2 came back as 862 characters with a run of 133 spaces at the end of a
+line, and 304 characters after `IngestionService.tidy` collapsed the padding. Padding costs tokens and changes what gets
+embedded. **The sample PDFs are generated by PDFBox inside this repository**, so the exact padding is an artefact of that
+generator and the reader; real PDFs pad differently. Look at what your own files produce before assuming it matches.
+
+Every chunk also gets `source_file`, `source_hash` and whatever the caller passes (the HTTP endpoint adds `tenant_id`
+and `doc_type`). Chunks written by `TokenTextSplitter` additionally carry `parent_document_id`, `chunk_index`
+and `total_chunks`. One stored row, as the end-to-end test read it back from PostgreSQL:
+
+```
+{"doc_type": "general", "tenant_id": "globex", "chunk_index": 0, "page_number": 1, "source_file": "globex-manual.pdf", "total_chunks": 1}
+```
+
+([output 10](output/10-end-to-end.txt); `source_hash` and `parent_document_id` were left out of that query.)
+
+## Uploading the same file twice
+
+[Output 05](output/05-ingestion-idempotency.txt), on real PostgreSQL, counting rows with `select count(*)`:
+
+| step | what happened | rows | texts embedded |
+|---|---|---|---|
+| first upload | `ingested`, 4 chunks | 4 | 4 |
+| same bytes again | `skipped` | 4 | 4 (nothing spent) |
+| page 2 edited | `updated`, 4 old chunks deleted first | 4 | 8 |
+| same text exported again | `updated` | 4 | 12 |
+| *naive:* `vectorStore.add()` on every upload, twice more | rows tripled | 12 | – |
+
+The naive version is what many tutorials show: each upload writes new chunk ids, so the same page is stored three times
+and retrieval returns duplicates that crowd out other passages. The service instead remembers, per file name, the SHA-256 of the
+bytes and the ids of the chunks it wrote, skips identical bytes, and on a change deletes the old chunk **by id** before adding.
+After the edit, zero rows still said "20 working days".
+
+The last row is a limit, not a feature: the hash is over the *file's bytes*. Two PDFs with identical text but different bytes
+(the PDFBox generator in the tests produces different bytes on every run for the same text; I did not check which field differs) count as a change and are re-embedded. Hashing the extracted
+text instead would avoid that, at the cost of reading the file first.
+
+## What the tracker does not do
+
+It lives in memory. A restart forgets every file, and the next upload of each one is re-embedded and its old chunks
+are **not** deleted, because the tracker no longer knows their ids. Two fixes: persist the tracker in a table, or store
+`source_file` in metadata (already done) and delete by filter, `vectorStore.delete(new FilterExpressionBuilder().eq("source_file", name).build())`.
+The second is not exercised by any test here.
+
+Next: [4. Retrieval and reranking](04-retrieval-and-reranking.md)
@@ -0,0 +1,65 @@
+# 4. Retrieval and reranking
+
+prev: [3. Ingestion](03-ingestion.md) · [Index](../README.md) · next: [5. Generation and the faithfulness check](05-generation-and-the-faithfulness-check.md)
+
+Tests: [`RetrievalTest`](../src/test/java/com/ankurm/rag/RetrievalTest.java),
+[`TenantFilterTest`](../src/test/java/com/ankurm/rag/TenantFilterTest.java),
+[`RerankTest`](../src/test/java/com/ankurm/rag/RerankTest.java). Wiring: [`RagConfig`](../src/main/java/com/ankurm/rag/config/RagConfig.java).
+
+**All similarity scores in this chapter come from the hashing embedding model.** They show how the components behave, not what a
+real model would score.
+
+## Top-k, and the threshold that is not there by default
+
+`VectorStoreDocumentRetriever` returns the `topK` nearest chunks. Its default similarity threshold accepts everything, and
+[output 06](output/06-retrieval-threshold-and-prompt.txt) shows what follows. For a question about leave, a threshold of 0.0 returned all four
+chunks (scores 0.59, 0.41, 0.25, 0.20); 0.3 returned two. For an off-topic question ("capital of Mongolia") the default still
+returned all four chunks, each with score 0.0000, and all four went into the prompt: **the "nothing found" safety net never
+fires while the threshold accepts everything.** With 0.3 nothing was retrieved.
+
+Which threshold is right depends entirely on the embedding model and cannot be copied from here. Find yours by asking questions
+you know the corpus cannot answer and looking at the top scores.
+
+## The empty-context path
+
+`ContextualQueryAugmenter` builds the prompt. With `allowEmptyContext(false)` (what `RagConfig` sets) and no chunks, the model
+receives only "The user query is outside your knowledge base. Politely inform the user that you can't answer it."
+With `allowEmptyContext(true)` the model receives the bare question and answers from whatever it was trained on, which for a
+document-QA service is the failure you built RAG to avoid. Both prompts are in output 06.
+`QuestionAnswerAdvisor` has no such switch: [output 11](output/11-question-answer-advisor.txt) shows it sending an empty context block instead.
+
+## Tenant isolation
+
+Put `tenant_id` in every chunk's metadata at ingestion (the controller does) and pass a filter at query time.
+[Output 07](output/07-tenant-filter-and-injection.txt), on the in-memory store and on PostgreSQL alike:
+
+- with no filter, the top three chunks for "How many days of annual leave" included one from the other tenant;
+- `new FilterExpressionBuilder().eq("tenant_id", "acme").build()` returned only that tenant's chunks;
+- a filter *string* built as `"tenant_id == '" + input + "'"` with `input = globex' || tenant_id == 'acme` parsed to
+  `tenant_id == 'globex' || tenant_id == 'acme'` and returned both tenants' chunks;
+- the same hostile text handed to `eq("tenant_id", input)` returned nothing, with a single or a double quote in it, on both stores.
+
+So: never build a filter string from a request value; pass the value to the builder. That is what
+[`RagController`](../src/main/java/com/ankurm/rag/web/RagController.java) does. The two hostile strings tried are the ones in the
+test; this is not a security audit of the filter parser.
+
+**Not tested:** how the filter performs. In pgvector, an approximate (HNSW) index is scanned first and the `WHERE` is applied to what
+it returns, so a selective filter can return fewer than `topK` rows; pgvector 0.8.0 added iterative index scans to address this
+([pgvector README, "Filtering"](https://github.com/pgvector/pgvector#filtering)). This repository ran pgvector **0.6.0**, which predates them,
+and never ran a filtered query against enough rows to see the effect. If one tenant is a small fraction of the table, test it.
+
+## Reranking
+
+Spring AI 2.0.1 has the hook (`DocumentPostProcessor`) and no reranker: `spring-ai-rag` contains no class with "rerank" in its
+name ([output 12](output/12-api-facts.txt)). [`LlmReranker`](../src/main/java/com/ankurm/rag/query/LlmReranker.java) is ours: it asks the
+chat model to rate each candidate 0-10 and keeps the best `topN` ([output 08](output/08-reranking.txt)).
+
+- **Cost:** one model call per candidate. Four candidates made four calls; with `topK` 20 every question costs 20 rating calls before the answer.
+- **Latency:** the calls run on virtual threads. With 200 ms per call standing in for the network, 20 calls took over 4 s one after another
+  and under 1 s through the reranker. That shows the structure works; a real API's rate limits are not modelled.
+- **Failure:** the reply must be a bare integer. `"Score: 8"` failed to parse for every candidate, all scores became 0, and the survivors were the
+  vector-search order. Nothing errors; only the `rag.rerank.failures` counter moves. A reply of `" 9\n"` parsed fine. Watch the counter.
+
+**Not tested:** whether reranking improves which chunks reach the prompt. The scripted rater is word overlap and agrees with the fake embeddings.
+
+Next: [5. Generation and the faithfulness check](05-generation-and-the-faithfulness-check.md)
@@ -0,0 +1,28 @@
+# 5. Generation and the faithfulness check
+
+prev: [4. Retrieval and reranking](04-retrieval-and-reranking.md) · [Index](../README.md) · next: [6. Observability and a production checklist](06-observability-and-production-checklist.md)
+
+Source: [`RagQueryService`](../src/main/java/com/ankurm/rag/query/RagQueryService.java). Test:
+[`FaithfulnessTest`](../src/test/java/com/ankurm/rag/FaithfulnessTest.java) ([output 09](output/09-faithfulness-check.txt)).
+
+After the model answers, `RagQueryService` asks a second model call, `FactCheckingEvaluator`, whether the answer is supported by the
+chunks that were in the prompt. The response reports one of three statuses:
+
+| status | when | fact check runs? |
+|---|---|---|
+| `answered` | chunks retrieved and the judge said "yes" | yes |
+| `ungrounded` | chunks retrieved and the judge said anything other than "yes" (case aside), or the judge call failed | yes |
+| `no_context` | nothing retrieved | no |
+
+Behaviours the test pins down:
+
+- **The verdict is matched literally.** A judge reply of `Yes.` counted as *not* grounded; `YES` counted as grounded. The evaluator
+  compares against "yes" ignoring case, not ignoring punctuation. A real judge model that likes to add a full stop would fail
+  every answer; instruct it to answer with one word, and watch `rag_queries_total{status="ungrounded"}`.
+- **A failing judge is not a passing judge.** If the judge call throws (a rate limit, a timeout) the answer is reported `ungrounded` and
+  `rag.faithfulness.judge_failures` increments. The answer text is still returned, so the caller decides what to do with an unchecked answer.
+- **No chunks, no check.** With nothing to check against, the service skips the judge (zero fact-check calls in the output) and reports `no_context`.
+- **The check is only as good as the judge.** The tests script the judge, so they prove the wiring and the four outcomes, not that a real model
+  spots an invented claim. Measure that on your own questions before you rely on it, and note that it doubles the model calls per question.
+
+Next: [6. Observability and a production checklist](06-observability-and-production-checklist.md)
@@ -0,0 +1,38 @@
+# 6. Observability and a production checklist
+
+prev: [5. Generation and the faithfulness check](05-generation-and-the-faithfulness-check.md) · [Index](../README.md)
+
+## What the application exports
+
+`GET /actuator/prometheus` from the end-to-end run ([output 10](output/10-end-to-end.txt)) contained these `rag_` series after two
+ingested files, one skipped re-upload and two questions:
+
+```
+rag_chunks_ingested_total 5.0
+rag_context_chunks_count 2
+rag_context_chunks_sum 5.0
+rag_context_chunks_max 4.0
+rag_ingestion_skipped_total 1.0
+rag_queries_total{status="answered"} 2.0
+rag_query_duration_seconds_count{status="answered"} 2
+```
+
+Counters that exist in the code but were not triggered in that run (so they do not appear): `rag_rerank_failures_total` and
+`rag_faithfulness_judge_failures_total`. Micrometer registers a counter when it is first incremented.
+
+The ones worth alerting on: a rising share of `ungrounded` and `no_context` in `rag_queries_total`; any increase in `rag_rerank_failures_total`
+(silent degradation, chapter 4); any increase in `rag_faithfulness_judge_failures_total`; and `rag_context_chunks` drifting toward its maximum
+(the threshold is accepting everything).
+
+## Checklist
+
+- [ ] Threshold set from questions your corpus cannot answer (chapter 4), not copied from a tutorial.
+- [ ] `allowEmptyContext(false)`, and a test that an off-topic question produces `no_context`.
+- [ ] Tenant filter built with `FilterExpressionBuilder`, never string concatenation; a test with a hostile tenant id.
+- [ ] Filtered queries tested on a table the size of production (HNSW and `WHERE`, chapter 4).
+- [ ] The ingestion tracker persisted, or deletion by `source_file` filter (chapter 3).
+- [ ] Judge prompt forces a one-word answer; `ungrounded` is shown to the user as such, not hidden.
+- [ ] Reranker replies parsed with a fallback and the failure counter alerted (chapter 4). Consider a structured-output call instead of parsing text.
+- [ ] `init.sql` (or your migration tool) owns the schema; `initialize-schema` stays `false`.
+- [ ] Embedding model and vector dimension changed together, and the table rebuilt when either changes: vectors from two models are not comparable.
+- [ ] An evaluation set of real questions with known answers, run on every change to chunking, threshold or models. **This repository has none**; it is the largest gap between this code and a production system.
@@ -0,0 +1,13 @@
+# The pipeline as Spring beans (the shipped configuration, fake models)
+
+stage / role                 bean name                    actual class
+DocumentTransformer          chunker                      TokenTextSplitter
+EmbeddingModel               embeddingModel               HashingEmbeddingModel
+VectorStore                  vectorStore                  PgVectorStore
+RetrievalAugmentationAdvisor retrievalAdvisor             RetrievalAugmentationAdvisor
+LlmReranker                  reranker                     LlmReranker
+ChatModel                    chatModel                    FakeChatModel
+ChatClient                   ragChatClient                DefaultChatClient
+FactCheckingEvaluator        factChecker                  FactCheckingEvaluator
+IngestionService             ingestionService             IngestionService
+RagQueryService              ragQueryService              RagQueryService
@@ -0,0 +1,27 @@
+# What TokenTextSplitter does to a 40-sentence document
+
+document: 4268 characters, 880 cl100k_base tokens
+
+--- new TokenTextSplitter()  (defaults: 800 tokens, 350 min chars, 5 min length to embed) ---
+chunks: 2, sizes in tokens: [792, 88]
+
+--- chunk size 100 tokens ---
+chunks: 10
+chunk 0:  88 tokens,  419 chars, starts "Sentence 1 describes t", ends "ry full-time employee."
+chunk 1:  88 tokens,  420 chars, starts "Sentence 5 describes t", ends "ry full-time employee."
+chunk 2:  88 tokens,  426 chars, starts "Sentence 9 describes t", ends "ry full-time employee."
+chunk 3:  88 tokens,  428 chars, starts "Sentence 13 describes ", ends "ry full-time employee."
+chunk 4:  88 tokens,  427 chars, starts "Sentence 17 describes ", ends "ry full-time employee."
+chunk 5:  88 tokens,  427 chars, starts "Sentence 21 describes ", ends "ry full-time employee."
+chunk 6:  88 tokens,  428 chars, starts "Sentence 25 describes ", ends "ry full-time employee."
+chunk 7:  88 tokens,  428 chars, starts "Sentence 29 describes ", ends "ry full-time employee."
+chunk 8:  88 tokens,  428 chars, starts "Sentence 33 describes ", ends "ry full-time employee."
+chunk 9:  88 tokens,  427 chars, starts "Sentence 37 describes ", ends "ry full-time employee."
+metadata of chunk 1 (parent_document_id, a random UUID, left out): {chunk_index=1, page_number=1, source_file=long.txt, total_chunks=10}
+
+--- is there any overlap between neighbouring chunks? ---
+boundaries where the first 30 characters of a chunk already appear in the chunk before it: 0 of 9
+
+--- minChunkSizeChars: where the cut lands ---
+minChunkSizeChars=350: 10 chunks, 9 of the first 9 end on a full stop
+minChunkSizeChars= 50: 10 chunks, 9 of the first 9 end on a full stop
@@ -0,0 +1,31 @@
+# RecursiveChunker and SemanticChunker
+
+
+--- RecursiveChunker(maxChars=400, overlapChars=80) ---
+chunks: 14, longest: 395 characters
+chunk 0: 314 chars, starts "Sentence 1 describes the", ends "very full-time employee."
+chunk 1: 388 chars, starts "for item 3 and states th", ends "very full-time employee."
+chunk 2: 387 chars, starts "for item 6 and states th", ends "very full-time employee."
+chunk 3: 394 chars, starts "for item 9 and states th", ends "very full-time employee."
+chunk 4: 394 chars, starts "for item 12 and states t", ends "very full-time employee."
+chunk 5: 395 chars, starts "for item 15 and states t", ends "very full-time employee."
+chunk 6: 395 chars, starts "for item 18 and states t", ends "very full-time employee."
+chunk 7: 394 chars, starts "for item 21 and states t", ends "very full-time employee."
+chunk 8: 395 chars, starts "for item 24 and states t", ends "very full-time employee."
+chunk 9: 394 chars, starts "for item 27 and states t", ends "very full-time employee."
+chunk 10: 395 chars, starts "for item 30 and states t", ends "very full-time employee."
+chunk 11: 395 chars, starts "for item 33 and states t", ends "very full-time employee."
+chunk 12: 394 chars, starts "for item 36 and states t", ends "very full-time employee."
+chunk 13: 180 chars, starts "for item 39 and states t", ends "very full-time employee."
+metadata of chunk 2: {chunk_index=2, chunk_total=14, page_number=1, source_file=long.txt}
+boundaries where the next chunk opens with words the previous one ended with: 13 of 13
+
+--- one 900-character word-salad with no separators falls back to a hard cut ---
+chunks: [400, 400, 100]
+
+--- SemanticChunker(distance 0.9) on three topics, four sentences each ---
+chunks: 3
+  [0] Annual leave is twenty days per year. Unused annual leave carries over until March. Leave requests go through the HR portal. Annual leave accrues monthly.
+  [1] Expenses need a receipt above fifty euros. Expense claims must be filed within thirty days. Receipts for expenses are uploaded as photos. Approved expenses are paid with salary.
+  [2] Remote work is allowed two days per week. Managers agree the remote work days. Remote work needs a quiet workspace. Remote work days are recorded in the calendar.
+texts sent to the embedding model: 12 (12 sentences, one batched call)
@@ -0,0 +1,27 @@
+# PagePdfDocumentReader: what one PDF page becomes
+
+pages in the PDF: 4, documents read: 4
+page document metadata: {page_number=1}
+page document metadata: {page_number=2}
+page document metadata: {page_number=3}
+page document metadata: {page_number=4}
+
+--- page 2 as read (single spaces as dots, runs of 4+ as [n spaces], line ends as a pilcrow) ---
+¶
+[12 spaces]4.1··Annual···Leave···Entitlement.···Full-time···employees[5 spaces]are··entitled·to·20··working···days[104 spaces]¶
+[12 spaces]of·annual···leave··per··calendar···year.··Part-time···employees[5 spaces]receive···leave··pro··rata.[107 spaces]¶
+¶
+[12 spaces]4.2··Leave···Carryover.···Unused[5 spaces]annual···leave··may···be··carried··over··for·a·maximum[6 spaces]of·5·days[96 spaces]¶
+[12 spaces]into·the··next··calendar···year··and···must··be··used···by·31··March.[133 spaces]¶
+
+
+--- page 2 after IngestionService.tidy ---
+4.1·Annual·Leave·Entitlement.·Full-time·employees·are·entitled·to·20·working·days¶
+of·annual·leave·per·calendar·year.·Part-time·employees·receive·leave·pro·rata.¶
+¶
+4.2·Leave·Carryover.·Unused·annual·leave·may·be·carried·over·for·a·maximum·of·5·days¶
+into·the·next·calendar·year·and·must·be·used·by·31·March.
+
+--- size of page 2 ---
+as read : 862 characters, longest run of spaces 133
+tidied  : 304 characters, longest run of spaces 1
@@ -0,0 +1,14 @@
+# Ingesting the same handbook more than once (real PostgreSQL + pgvector)
+
+rows are counted with: select count(*) from vector_store
+1. first upload                    -> status=ingested chunksWritten=4 chunksReplaced=0 | rows in table=4, texts embedded so far=4
+2. same bytes again                -> status=skipped  chunksWritten=0 chunksReplaced=0 | rows in table=4, texts embedded so far=4
+3. page 2 edited (20 -> 22 days)   -> status=updated  chunksWritten=4 chunksReplaced=4 | rows in table=4, texts embedded so far=8
+rows still saying "20 working days": 0, rows saying "22 working days": 1
+4. same text, exported again       -> status=updated  chunksWritten=4 chunksReplaced=4 | rows in table=4, texts embedded so far=12
+the two PDFs have identical text and different bytes: true
+the file hash is a hash of bytes, so a re-export counts as a change and is re-embedded
+
+--- the naive version: vectorStore.add() on every upload, nothing remembered ---
+the same 4 chunks added on two more uploads: rows in table 4 -> 12
+rows saying "22 working days" now: 3
@@ -0,0 +1,61 @@
+# Retrieval: threshold, prompt shape and the empty-context path
+
+the store holds the 4 pages of one handbook as 4 chunks; topK = 20
+
+--- question: "How many days of annual leave do employees get?" ---
+similarityThreshold 0.0 (the default): 4 chunk(s)
+    score 0.5891  page 2  "4.1 Annual Leave Entitlement. Full-time empl..."
+    score 0.4077  page 3  "4.3 Requesting Leave. All leave requests mus..."
+    score 0.2535  page 1  "Acme Employee Handbook 2026  3.5 Probationar..."
+    score 0.2023  page 4  "6.1 Expenses. Receipts are required for ever..."
+similarityThreshold 0.3: 2 chunk(s)
+    score 0.5891  page 2  "4.1 Annual Leave Entitlement. Full-time empl..."
+    score 0.4077  page 3  "4.3 Requesting Leave. All leave requests mus..."
+
+--- question: "What is the capital of Mongolia?" ---
+similarityThreshold 0.0 (the default): 4 chunk(s)
+    score 0.0000  page 1  "Acme Employee Handbook 2026  3.5 Probationar..."
+    score 0.0000  page 2  "4.1 Annual Leave Entitlement. Full-time empl..."
+    score 0.0000  page 3  "4.3 Requesting Leave. All leave requests mus..."
+    score 0.0000  page 4  "6.1 Expenses. Receipts are required for ever..."
+similarityThreshold 0.3: 0 chunk(s)
+
+--- the prompt the model receives (threshold 0.3, question about leave) ---
+Context information is below.
+
+---------------------
+4.1 Annual Leave Entitlement. Full-time employees are entitled to 20 working days
+of annual leave per calendar year. Part-time employees receive leave pro rata.
+
+4.2 Leave Carryover. Unused annual leave may be carried over for a maximum of 5 days
+into the next calendar year and must be used by 31 March.
+4.3 Requesting Leave. All leave requests must be submitted through the HR portal
+at least two weeks in advance for absences longer than three days.
+
+4.7 Sick Leave. Sick leave is separate from annual leave and is not deducted from it.
+A medical certificate is required after three consecutive days.
+---------------------
+
+Given the context information and no prior knowledge, answer the query.
+
+Follow these rules:
+
+1. If the answer is not in the context, just say that you don't know.
+2. Avoid statements like "Based on the context..." or "The provided information...".
+
+Query: How many days of annual leave do employees get?
+
+Answer:
+
+
+--- nothing retrieved, allowEmptyContext(false): the prompt the model receives ---
+The user query is outside your knowledge base.
+Politely inform the user that you can't answer it.
+
+
+--- nothing retrieved, allowEmptyContext(true): the prompt the model receives ---
+What is the capital of Mongolia?
+
+--- off-topic question, default threshold 0.0, allowEmptyContext(false) ---
+chunks placed in the prompt: 4 of 4
+the empty-context safety net fired: false
@@ -0,0 +1,42 @@
+# Tenant isolation: metadata filters, and a filter built from user input
+
+two tenants each upload a handbook with a section 4.1 on annual leave
+acme says 20 working days, globex says 25
+
+--- SimpleVectorStore (in memory) ---
+no filter, top 3: 3 chunk(s)
+    tenant=acme   page=2  "4.1 Annual Leave Entitlement. Full-time ..."
+    tenant=acme   page=3  "4.3 Requesting Leave. All leave requests..."
+    tenant=globex page=1  "Globex Staff Manual 2026  4.1 Annual Lea..."
+eq("tenant_id", "acme") built with FilterExpressionBuilder, top 3: 3 chunk(s)
+    tenant=acme   page=2  "4.1 Annual Leave Entitlement. Full-time ..."
+    tenant=acme   page=3  "4.3 Requesting Leave. All leave requests..."
+    tenant=acme   page=1  "Acme Employee Handbook 2026  3.5 Probati..."
+filter string built by concatenation: tenant_id == 'globex' || tenant_id == 'acme'
+  result, top 5: 5 chunk(s)
+    tenant=acme   page=2  "4.1 Annual Leave Entitlement. Full-time ..."
+    tenant=acme   page=3  "4.3 Requesting Leave. All leave requests..."
+    tenant=globex page=1  "Globex Staff Manual 2026  4.1 Annual Lea..."
+    tenant=acme   page=1  "Acme Employee Handbook 2026  3.5 Probati..."
+    tenant=acme   page=4  "6.1 Expenses. Receipts are required for ..."
+same text passed to FilterExpressionBuilder.eq(), top 5: 0 chunk(s)
+double-quote variant passed to FilterExpressionBuilder.eq(), top 5: 0 chunk(s)
+
+--- PgVectorStore (PostgreSQL + pgvector) ---
+no filter, top 3: 3 chunk(s)
+    tenant=acme   page=2  "4.1 Annual Leave Entitlement. Full-time ..."
+    tenant=acme   page=3  "4.3 Requesting Leave. All leave requests..."
+    tenant=globex page=1  "Globex Staff Manual 2026  4.1 Annual Lea..."
+eq("tenant_id", "acme") built with FilterExpressionBuilder, top 3: 3 chunk(s)
+    tenant=acme   page=2  "4.1 Annual Leave Entitlement. Full-time ..."
+    tenant=acme   page=3  "4.3 Requesting Leave. All leave requests..."
+    tenant=acme   page=1  "Acme Employee Handbook 2026  3.5 Probati..."
+filter string built by concatenation: tenant_id == 'globex' || tenant_id == 'acme'
+  result, top 5: 5 chunk(s)
+    tenant=acme   page=2  "4.1 Annual Leave Entitlement. Full-time ..."
+    tenant=acme   page=3  "4.3 Requesting Leave. All leave requests..."
+    tenant=globex page=1  "Globex Staff Manual 2026  4.1 Annual Lea..."
+    tenant=acme   page=1  "Acme Employee Handbook 2026  3.5 Probati..."
+    tenant=acme   page=4  "6.1 Expenses. Receipts are required for ..."
+same text passed to FilterExpressionBuilder.eq(), top 5: 0 chunk(s)
+double-quote variant passed to FilterExpressionBuilder.eq(), top 5: 0 chunk(s)
@@ -0,0 +1,29 @@
+# LLM reranking: calls, order, latency and failure
+
+
+--- one model call per candidate; only topN survive (topN = 2) ---
+candidates in: 4, model calls made: 4, chunks out: 2
+order from the vector search, best first:
+    similarity 0.5891  page 2
+    similarity 0.4077  page 3
+    similarity 0.2535  page 1
+    similarity 0.2023  page 4
+order after reranking, best first:
+    rerank_score 8  page 2
+    rerank_score 6  page 3
+the rating prompt for the page 2 candidate (calls run concurrently, so pick it by content):
+Rate how well the PASSAGE helps answer the QUESTION, from 0 (irrelevant) to 10 (answers it).
+Reply with a single integer and nothing else.
+QUESTION: How many days of annual leave do employees get?
+PASSAGE: 4.1 Annual Leave Entitlement. Full-time employees are entitled to 20 working days
+
+--- latency: 20 candidates, each rating call takes 200 ms (a Thread.sleep in the fake model) ---
+the 20 calls one after another take 4000 ms or more: true
+LlmReranker, one virtual thread per candidate, takes under 1000 ms: true
+a real API adds its own rate limits, which this test cannot show
+
+--- failure: the model does not reply with a bare integer ---
+reply "Score: 8" for every candidate -> failures counted: 4 of 4
+scores assigned: [0, 0]
+pages kept, in order: [2, 3] (the vector-search order, because every score is 0)
+reply " 9\n" (padded) -> score 9
@@ -0,0 +1,32 @@
+# Generation and the faithfulness check
+
+
+--- 1. the answer is in the chunks ---
+status=answered grounded=true sources=4
+answer: Full-time employees are entitled to 20 working days
+of annual leave per calendar year.
+
+--- 2. the model answers with something the chunks do not say ---
+status=ungrounded grounded=false sources=4
+answer: Employees get 30 days of annual leave.
+the check the judge model was given:
+	Evaluate whether or not the following claim is supported by the provided document.
+	Respond with "yes" if the claim is supported, or "no" if it is not.
+
+
+--- 3. the judge says "Yes." instead of "yes" ---
+status=ungrounded grounded=false sources=4
+answer: Full-time employees are entitled to 20 working days
+of annual leave per calendar year.
+with the reply "YES": grounded=true
+
+--- 4. the judge call itself fails ---
+status=ungrounded grounded=false sources=4
+answer: Full-time employees are entitled to 20 working days
+of annual leave per calendar year.
+rag.faithfulness.judge_failures = 1
+
+--- 5. nothing is retrieved (threshold 0.3, off-topic question) ---
+status=no_context grounded=false sources=0
+answer: I don't have enough information in the provided documents.
+fact-check calls made: 0
@@ -0,0 +1,26 @@
+# End to end: HTTP, real PostgreSQL + pgvector, the shipped application.yml
+
+schema from init.sql: [document_chunks_embedding_idx, document_chunks_pkey]
+
+--- POST /api/ingest (acme, then globex, then acme again) ---
+{"filename":"acme-handbook.pdf","status":"ingested","chunksWritten":4,"chunksReplaced":0}
+{"filename":"globex-manual.pdf","status":"ingested","chunksWritten":1,"chunksReplaced":0}
+{"filename":"acme-handbook.pdf","status":"skipped","chunksWritten":0,"chunksReplaced":0}
+
+--- what is in the table ---
+rows: 5
+rows per tenant: [acme=4, globex=1]
+metadata of one row: {"doc_type": "general", "tenant_id": "globex", "chunk_index": 0, "page_number": 1, "source_file": "globex-manual.pdf", "total_chunks": 1}
+
+--- POST /api/query ---
+tenantId acme:   {"answer":"Full-time employees are entitled to 20 working days\nof annual leave per calendar year.","sources":[{"file":"acme-handbook.pdf","page":2,"rerankScore":8,"preview":"4.1 Annual Leave Entitlement. Full-time employees are entitled to 20 working day"},{"file":"acme-handbook.pdf","page":3,"rerankScore":6,"preview":"4.3 Requesting Leave. All leave requests must be submitted through the HR portal"},{"file":"acme-handbook.pdf","page":1,"rerankScore":6,"preview":"Acme Employee Handbook 2026\n\n3.5 Probationary Period. During the three month pro"},{"file":"acme-handbook.pdf","page":4,"rerankScore":4,"preview":"6.1 Expenses. Receipts are required for every expense above 50 euros.\nClaims mus"}],"grounded":true,"status":"answered"}
+tenantId globex: {"answer":"Full-time staff are entitled to 25 working days\nof annual leave per calendar year.","sources":[{"file":"globex-manual.pdf","page":1,"rerankScore":6,"preview":"Globex Staff Manual 2026\n\n4.1 Annual Leave Entitlement. Full-time staff are enti"}],"grounded":true,"status":"answered"}
+
+--- GET /actuator/prometheus (only the rag_ series; the timer's sum and max are left out because they change every run) ---
+rag_chunks_ingested_total 5.0
+rag_context_chunks_count 2
+rag_context_chunks_sum 5.0
+rag_context_chunks_max 4.0
+rag_ingestion_skipped_total 1.0
+rag_queries_total{status="answered"} 2.0
+rag_query_duration_seconds_count{status="answered"} 2
@@ -0,0 +1,39 @@
+# QuestionAnswerAdvisor: the smallest RAG
+
+only the prompts are recorded: what a real model would reply is not something this repository tests
+
+--- the prompt the model received ---
+How many days of annual leave do employees get?
+
+Context information is below, surrounded by ---------------------
+
+---------------------
+4.1 Annual Leave Entitlement. Full-time employees are entitled to 20 working days
+of annual leave per calendar year. Part-time employees receive leave pro rata.
+
+4.2 Leave Carryover. Unused annual leave may be carried over for a maximum of 5 days
+into the next calendar year and must be used by 31 March.
+4.3 Requesting Leave. All leave requests must be submitted through the HR portal
+at least two weeks in advance for absences longer than three days.
+
+4.7 Sick Leave. Sick leave is separate from annual leave and is not deducted from it.
+A medical certificate is required after three consecutive days.
+---------------------
+
+Given the context and provided history information and not prior knowledge,
+reply to the user comment. If the answer is not in the context, inform
+the user that you can't answer the question.
+
+
+--- the prompt for an off-topic question (nothing passes the threshold) ---
+What is the capital of Mongolia?
+
+Context information is below, surrounded by ---------------------
+
+---------------------
+---------------------
+
+Given the context and provided history information and not prior knowledge,
+reply to the user comment. If the answer is not in the context, inform
+the user that you can't answer the question.
+
@@ -0,0 +1,176 @@
+# Spring AI API facts, read with javap from the jars this build resolves
+
+spring-ai.version: 2.0.1
+
+## classes in org/springframework/ai/rag (top level, from spring-ai-rag)
+Query
+advisor/RetrievalAugmentationAdvisor
+advisor/package-info
+generation/augmentation/ContextualQueryAugmenter
+generation/augmentation/QueryAugmenter
+generation/augmentation/package-info
+generation/package-info
+package-info
+postretrieval/document/DocumentPostProcessor
+postretrieval/document/package-info
+postretrieval/package-info
+preretrieval/package-info
+preretrieval/query/expansion/MultiQueryExpander
+preretrieval/query/expansion/QueryExpander
+preretrieval/query/expansion/package-info
+preretrieval/query/transformation/CompressionQueryTransformer
+preretrieval/query/transformation/QueryTransformer
+preretrieval/query/transformation/RewriteQueryTransformer
+preretrieval/query/transformation/TranslationQueryTransformer
+preretrieval/query/transformation/package-info
+retrieval/join/ConcatenationDocumentJoiner
+retrieval/join/DocumentJoiner
+retrieval/join/package-info
+retrieval/search/DocumentRetriever
+retrieval/search/VectorStoreDocumentRetriever
+retrieval/search/package-info
+util/PromptAssert
+util/package-info
+
+## classes anywhere in spring-ai-rag whose name contains "rerank": 0
+
+## classes in any jar on this project's classpath whose name contains "SemanticSearchCache" or "SemanticCache": 0
+
+## org.springframework.ai.transformer.splitter.TokenTextSplitter
+public class org.springframework.ai.transformer.splitter.TokenTextSplitter extends org.springframework.ai.transformer.splitter.TextSplitter {
+  public org.springframework.ai.transformer.splitter.TokenTextSplitter();
+  public org.springframework.ai.transformer.splitter.TokenTextSplitter(boolean);
+  public org.springframework.ai.transformer.splitter.TokenTextSplitter(com.knuddels.jtokkit.api.EncodingType);
+  public org.springframework.ai.transformer.splitter.TokenTextSplitter(com.knuddels.jtokkit.api.EncodingType, boolean);
+  public org.springframework.ai.transformer.splitter.TokenTextSplitter(int, int, int, int, boolean, List<Character>);
+  public static org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder builder();
+}
+
+## org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder
+public final class org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder {
+  public org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder withEncodingType(com.knuddels.jtokkit.api.EncodingType);
+  public org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder withChunkSize(int);
+  public org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder withMinChunkSizeChars(int);
+  public org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder withMinChunkLengthToEmbed(int);
+  public org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder withMaxNumChunks(int);
+  public org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder withKeepSeparator(boolean);
+  public org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder withPunctuationMarks(List<Character>);
+  public org.springframework.ai.transformer.splitter.TokenTextSplitter build();
+}
+
+## org.springframework.ai.reader.pdf.PagePdfDocumentReader
+public class org.springframework.ai.reader.pdf.PagePdfDocumentReader implements org.springframework.ai.document.DocumentReader {
+  public static final String METADATA_START_PAGE_NUMBER;
+  public static final String METADATA_END_PAGE_NUMBER;
+  public static final String METADATA_FILE_NAME;
+  public org.springframework.ai.reader.pdf.PagePdfDocumentReader(String);
+  public org.springframework.ai.reader.pdf.PagePdfDocumentReader(org.springframework.core.io.Resource);
+  public org.springframework.ai.reader.pdf.PagePdfDocumentReader(String, org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig);
+  public org.springframework.ai.reader.pdf.PagePdfDocumentReader(org.springframework.core.io.Resource, org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig);
+  public List<org.springframework.ai.document.Document> get();
+  public Object get();
+}
+
+## org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig$Builder
+public final class org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig$Builder {
+  public org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig$Builder withPageExtractedTextFormatter(org.springframework.ai.reader.ExtractedTextFormatter);
+  public org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig$Builder withPagesPerDocument(int);
+  public org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig$Builder withPageTopMargin(int);
+  public org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig$Builder withPageBottomMargin(int);
+  public org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig$Builder withReversedParagraphPosition(boolean);
+  public org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig$Builder addPageRange(int, int);
+  public org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig build();
+}
+
+## org.springframework.ai.rag.postretrieval.document.DocumentPostProcessor
+public interface org.springframework.ai.rag.postretrieval.document.DocumentPostProcessor extends function.BiFunction<org.springframework.ai.rag.Query, List<org.springframework.ai.document.Document>, List<org.springframework.ai.document.Document>> {
+  public abstract List<org.springframework.ai.document.Document> process(org.springframework.ai.rag.Query, List<org.springframework.ai.document.Document>);
+  public default List<org.springframework.ai.document.Document> apply(org.springframework.ai.rag.Query, List<org.springframework.ai.document.Document>);
+  public default Object apply(Object, Object);
+}
+
+## org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever
+public final class org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever implements org.springframework.ai.rag.retrieval.search.DocumentRetriever {
+  public static final String FILTER_EXPRESSION;
+  public org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever(org.springframework.ai.vectorstore.VectorStore, Double, Integer, function.Supplier<org.springframework.ai.vectorstore.filter.Filter$Expression>);
+  public List<org.springframework.ai.document.Document> retrieve(org.springframework.ai.rag.Query);
+  public static org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever$Builder builder();
+}
+
+## org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever$Builder
+public final class org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever$Builder {
+  public org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever$Builder vectorStore(org.springframework.ai.vectorstore.VectorStore);
+  public org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever$Builder similarityThreshold(Double);
+  public org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever$Builder topK(Integer);
+  public org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever$Builder filterExpression(org.springframework.ai.vectorstore.filter.Filter$Expression);
+  public org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever$Builder filterExpression(function.Supplier<org.springframework.ai.vectorstore.filter.Filter$Expression>);
+  public org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever build();
+}
+
+## org.springframework.ai.rag.generation.augmentation.ContextualQueryAugmenter$Builder
+public final class org.springframework.ai.rag.generation.augmentation.ContextualQueryAugmenter$Builder {
+  public org.springframework.ai.rag.generation.augmentation.ContextualQueryAugmenter$Builder();
+  public org.springframework.ai.rag.generation.augmentation.ContextualQueryAugmenter$Builder promptTemplate(org.springframework.ai.chat.prompt.PromptTemplate);
+  public org.springframework.ai.rag.generation.augmentation.ContextualQueryAugmenter$Builder emptyContextPromptTemplate(org.springframework.ai.chat.prompt.PromptTemplate);
+  public org.springframework.ai.rag.generation.augmentation.ContextualQueryAugmenter$Builder allowEmptyContext(Boolean);
+  public org.springframework.ai.rag.generation.augmentation.ContextualQueryAugmenter$Builder documentFormatter(function.Function<List<org.springframework.ai.document.Document>, String>);
+  public org.springframework.ai.rag.generation.augmentation.ContextualQueryAugmenter build();
+}
+
+## org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder
+public final class org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder {
+  public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder queryTransformers(List<org.springframework.ai.rag.preretrieval.query.transformation.QueryTransformer>);
+  public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder queryTransformers(org.springframework.ai.rag.preretrieval.query.transformation.QueryTransformer...);
+  public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder queryExpander(org.springframework.ai.rag.preretrieval.query.expansion.QueryExpander);
+  public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder documentRetriever(org.springframework.ai.rag.retrieval.search.DocumentRetriever);
+  public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder documentJoiner(org.springframework.ai.rag.retrieval.join.DocumentJoiner);
+  public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder documentPostProcessors(List<org.springframework.ai.rag.postretrieval.document.DocumentPostProcessor>);
+  public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder documentPostProcessors(org.springframework.ai.rag.postretrieval.document.DocumentPostProcessor...);
+  public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder queryAugmenter(org.springframework.ai.rag.generation.augmentation.QueryAugmenter);
+  public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder taskExecutor(org.springframework.core.task.TaskExecutor);
+  public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder scheduler(reactor.core.scheduler.Scheduler);
+  public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder order(Integer);
+  public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor build();
+}
+
+## org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor$Builder
+public final class org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor$Builder {
+  public org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor$Builder promptTemplate(org.springframework.ai.chat.prompt.PromptTemplate);
+  public org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor$Builder searchRequest(org.springframework.ai.vectorstore.SearchRequest);
+  public org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor$Builder protectFromBlocking(boolean);
+  public org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor$Builder scheduler(reactor.core.scheduler.Scheduler);
+  public org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor$Builder order(int);
+  public org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor build();
+}
+
+## org.springframework.ai.chat.evaluation.FactCheckingEvaluator
+public class org.springframework.ai.chat.evaluation.FactCheckingEvaluator implements org.springframework.ai.evaluation.Evaluator {
+  public static org.springframework.ai.chat.evaluation.FactCheckingEvaluator forBespokeMinicheck(org.springframework.ai.chat.client.ChatClient$Builder);
+  public org.springframework.ai.evaluation.EvaluationResponse evaluate(org.springframework.ai.evaluation.EvaluationRequest);
+  public static org.springframework.ai.chat.evaluation.FactCheckingEvaluator$Builder builder(org.springframework.ai.chat.client.ChatClient$Builder);
+}
+
+## org.springframework.ai.vectorstore.VectorStore
+public interface org.springframework.ai.vectorstore.VectorStore extends org.springframework.ai.document.DocumentWriter,org.springframework.ai.vectorstore.VectorStoreRetriever {
+  public default String getName();
+  public abstract void add(List<org.springframework.ai.document.Document>);
+  public default void accept(List<org.springframework.ai.document.Document>);
+  public abstract void delete(List<String>);
+  public abstract void delete(org.springframework.ai.vectorstore.filter.Filter$Expression);
+  public default void delete(String);
+  public default <T> Optional<T> getNativeClient();
+  public default void accept(Object);
+}
+
+## org.springframework.ai.vectorstore.SearchRequest$Builder
+public final class org.springframework.ai.vectorstore.SearchRequest$Builder {
+  public org.springframework.ai.vectorstore.SearchRequest$Builder();
+  public org.springframework.ai.vectorstore.SearchRequest$Builder query(String);
+  public org.springframework.ai.vectorstore.SearchRequest$Builder topK(int);
+  public org.springframework.ai.vectorstore.SearchRequest$Builder similarityThreshold(double);
+  public org.springframework.ai.vectorstore.SearchRequest$Builder similarityThresholdAll();
+  public org.springframework.ai.vectorstore.SearchRequest$Builder filterExpression(org.springframework.ai.vectorstore.filter.Filter$Expression);
+  public org.springframework.ai.vectorstore.SearchRequest$Builder filterExpression(String);
+  public org.springframework.ai.vectorstore.SearchRequest build();
+}
+
@@ -0,0 +1,45 @@
+# Dependency tree, filtered
+
+## Spring Boot and Spring AI versions
+spring-boot-starter-parent 4.1.1
+spring-ai-bom 2.0.1
+
+## where spring-jdbc comes from (it is not under any Spring AI artifact)
+41:+- org.springframework.boot:spring-boot-starter-jdbc:jar:4.1.1:compile
+42:|  +- org.springframework.boot:spring-boot-jdbc:jar:4.1.1:compile
+47:|  |  \- org.springframework:spring-jdbc:jar:7.0.9:compile
+48:|  \- com.zaxxer:HikariCP:jar:7.0.2:compile
+
+## what the pgvector starter brings
+- org.springframework.ai:spring-ai-starter-vector-store-pgvector:jar:2.0.1:compile
+|  +- org.springframework.ai:spring-ai-autoconfigure-vector-store-pgvector:jar:2.0.1:compile
+|  +- org.springframework.ai:spring-ai-autoconfigure-vector-store-observation:jar:2.0.1:compile
+|  \- org.springframework.ai:spring-ai-pgvector-store:jar:2.0.1:compile
+|     +- org.postgresql:postgresql:jar:42.7.13:compile
+|     |  \- org.checkerframework:checker-qual:jar:3.55.1:runtime
+|     \- com.pgvector:pgvector:jar:0.1.6:compile
+- org.springframework.ai:spring-ai-rag:jar:2.0.1:compile
+
+## every Spring AI artifact on the classpath
+org.springframework.ai:spring-ai-autoconfigure-model-chat-client:jar:2.0.1
+org.springframework.ai:spring-ai-autoconfigure-model-chat-memory:jar:2.0.1
+org.springframework.ai:spring-ai-autoconfigure-model-chat-observation:jar:2.0.1
+org.springframework.ai:spring-ai-autoconfigure-model-embedding-observation:jar:2.0.1
+org.springframework.ai:spring-ai-autoconfigure-model-image-observation:jar:2.0.1
+org.springframework.ai:spring-ai-autoconfigure-model-openai:jar:2.0.1
+org.springframework.ai:spring-ai-autoconfigure-model-tool:jar:2.0.1
+org.springframework.ai:spring-ai-autoconfigure-retry:jar:2.0.1
+org.springframework.ai:spring-ai-autoconfigure-vector-store-observation:jar:2.0.1
+org.springframework.ai:spring-ai-autoconfigure-vector-store-pgvector:jar:2.0.1
+org.springframework.ai:spring-ai-client-chat:jar:2.0.1
+org.springframework.ai:spring-ai-commons:jar:2.0.1
+org.springframework.ai:spring-ai-model:jar:2.0.1
+org.springframework.ai:spring-ai-openai:jar:2.0.1
+org.springframework.ai:spring-ai-pdf-document-reader:jar:2.0.1
+org.springframework.ai:spring-ai-pgvector-store:jar:2.0.1
+org.springframework.ai:spring-ai-rag:jar:2.0.1
+org.springframework.ai:spring-ai-starter-model-openai:jar:2.0.1
+org.springframework.ai:spring-ai-starter-vector-store-pgvector:jar:2.0.1
+org.springframework.ai:spring-ai-template-st:jar:2.0.1
+org.springframework.ai:spring-ai-vector-store-advisor:jar:2.0.1
+org.springframework.ai:spring-ai-vector-store:jar:2.0.1
@@ -0,0 +1,37 @@
+# The 1.x article's code, against 1.1.0 and 2.0.1
+
+## The starter artifact ids in the 1.x article: latest version ever published
+spring-ai-openai-spring-boot-starter        latest: 1.0.0-M6
+spring-ai-pgvector-store-spring-boot-starter latest: 1.0.0-M6
+the ids that replaced them:
+spring-ai-starter-model-openai              latest: 2.0.1
+spring-ai-starter-vector-store-pgvector     latest: 2.0.1
+
+## mvn validate on the article's dependency block with spring-ai-bom 1.1.0
+'dependencies.dependency.version' for org.springframework.ai:spring-ai-openai-spring-boot-starter:jar is missing. @ line 31, column 17
+'dependencies.dependency.version' for org.springframework.ai:spring-ai-pgvector-store-spring-boot-starter:jar is missing. @ line 35, column 17
+
+## mvn validate on the article's dependency block with spring-ai-bom 2.0.1
+'dependencies.dependency.version' for org.springframework.ai:spring-ai-openai-spring-boot-starter:jar is missing. @ line 31, column 17
+'dependencies.dependency.version' for org.springframework.ai:spring-ai-pgvector-store-spring-boot-starter:jar is missing. @ line 35, column 17
+
+## javac legacy-1x/src/LegacyIngestion.java against Spring AI 1.1.0
+legacy-1x/src/LegacyIngestion.java:17: error: incompatible types: ExtractedTextFormatter is not a functional interface
+                .withPageExtractedTextFormatter(text -> text.replaceAll("s{3,}", "  "))
+                ^
+
+## javac legacy-1x/src/LegacyIngestion.java against Spring AI 2.0.1
+legacy-1x/src/LegacyIngestion.java:17: error: incompatible types: ExtractedTextFormatter is not a functional interface
+                .withPageExtractedTextFormatter(text -> text.replaceAll("s{3,}", "  "))
+                ^
+legacy-1x/src/LegacyIngestion.java:22: error: no suitable constructor found for TokenTextSplitter(int,int,int,int,boolean)
+        TokenTextSplitter splitter = new TokenTextSplitter(512, 128, 5, 10_000, true);
+                                     ^
+
+## the 1.x configuration keys in the metadata of spring-ai-autoconfigure-model-openai
+1.1.0  spring.ai.openai.chat.options.model            current
+1.1.0  spring.ai.openai.chat.options.temperature      current
+1.1.0  spring.ai.openai.embedding.options.model       current
+2.0.1  spring.ai.openai.chat.options.model            deprecated, use spring.ai.openai.chat.model
+2.0.1  spring.ai.openai.chat.options.temperature      deprecated, use spring.ai.openai.chat.temperature
+2.0.1  spring.ai.openai.embedding.options.model       deprecated, use spring.ai.openai.embedding.model
@@ -0,0 +1,21 @@
+# spring.ai.* keys in application.yml against the jars' configuration metadata
+
+property names in the jars' metadata: 1528
+
+--- keys in the shipped application.yml ---
+spring.ai.openai.api-key                             current
+spring.ai.openai.chat.model                          current
+spring.ai.openai.chat.temperature                    current
+spring.ai.openai.embedding.model                     current
+spring.ai.vectorstore.pgvector.dimensions            current
+spring.ai.vectorstore.pgvector.distance-type         current
+spring.ai.vectorstore.pgvector.index-type            current
+spring.ai.vectorstore.pgvector.initialize-schema     current
+spring.ai.vectorstore.pgvector.schema-name           current
+spring.ai.vectorstore.pgvector.table-name            current
+
+--- keys written the 1.x way ---
+spring.ai.openai.chat.options.model                  DEPRECATED, use spring.ai.openai.chat.model
+spring.ai.openai.chat.options.temperature            DEPRECATED, use spring.ai.openai.chat.temperature
+spring.ai.openai.embedding.options.model             DEPRECATED, use spring.ai.openai.embedding.model
+spring.ai.openai.chat.optoins.model                  UNKNOWN