commit 1d4625a1c28989593137327514978fa932b9a152 Author: Claude Date: Mon Sep 21 18:52:48 2026 +0000 Add rag module: Spring AI 2.0 RAG with pgvector, chunking, reranking and a faithfulness check Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01B38FGKKam5SCGgwgduVAh3 diff --git a/README.md b/README.md new file mode 100644 index 0000000..f7d104b --- /dev/null +++ b/README.md @@ -0,0 +1,9 @@ +# spring-ai + +Runnable companion code for the Spring AI articles on [ankurm.com](https://ankurm.com). One directory per module; each module is one commit and carries its own README, tests and captured output. + +| Module | What it is | Article | +|---|---|---| +| [`rag/`](rag) | Ingest PDFs, chunk, retrieve from pgvector, rerank, answer, check the answer. Spring Boot 4.1.1, Spring AI 2.0.1, Java 25. | [Production-grade RAG with Spring AI](https://ankurm.com/production-rag-spring-ai-java/) and [the complete example](https://ankurm.com/spring-ai-rag-complete-example/) | + +Upgrading from Spring AI 1.x: [migration guide](https://ankurm.com/spring-ai-1-to-2-migration-guide/). diff --git a/rag/.gitignore b/rag/.gitignore new file mode 100644 index 0000000..1ebf457 --- /dev/null +++ b/rag/.gitignore @@ -0,0 +1,3 @@ +target/ +*.iml +.idea/ diff --git a/rag/README.md b/rag/README.md new file mode 100644 index 0000000..72e71b0 --- /dev/null +++ b/rag/README.md @@ -0,0 +1,118 @@ +# Spring AI 2.0 RAG: ingest PDFs, retrieve, rerank, answer, check + +A retrieval-augmented-generation service on **Spring Boot 4.1.1**, **Spring AI 2.0.1**, **Java 25** and PostgreSQL with +pgvector. It is the companion code for the two articles on ankurm.com +([the explanation](https://ankurm.com/production-rag-spring-ai-java/) and +[the code tour](https://ankurm.com/spring-ai-rag-complete-example/)), and it replaces the 1.x code those articles used to carry. +For upgrading an existing Spring AI 1.x project, see the +[migration guide](https://ankurm.com/spring-ai-1-to-2-migration-guide/). + +**Every number in the articles and in `docs/` comes from a file in [`docs/output/`](docs/output), and the tests that write those files +also assert them.** The one big caveat is what is *not* real: the chat and embedding models in the tests are scripted stand-ins (next section). + +## What is real and what is scripted + +| Piece | In the tests | In the shipped application | +|---|---|---| +| PDF reading, chunking, `PgVectorStore`, PostgreSQL + pgvector, metadata filters | real | real | +| `RetrievalAugmentationAdvisor`, `QuestionAnswerAdvisor`, `FactCheckingEvaluator`, `ChatClient`, HTTP layer, Micrometer | real | real | +| Embedding model | [`HashingEmbeddingModel`](src/test/java/com/ankurm/rag/support/HashingEmbeddingModel.java): words hashed into a vector | OpenAI `text-embedding-3-small` | +| Chat model (answers, reranker ratings, judge verdicts) | [`FakeChatModel`](src/test/java/com/ankurm/rag/support/FakeChatModel.java): scripted from the prompt | OpenAI `gpt-4o` | + +So the tests show what Spring AI sends to the model and how this code handles each kind of reply. They do **not** show how a real model +answers, what similarity scores or thresholds suit a real embedding model, whether reranking improves answers, or how well a real judge catches +an invented claim. There is no relevance benchmark here. The application itself was **never run against the OpenAI API** in authoring. + +## Versions + +| | | +|---|---| +| Java | 25 | +| Spring Boot | 4.1.1 | +| Spring AI | 2.0.1 (`spring-ai-bom`) | +| PostgreSQL / pgvector | 16 / **0.6.0** used for every transcript (Debian package). `docker-compose.yml` uses `pgvector/pgvector:pg16`, which was **not run** here | + +## Run it + +```bash +# 1. PostgreSQL with pgvector, either: +docker compose up -d # localhost:5432, runs init.sql +# or, with no Docker (Debian/Ubuntu, needs postgresql-16 and postgresql-16-pgvector packages): +scripts/pg-up.sh # 127.0.0.1:5439; prints the RAG_PG_URL to export + +# 2. Regenerate every transcript (no API key needed) +export RAG_PG_URL=jdbc:postgresql://localhost:5432/ragdb # or the one pg-up.sh printed +scripts/run-all.sh + +# 3. Run the application against OpenAI +export OPENAI_API_KEY=sk-... +mvn spring-boot:run +``` + +| Endpoint | What it does | +|---|---| +| `POST /api/ingest` (multipart: `file`, `tenantId`, `docType`) | reads, chunks, embeds and stores a PDF; returns `ingested`, `updated` or `skipped` | +| `POST /api/query` (`{"question": "...", "tenantId": "acme"}`) | answer, the chunks used with their rerank scores, `grounded`, and `status` (`answered`, `ungrounded`, `no_context`) | +| `GET /actuator/prometheus` | `rag_*` metrics | + +```bash +curl -F file=@handbook.pdf -F tenantId=acme localhost:8080/api/ingest +curl -H 'Content-Type: application/json' -d '{"question":"How many days of annual leave?","tenantId":"acme"}' localhost:8080/api/query +``` + +## Chapters + +| | | +|---|---| +| [1. The shape of a RAG pipeline](docs/01-the-shape-of-a-rag-pipeline.md) | stages, the smallest thing that works, what is not tested | +| [2. Chunking](docs/02-chunking.md) | `TokenTextSplitter` measured, a recursive chunker with overlap, a semantic chunker | +| [3. Ingestion](docs/03-ingestion.md) | PDF pages, whitespace, metadata, uploading twice | +| [4. Retrieval and reranking](docs/04-retrieval-and-reranking.md) | threshold, empty context, tenant filter and injection, LLM reranking | +| [5. Generation and the faithfulness check](docs/05-generation-and-the-faithfulness-check.md) | the judge, its four outcomes | +| [6. Observability and a production checklist](docs/06-observability-and-production-checklist.md) | metrics, alerts, checklist | + +## Captured output + +Written by `scripts/run-all.sh`. Files 01-11 and 15 are written by the tests (the same test asserts the numbers); 12-14 by the scripts. + +| File | Shows | Written by | +|---|---|---| +| [01-pipeline-beans.txt](docs/output/01-pipeline-beans.txt) | the beans of the running application | `EndToEndTest` | +| [02-token-text-splitter.txt](docs/output/02-token-text-splitter.txt) | what `TokenTextSplitter` does to a fixed document | `ChunkingTest` | +| [03-recursive-and-semantic-chunkers.txt](docs/output/03-recursive-and-semantic-chunkers.txt) | the two chunkers of ours | `ChunkingTest` | +| [04-pdf-pages-and-metadata.txt](docs/output/04-pdf-pages-and-metadata.txt) | a PDF page as read, and tidied | `IngestionTest` | +| [05-ingestion-idempotency.txt](docs/output/05-ingestion-idempotency.txt) | upload twice, edit, re-export, naive `add()` (PostgreSQL) | `IngestionTest` | +| [06-retrieval-threshold-and-prompt.txt](docs/output/06-retrieval-threshold-and-prompt.txt) | threshold, augmented prompt, empty context | `RetrievalTest` | +| [07-tenant-filter-and-injection.txt](docs/output/07-tenant-filter-and-injection.txt) | tenant filter, filter-string injection, both stores | `TenantFilterTest` | +| [08-reranking.txt](docs/output/08-reranking.txt) | reranker calls, latency, parse failure | `RerankTest` | +| [09-faithfulness-check.txt](docs/output/09-faithfulness-check.txt) | the judge and the four outcomes | `FaithfulnessTest` | +| [10-end-to-end.txt](docs/output/10-end-to-end.txt) | HTTP ingest and query, rows in PostgreSQL, metrics | `EndToEndTest` | +| [11-question-answer-advisor.txt](docs/output/11-question-answer-advisor.txt) | the smallest RAG and its prompts | `SimpleAdvisorTest` | +| [12-api-facts.txt](docs/output/12-api-facts.txt) | `javap` of the Spring AI classes used, and a count of any `SemanticSearchCache` class on the classpath | `scripts/capture-javap.sh` | +| [13-dependencies.txt](docs/output/13-dependencies.txt) | which artifact brings what | `scripts/capture-dependencies.sh` | +| [14-legacy-1x.txt](docs/output/14-legacy-1x.txt) | the 1.x article's code and configuration keys against 1.1.0 and 2.0.1 | `scripts/capture-legacy-compile.sh` | +| [15-config-keys.txt](docs/output/15-config-keys.txt) | every `spring.ai.*` key against the jars' metadata | `ConfigKeysTest` | + +## Layout + +``` +init.sql the schema (rag.document_chunks, HNSW cosine index); the application never creates it +docker-compose.yml PostgreSQL 16 + pgvector +src/main/java/.../ + ingest/ IngestionService, IngestionTracker + chunk/ RecursiveChunker, SemanticChunker (TokenTextSplitter is Spring AI's) + query/ LlmReranker, RagQueryService + config/ RagConfig (the wiring), RagProperties + web/ RagController +src/test/java/.../ one test class per chapter topic; support/ holds the scripted models and the sample PDFs +legacy-1x/ the 1.x article's dependency block and ingestion calls, kept only to show they do not build +scripts/ run-all.sh, pg-up.sh, capture-*.sh +``` + +## Known limits + +- The ingestion tracker is in memory (chapter 3). +- The reranker parses a bare integer and degrades silently when the model does not reply with one (chapter 4). +- Filtered queries were not tested at a size where pgvector's approximate index matters (chapter 4). +- The application has no authentication; `tenantId` in the request body is trusted, which is a demonstration, not a design. +- Not run against a real model, real embeddings, or the `pgvector/pgvector` Docker image. diff --git a/rag/docker-compose.yml b/rag/docker-compose.yml new file mode 100644 index 0000000..5709ffe --- /dev/null +++ b/rag/docker-compose.yml @@ -0,0 +1,21 @@ +# PostgreSQL 16 with the pgvector extension. init.sql runs once, on the first start of an empty volume. +services: + postgres: + image: pgvector/pgvector:pg16 + environment: + POSTGRES_DB: ragdb + POSTGRES_USER: raguser + POSTGRES_PASSWORD: ragpass + ports: + - "5432:5432" + volumes: + - ./init.sql:/docker-entrypoint-initdb.d/init.sql:ro + - pgdata:/var/lib/postgresql/data + healthcheck: + test: ["CMD-SHELL", "pg_isready -U raguser -d ragdb"] + interval: 5s + timeout: 3s + retries: 10 + +volumes: + pgdata: diff --git a/rag/docs/01-the-shape-of-a-rag-pipeline.md b/rag/docs/01-the-shape-of-a-rag-pipeline.md new file mode 100644 index 0000000..071c443 --- /dev/null +++ b/rag/docs/01-the-shape-of-a-rag-pipeline.md @@ -0,0 +1,63 @@ +# 1. The shape of a RAG pipeline + +[Index](../README.md) · next: [2. Chunking](02-chunking.md) + +A language model only knows what it was trained on and what is in the prompt. Retrieval-augmented generation +(RAG) puts the right few paragraphs of *your* documents into the prompt, so the model can answer from them. +Everything else in this repository is a decision about which paragraphs, how they get there, and what to do +when the answer is not in them. + +## Two phases, six stages + +``` + ingest (once per file) query (once per question) + ────────────────────── ───────────────────────── + PDF ─► read pages ─► chunk ─► embed ─► store question ─► embed ─► search ─► rerank ─► prompt ─► model ─► check + IngestionService VectorStore └────── RetrievalAugmentationAdvisor ──────┘ RagQueryService +``` + +| Stage | Class here | Spring AI part it stands on | +|---|---|---| +| read | [`IngestionService`](../src/main/java/com/ankurm/rag/ingest/IngestionService.java) | `PagePdfDocumentReader` | +| chunk | `DocumentTransformer chunker` in [`RagConfig`](../src/main/java/com/ankurm/rag/config/RagConfig.java) | `TokenTextSplitter` (or [`RecursiveChunker`](../src/main/java/com/ankurm/rag/chunk/RecursiveChunker.java), [`SemanticChunker`](../src/main/java/com/ankurm/rag/chunk/SemanticChunker.java)) | +| embed + store | `VectorStore` bean | `PgVectorStore`, built by the pgvector starter | +| search | `retrievalAdvisor` bean | `VectorStoreDocumentRetriever` | +| rerank | [`LlmReranker`](../src/main/java/com/ankurm/rag/query/LlmReranker.java) | the `DocumentPostProcessor` hook (Spring AI ships no reranker) | +| prompt | `retrievalAdvisor` bean | `ContextualQueryAugmenter` | +| model + check | [`RagQueryService`](../src/main/java/com/ankurm/rag/query/RagQueryService.java) | `ChatClient`, `FactCheckingEvaluator` | + +The beans as the application actually wires them are in [output 01](output/01-pipeline-beans.txt). + +## The smallest thing that works + +One advisor, no other class of ours: + +```java +ChatClient.builder(chatModel) + .defaultAdvisors(QuestionAnswerAdvisor.builder(vectorStore) + .searchRequest(SearchRequest.builder().topK(2).similarityThreshold(0.3).build()) + .build()) + .build(); +``` + +That is [`SimpleAdvisorTest`](../src/test/java/com/ankurm/rag/SimpleAdvisorTest.java). [Output 11](output/11-question-answer-advisor.txt) +shows the two prompts it produces. For a question the store can answer, the chunks are pasted between two lines of +dashes. For a question nothing matches, the prompt still has the same template with an **empty** block between the +dashes, and the model is left to follow the template's last sentence ("if the answer is not in the context, inform +the user that you can't answer"). Nothing in code decides that nothing was found. That is the first thing the +larger pipeline adds: see [chapter 4](04-retrieval-and-reranking.md). + +## What this repository does not test + +The chat model and the embedding model are scripted stand-ins +([`FakeChatModel`](../src/test/java/com/ankurm/rag/support/FakeChatModel.java), +[`HashingEmbeddingModel`](../src/test/java/com/ankurm/rag/support/HashingEmbeddingModel.java)). That is deliberate: +anyone can run every test with no API key, and what is asserted is what Spring AI *sends* and how this code +*handles* what comes back. It also means **nothing here shows how a real model answers, how a real embedding +model scores similarity, whether reranking improves answers, or how good a real judge is at spotting an +invented claim.** Similarity scores and thresholds in the outputs belong to the hashing model; do not carry them over. + +The vector store is real: every test that touches persistence runs against PostgreSQL with pgvector +(`scripts/pg-up.sh` or `docker-compose.yml`), and the end-to-end test starts the whole application over HTTP. + +Next: [2. Chunking](02-chunking.md) diff --git a/rag/docs/02-chunking.md b/rag/docs/02-chunking.md new file mode 100644 index 0000000..ac16dd5 --- /dev/null +++ b/rag/docs/02-chunking.md @@ -0,0 +1,59 @@ +# 2. Chunking + +prev: [1. The shape of a RAG pipeline](01-the-shape-of-a-rag-pipeline.md) · [Index](../README.md) · next: [3. Ingestion](03-ingestion.md) + +A chunk is the unit that gets embedded, stored, retrieved and pasted into the prompt. Cut too big and the +prompt fills with text unrelated to the question; cut too small and an answer is split across two chunks. + +This chapter measures what the chunkers do to one fixed document, not which one retrieves best. **There is no +retrieval-quality benchmark in this repository**, so nothing here says which strategy is best for your +documents. The fixture is forty numbered sentences, 4,268 characters, 880 `cl100k_base` tokens. + +## `TokenTextSplitter` in Spring AI 2.0.1 + +Transcript: [output 02](output/02-token-text-splitter.txt). Test: [`ChunkingTest`](../src/test/java/com/ankurm/rag/ChunkingTest.java). + +- **The defaults are large.** `new TokenTextSplitter()` (800-token chunks) turned the 880-token fixture into + two chunks of 792 and 88 tokens. A short handbook page becomes one chunk. +- **It prefers sentence ends.** With `withChunkSize(100)` the result was ten chunks of 88 tokens, every one ending + on a full stop; it did not cut at exactly 100 tokens. +- **It has no overlap.** Zero of nine neighbouring boundaries repeated any text. A sentence that straddles a cut + belongs to one chunk only. +- **`withMinChunkSizeChars` did nothing on this fixture.** 350 and 50 produced identical ten-chunk results. Reading + the bytecode of the splitter: when a chunk exceeds the token limit it is cut back to its last sentence-ending + mark, but only if that mark is more than `minChunkSizeChars` characters into the chunk. On text with a full stop + every ~100 characters the mark is always far enough in, so neither value matters. Do not read this as "the + setting is useless", only as "this document does not exercise it". +- **The constructor changed.** The 1.x five-argument constructor `new TokenTextSplitter(512, 128, 5, 10_000, true)` + does not exist in 2.0.1; use the builder. See [output 14](output/14-legacy-1x.txt) and [output 12](output/12-api-facts.txt). + +## `RecursiveChunker` (ours) + +Transcript: [output 03](output/03-recursive-and-semantic-chunkers.txt). Source: [`RecursiveChunker`](../src/main/java/com/ankurm/rag/chunk/RecursiveChunker.java). + +Cuts at the coarsest boundary that fits (blank line, line break, sentence end, space, and only then mid-word), +and repeats the last `overlapChars` characters at the start of the next chunk. With `(400, 80)` the fixture became +14 chunks, longest 395 characters, and all 13 boundaries carried overlap. Sizes are characters, not tokens. A +900-character string with no separators is cut hard at `[400, 400, 100]`. + +The overlap starts at a word boundary, so a chunk often *opens mid-sentence* ("for item 3 and states th..."). That is +the price of overlap; whether it helps retrieval was not measured. + +## `SemanticChunker` (ours) + +Same transcript. Source: [`SemanticChunker`](../src/main/java/com/ankurm/rag/chunk/SemanticChunker.java). + +Embeds every sentence in **one batched call** and starts a new chunk where the cosine distance between neighbours exceeds a +threshold. On twelve sentences about three topics it produced three chunks, one per topic, and sent 12 texts to the +embedding model. That result is from the hashing model, which is good at exactly this (topics share words). With a +real embedding model the right threshold is different and must be found on your own text; the value 0.9 has no meaning +outside this test. It costs an embedding call per sentence at ingestion time. + +## Choosing + +Start with `TokenTextSplitter` and a size you set on purpose. Move to the recursive chunker when a fact is being cut +in half at the boundary and you can see it in real retrievals. Consider semantic chunking only when documents mix +topics without headings and you can afford the extra embedding calls. Whichever you use, change it by replacing the +one `DocumentTransformer` bean; nothing else in the pipeline notices. + +prev: [1](01-the-shape-of-a-rag-pipeline.md) · next: [3. Ingestion](03-ingestion.md) diff --git a/rag/docs/03-ingestion.md b/rag/docs/03-ingestion.md new file mode 100644 index 0000000..d0cbd00 --- /dev/null +++ b/rag/docs/03-ingestion.md @@ -0,0 +1,58 @@ +# 3. Ingestion + +prev: [2. Chunking](02-chunking.md) · [Index](../README.md) · next: [4. Retrieval and reranking](04-retrieval-and-reranking.md) + +Source: [`IngestionService`](../src/main/java/com/ankurm/rag/ingest/IngestionService.java), +[`IngestionTracker`](../src/main/java/com/ankurm/rag/ingest/IngestionTracker.java). +Test: [`IngestionTest`](../src/test/java/com/ankurm/rag/IngestionTest.java). + +## What a PDF page becomes + +`PagePdfDocumentReader` with `withPagesPerDocument(1)` returns one `Document` per page, and the only metadata it adds +is `page_number` ([output 04](output/04-pdf-pages-and-metadata.txt)). That is the number a citation needs, so keep one +page per document until you have chosen a chunker; chunks inherit their page's metadata. + +The text is padded. In the sample PDF, page 2 came back as 862 characters with a run of 133 spaces at the end of a +line, and 304 characters after `IngestionService.tidy` collapsed the padding. Padding costs tokens and changes what gets +embedded. **The sample PDFs are generated by PDFBox inside this repository**, so the exact padding is an artefact of that +generator and the reader; real PDFs pad differently. Look at what your own files produce before assuming it matches. + +Every chunk also gets `source_file`, `source_hash` and whatever the caller passes (the HTTP endpoint adds `tenant_id` +and `doc_type`). Chunks written by `TokenTextSplitter` additionally carry `parent_document_id`, `chunk_index` +and `total_chunks`. One stored row, as the end-to-end test read it back from PostgreSQL: + +``` +{"doc_type": "general", "tenant_id": "globex", "chunk_index": 0, "page_number": 1, "source_file": "globex-manual.pdf", "total_chunks": 1} +``` + +([output 10](output/10-end-to-end.txt); `source_hash` and `parent_document_id` were left out of that query.) + +## Uploading the same file twice + +[Output 05](output/05-ingestion-idempotency.txt), on real PostgreSQL, counting rows with `select count(*)`: + +| step | what happened | rows | texts embedded | +|---|---|---|---| +| first upload | `ingested`, 4 chunks | 4 | 4 | +| same bytes again | `skipped` | 4 | 4 (nothing spent) | +| page 2 edited | `updated`, 4 old chunks deleted first | 4 | 8 | +| same text exported again | `updated` | 4 | 12 | +| *naive:* `vectorStore.add()` on every upload, twice more | rows tripled | 12 | – | + +The naive version is what many tutorials show: each upload writes new chunk ids, so the same page is stored three times +and retrieval returns duplicates that crowd out other passages. The service instead remembers, per file name, the SHA-256 of the +bytes and the ids of the chunks it wrote, skips identical bytes, and on a change deletes the old chunk **by id** before adding. +After the edit, zero rows still said "20 working days". + +The last row is a limit, not a feature: the hash is over the *file's bytes*. Two PDFs with identical text but different bytes +(the PDFBox generator in the tests produces different bytes on every run for the same text; I did not check which field differs) count as a change and are re-embedded. Hashing the extracted +text instead would avoid that, at the cost of reading the file first. + +## What the tracker does not do + +It lives in memory. A restart forgets every file, and the next upload of each one is re-embedded and its old chunks +are **not** deleted, because the tracker no longer knows their ids. Two fixes: persist the tracker in a table, or store +`source_file` in metadata (already done) and delete by filter, `vectorStore.delete(new FilterExpressionBuilder().eq("source_file", name).build())`. +The second is not exercised by any test here. + +Next: [4. Retrieval and reranking](04-retrieval-and-reranking.md) diff --git a/rag/docs/04-retrieval-and-reranking.md b/rag/docs/04-retrieval-and-reranking.md new file mode 100644 index 0000000..e56b6ab --- /dev/null +++ b/rag/docs/04-retrieval-and-reranking.md @@ -0,0 +1,65 @@ +# 4. Retrieval and reranking + +prev: [3. Ingestion](03-ingestion.md) · [Index](../README.md) · next: [5. Generation and the faithfulness check](05-generation-and-the-faithfulness-check.md) + +Tests: [`RetrievalTest`](../src/test/java/com/ankurm/rag/RetrievalTest.java), +[`TenantFilterTest`](../src/test/java/com/ankurm/rag/TenantFilterTest.java), +[`RerankTest`](../src/test/java/com/ankurm/rag/RerankTest.java). Wiring: [`RagConfig`](../src/main/java/com/ankurm/rag/config/RagConfig.java). + +**All similarity scores in this chapter come from the hashing embedding model.** They show how the components behave, not what a +real model would score. + +## Top-k, and the threshold that is not there by default + +`VectorStoreDocumentRetriever` returns the `topK` nearest chunks. Its default similarity threshold accepts everything, and +[output 06](output/06-retrieval-threshold-and-prompt.txt) shows what follows. For a question about leave, a threshold of 0.0 returned all four +chunks (scores 0.59, 0.41, 0.25, 0.20); 0.3 returned two. For an off-topic question ("capital of Mongolia") the default still +returned all four chunks, each with score 0.0000, and all four went into the prompt: **the "nothing found" safety net never +fires while the threshold accepts everything.** With 0.3 nothing was retrieved. + +Which threshold is right depends entirely on the embedding model and cannot be copied from here. Find yours by asking questions +you know the corpus cannot answer and looking at the top scores. + +## The empty-context path + +`ContextualQueryAugmenter` builds the prompt. With `allowEmptyContext(false)` (what `RagConfig` sets) and no chunks, the model +receives only "The user query is outside your knowledge base. Politely inform the user that you can't answer it." +With `allowEmptyContext(true)` the model receives the bare question and answers from whatever it was trained on, which for a +document-QA service is the failure you built RAG to avoid. Both prompts are in output 06. +`QuestionAnswerAdvisor` has no such switch: [output 11](output/11-question-answer-advisor.txt) shows it sending an empty context block instead. + +## Tenant isolation + +Put `tenant_id` in every chunk's metadata at ingestion (the controller does) and pass a filter at query time. +[Output 07](output/07-tenant-filter-and-injection.txt), on the in-memory store and on PostgreSQL alike: + +- with no filter, the top three chunks for "How many days of annual leave" included one from the other tenant; +- `new FilterExpressionBuilder().eq("tenant_id", "acme").build()` returned only that tenant's chunks; +- a filter *string* built as `"tenant_id == '" + input + "'"` with `input = globex' || tenant_id == 'acme` parsed to + `tenant_id == 'globex' || tenant_id == 'acme'` and returned both tenants' chunks; +- the same hostile text handed to `eq("tenant_id", input)` returned nothing, with a single or a double quote in it, on both stores. + +So: never build a filter string from a request value; pass the value to the builder. That is what +[`RagController`](../src/main/java/com/ankurm/rag/web/RagController.java) does. The two hostile strings tried are the ones in the +test; this is not a security audit of the filter parser. + +**Not tested:** how the filter performs. In pgvector, an approximate (HNSW) index is scanned first and the `WHERE` is applied to what +it returns, so a selective filter can return fewer than `topK` rows; pgvector 0.8.0 added iterative index scans to address this +([pgvector README, "Filtering"](https://github.com/pgvector/pgvector#filtering)). This repository ran pgvector **0.6.0**, which predates them, +and never ran a filtered query against enough rows to see the effect. If one tenant is a small fraction of the table, test it. + +## Reranking + +Spring AI 2.0.1 has the hook (`DocumentPostProcessor`) and no reranker: `spring-ai-rag` contains no class with "rerank" in its +name ([output 12](output/12-api-facts.txt)). [`LlmReranker`](../src/main/java/com/ankurm/rag/query/LlmReranker.java) is ours: it asks the +chat model to rate each candidate 0-10 and keeps the best `topN` ([output 08](output/08-reranking.txt)). + +- **Cost:** one model call per candidate. Four candidates made four calls; with `topK` 20 every question costs 20 rating calls before the answer. +- **Latency:** the calls run on virtual threads. With 200 ms per call standing in for the network, 20 calls took over 4 s one after another + and under 1 s through the reranker. That shows the structure works; a real API's rate limits are not modelled. +- **Failure:** the reply must be a bare integer. `"Score: 8"` failed to parse for every candidate, all scores became 0, and the survivors were the + vector-search order. Nothing errors; only the `rag.rerank.failures` counter moves. A reply of `" 9\n"` parsed fine. Watch the counter. + +**Not tested:** whether reranking improves which chunks reach the prompt. The scripted rater is word overlap and agrees with the fake embeddings. + +Next: [5. Generation and the faithfulness check](05-generation-and-the-faithfulness-check.md) diff --git a/rag/docs/05-generation-and-the-faithfulness-check.md b/rag/docs/05-generation-and-the-faithfulness-check.md new file mode 100644 index 0000000..40efdb8 --- /dev/null +++ b/rag/docs/05-generation-and-the-faithfulness-check.md @@ -0,0 +1,28 @@ +# 5. Generation and the faithfulness check + +prev: [4. Retrieval and reranking](04-retrieval-and-reranking.md) · [Index](../README.md) · next: [6. Observability and a production checklist](06-observability-and-production-checklist.md) + +Source: [`RagQueryService`](../src/main/java/com/ankurm/rag/query/RagQueryService.java). Test: +[`FaithfulnessTest`](../src/test/java/com/ankurm/rag/FaithfulnessTest.java) ([output 09](output/09-faithfulness-check.txt)). + +After the model answers, `RagQueryService` asks a second model call, `FactCheckingEvaluator`, whether the answer is supported by the +chunks that were in the prompt. The response reports one of three statuses: + +| status | when | fact check runs? | +|---|---|---| +| `answered` | chunks retrieved and the judge said "yes" | yes | +| `ungrounded` | chunks retrieved and the judge said anything other than "yes" (case aside), or the judge call failed | yes | +| `no_context` | nothing retrieved | no | + +Behaviours the test pins down: + +- **The verdict is matched literally.** A judge reply of `Yes.` counted as *not* grounded; `YES` counted as grounded. The evaluator + compares against "yes" ignoring case, not ignoring punctuation. A real judge model that likes to add a full stop would fail + every answer; instruct it to answer with one word, and watch `rag_queries_total{status="ungrounded"}`. +- **A failing judge is not a passing judge.** If the judge call throws (a rate limit, a timeout) the answer is reported `ungrounded` and + `rag.faithfulness.judge_failures` increments. The answer text is still returned, so the caller decides what to do with an unchecked answer. +- **No chunks, no check.** With nothing to check against, the service skips the judge (zero fact-check calls in the output) and reports `no_context`. +- **The check is only as good as the judge.** The tests script the judge, so they prove the wiring and the four outcomes, not that a real model + spots an invented claim. Measure that on your own questions before you rely on it, and note that it doubles the model calls per question. + +Next: [6. Observability and a production checklist](06-observability-and-production-checklist.md) diff --git a/rag/docs/06-observability-and-production-checklist.md b/rag/docs/06-observability-and-production-checklist.md new file mode 100644 index 0000000..7d2b3ba --- /dev/null +++ b/rag/docs/06-observability-and-production-checklist.md @@ -0,0 +1,38 @@ +# 6. Observability and a production checklist + +prev: [5. Generation and the faithfulness check](05-generation-and-the-faithfulness-check.md) · [Index](../README.md) + +## What the application exports + +`GET /actuator/prometheus` from the end-to-end run ([output 10](output/10-end-to-end.txt)) contained these `rag_` series after two +ingested files, one skipped re-upload and two questions: + +``` +rag_chunks_ingested_total 5.0 +rag_context_chunks_count 2 +rag_context_chunks_sum 5.0 +rag_context_chunks_max 4.0 +rag_ingestion_skipped_total 1.0 +rag_queries_total{status="answered"} 2.0 +rag_query_duration_seconds_count{status="answered"} 2 +``` + +Counters that exist in the code but were not triggered in that run (so they do not appear): `rag_rerank_failures_total` and +`rag_faithfulness_judge_failures_total`. Micrometer registers a counter when it is first incremented. + +The ones worth alerting on: a rising share of `ungrounded` and `no_context` in `rag_queries_total`; any increase in `rag_rerank_failures_total` +(silent degradation, chapter 4); any increase in `rag_faithfulness_judge_failures_total`; and `rag_context_chunks` drifting toward its maximum +(the threshold is accepting everything). + +## Checklist + +- [ ] Threshold set from questions your corpus cannot answer (chapter 4), not copied from a tutorial. +- [ ] `allowEmptyContext(false)`, and a test that an off-topic question produces `no_context`. +- [ ] Tenant filter built with `FilterExpressionBuilder`, never string concatenation; a test with a hostile tenant id. +- [ ] Filtered queries tested on a table the size of production (HNSW and `WHERE`, chapter 4). +- [ ] The ingestion tracker persisted, or deletion by `source_file` filter (chapter 3). +- [ ] Judge prompt forces a one-word answer; `ungrounded` is shown to the user as such, not hidden. +- [ ] Reranker replies parsed with a fallback and the failure counter alerted (chapter 4). Consider a structured-output call instead of parsing text. +- [ ] `init.sql` (or your migration tool) owns the schema; `initialize-schema` stays `false`. +- [ ] Embedding model and vector dimension changed together, and the table rebuilt when either changes: vectors from two models are not comparable. +- [ ] An evaluation set of real questions with known answers, run on every change to chunking, threshold or models. **This repository has none**; it is the largest gap between this code and a production system. diff --git a/rag/docs/output/01-pipeline-beans.txt b/rag/docs/output/01-pipeline-beans.txt new file mode 100644 index 0000000..a791142 --- /dev/null +++ b/rag/docs/output/01-pipeline-beans.txt @@ -0,0 +1,13 @@ +# The pipeline as Spring beans (the shipped configuration, fake models) + +stage / role bean name actual class +DocumentTransformer chunker TokenTextSplitter +EmbeddingModel embeddingModel HashingEmbeddingModel +VectorStore vectorStore PgVectorStore +RetrievalAugmentationAdvisor retrievalAdvisor RetrievalAugmentationAdvisor +LlmReranker reranker LlmReranker +ChatModel chatModel FakeChatModel +ChatClient ragChatClient DefaultChatClient +FactCheckingEvaluator factChecker FactCheckingEvaluator +IngestionService ingestionService IngestionService +RagQueryService ragQueryService RagQueryService diff --git a/rag/docs/output/02-token-text-splitter.txt b/rag/docs/output/02-token-text-splitter.txt new file mode 100644 index 0000000..75dc701 --- /dev/null +++ b/rag/docs/output/02-token-text-splitter.txt @@ -0,0 +1,27 @@ +# What TokenTextSplitter does to a 40-sentence document + +document: 4268 characters, 880 cl100k_base tokens + +--- new TokenTextSplitter() (defaults: 800 tokens, 350 min chars, 5 min length to embed) --- +chunks: 2, sizes in tokens: [792, 88] + +--- chunk size 100 tokens --- +chunks: 10 +chunk 0: 88 tokens, 419 chars, starts "Sentence 1 describes t", ends "ry full-time employee." +chunk 1: 88 tokens, 420 chars, starts "Sentence 5 describes t", ends "ry full-time employee." +chunk 2: 88 tokens, 426 chars, starts "Sentence 9 describes t", ends "ry full-time employee." +chunk 3: 88 tokens, 428 chars, starts "Sentence 13 describes ", ends "ry full-time employee." +chunk 4: 88 tokens, 427 chars, starts "Sentence 17 describes ", ends "ry full-time employee." +chunk 5: 88 tokens, 427 chars, starts "Sentence 21 describes ", ends "ry full-time employee." +chunk 6: 88 tokens, 428 chars, starts "Sentence 25 describes ", ends "ry full-time employee." +chunk 7: 88 tokens, 428 chars, starts "Sentence 29 describes ", ends "ry full-time employee." +chunk 8: 88 tokens, 428 chars, starts "Sentence 33 describes ", ends "ry full-time employee." +chunk 9: 88 tokens, 427 chars, starts "Sentence 37 describes ", ends "ry full-time employee." +metadata of chunk 1 (parent_document_id, a random UUID, left out): {chunk_index=1, page_number=1, source_file=long.txt, total_chunks=10} + +--- is there any overlap between neighbouring chunks? --- +boundaries where the first 30 characters of a chunk already appear in the chunk before it: 0 of 9 + +--- minChunkSizeChars: where the cut lands --- +minChunkSizeChars=350: 10 chunks, 9 of the first 9 end on a full stop +minChunkSizeChars= 50: 10 chunks, 9 of the first 9 end on a full stop diff --git a/rag/docs/output/03-recursive-and-semantic-chunkers.txt b/rag/docs/output/03-recursive-and-semantic-chunkers.txt new file mode 100644 index 0000000..bb824fc --- /dev/null +++ b/rag/docs/output/03-recursive-and-semantic-chunkers.txt @@ -0,0 +1,31 @@ +# RecursiveChunker and SemanticChunker + + +--- RecursiveChunker(maxChars=400, overlapChars=80) --- +chunks: 14, longest: 395 characters +chunk 0: 314 chars, starts "Sentence 1 describes the", ends "very full-time employee." +chunk 1: 388 chars, starts "for item 3 and states th", ends "very full-time employee." +chunk 2: 387 chars, starts "for item 6 and states th", ends "very full-time employee." +chunk 3: 394 chars, starts "for item 9 and states th", ends "very full-time employee." +chunk 4: 394 chars, starts "for item 12 and states t", ends "very full-time employee." +chunk 5: 395 chars, starts "for item 15 and states t", ends "very full-time employee." +chunk 6: 395 chars, starts "for item 18 and states t", ends "very full-time employee." +chunk 7: 394 chars, starts "for item 21 and states t", ends "very full-time employee." +chunk 8: 395 chars, starts "for item 24 and states t", ends "very full-time employee." +chunk 9: 394 chars, starts "for item 27 and states t", ends "very full-time employee." +chunk 10: 395 chars, starts "for item 30 and states t", ends "very full-time employee." +chunk 11: 395 chars, starts "for item 33 and states t", ends "very full-time employee." +chunk 12: 394 chars, starts "for item 36 and states t", ends "very full-time employee." +chunk 13: 180 chars, starts "for item 39 and states t", ends "very full-time employee." +metadata of chunk 2: {chunk_index=2, chunk_total=14, page_number=1, source_file=long.txt} +boundaries where the next chunk opens with words the previous one ended with: 13 of 13 + +--- one 900-character word-salad with no separators falls back to a hard cut --- +chunks: [400, 400, 100] + +--- SemanticChunker(distance 0.9) on three topics, four sentences each --- +chunks: 3 + [0] Annual leave is twenty days per year. Unused annual leave carries over until March. Leave requests go through the HR portal. Annual leave accrues monthly. + [1] Expenses need a receipt above fifty euros. Expense claims must be filed within thirty days. Receipts for expenses are uploaded as photos. Approved expenses are paid with salary. + [2] Remote work is allowed two days per week. Managers agree the remote work days. Remote work needs a quiet workspace. Remote work days are recorded in the calendar. +texts sent to the embedding model: 12 (12 sentences, one batched call) diff --git a/rag/docs/output/04-pdf-pages-and-metadata.txt b/rag/docs/output/04-pdf-pages-and-metadata.txt new file mode 100644 index 0000000..8473c76 --- /dev/null +++ b/rag/docs/output/04-pdf-pages-and-metadata.txt @@ -0,0 +1,27 @@ +# PagePdfDocumentReader: what one PDF page becomes + +pages in the PDF: 4, documents read: 4 +page document metadata: {page_number=1} +page document metadata: {page_number=2} +page document metadata: {page_number=3} +page document metadata: {page_number=4} + +--- page 2 as read (single spaces as dots, runs of 4+ as [n spaces], line ends as a pilcrow) --- +¶ +[12 spaces]4.1··Annual···Leave···Entitlement.···Full-time···employees[5 spaces]are··entitled·to·20··working···days[104 spaces]¶ +[12 spaces]of·annual···leave··per··calendar···year.··Part-time···employees[5 spaces]receive···leave··pro··rata.[107 spaces]¶ +¶ +[12 spaces]4.2··Leave···Carryover.···Unused[5 spaces]annual···leave··may···be··carried··over··for·a·maximum[6 spaces]of·5·days[96 spaces]¶ +[12 spaces]into·the··next··calendar···year··and···must··be··used···by·31··March.[133 spaces]¶ + + +--- page 2 after IngestionService.tidy --- +4.1·Annual·Leave·Entitlement.·Full-time·employees·are·entitled·to·20·working·days¶ +of·annual·leave·per·calendar·year.·Part-time·employees·receive·leave·pro·rata.¶ +¶ +4.2·Leave·Carryover.·Unused·annual·leave·may·be·carried·over·for·a·maximum·of·5·days¶ +into·the·next·calendar·year·and·must·be·used·by·31·March. + +--- size of page 2 --- +as read : 862 characters, longest run of spaces 133 +tidied : 304 characters, longest run of spaces 1 diff --git a/rag/docs/output/05-ingestion-idempotency.txt b/rag/docs/output/05-ingestion-idempotency.txt new file mode 100644 index 0000000..6a41b48 --- /dev/null +++ b/rag/docs/output/05-ingestion-idempotency.txt @@ -0,0 +1,14 @@ +# Ingesting the same handbook more than once (real PostgreSQL + pgvector) + +rows are counted with: select count(*) from vector_store +1. first upload -> status=ingested chunksWritten=4 chunksReplaced=0 | rows in table=4, texts embedded so far=4 +2. same bytes again -> status=skipped chunksWritten=0 chunksReplaced=0 | rows in table=4, texts embedded so far=4 +3. page 2 edited (20 -> 22 days) -> status=updated chunksWritten=4 chunksReplaced=4 | rows in table=4, texts embedded so far=8 +rows still saying "20 working days": 0, rows saying "22 working days": 1 +4. same text, exported again -> status=updated chunksWritten=4 chunksReplaced=4 | rows in table=4, texts embedded so far=12 +the two PDFs have identical text and different bytes: true +the file hash is a hash of bytes, so a re-export counts as a change and is re-embedded + +--- the naive version: vectorStore.add() on every upload, nothing remembered --- +the same 4 chunks added on two more uploads: rows in table 4 -> 12 +rows saying "22 working days" now: 3 diff --git a/rag/docs/output/06-retrieval-threshold-and-prompt.txt b/rag/docs/output/06-retrieval-threshold-and-prompt.txt new file mode 100644 index 0000000..16fa8c5 --- /dev/null +++ b/rag/docs/output/06-retrieval-threshold-and-prompt.txt @@ -0,0 +1,61 @@ +# Retrieval: threshold, prompt shape and the empty-context path + +the store holds the 4 pages of one handbook as 4 chunks; topK = 20 + +--- question: "How many days of annual leave do employees get?" --- +similarityThreshold 0.0 (the default): 4 chunk(s) + score 0.5891 page 2 "4.1 Annual Leave Entitlement. Full-time empl..." + score 0.4077 page 3 "4.3 Requesting Leave. All leave requests mus..." + score 0.2535 page 1 "Acme Employee Handbook 2026 3.5 Probationar..." + score 0.2023 page 4 "6.1 Expenses. Receipts are required for ever..." +similarityThreshold 0.3: 2 chunk(s) + score 0.5891 page 2 "4.1 Annual Leave Entitlement. Full-time empl..." + score 0.4077 page 3 "4.3 Requesting Leave. All leave requests mus..." + +--- question: "What is the capital of Mongolia?" --- +similarityThreshold 0.0 (the default): 4 chunk(s) + score 0.0000 page 1 "Acme Employee Handbook 2026 3.5 Probationar..." + score 0.0000 page 2 "4.1 Annual Leave Entitlement. Full-time empl..." + score 0.0000 page 3 "4.3 Requesting Leave. All leave requests mus..." + score 0.0000 page 4 "6.1 Expenses. Receipts are required for ever..." +similarityThreshold 0.3: 0 chunk(s) + +--- the prompt the model receives (threshold 0.3, question about leave) --- +Context information is below. + +--------------------- +4.1 Annual Leave Entitlement. Full-time employees are entitled to 20 working days +of annual leave per calendar year. Part-time employees receive leave pro rata. + +4.2 Leave Carryover. Unused annual leave may be carried over for a maximum of 5 days +into the next calendar year and must be used by 31 March. +4.3 Requesting Leave. All leave requests must be submitted through the HR portal +at least two weeks in advance for absences longer than three days. + +4.7 Sick Leave. Sick leave is separate from annual leave and is not deducted from it. +A medical certificate is required after three consecutive days. +--------------------- + +Given the context information and no prior knowledge, answer the query. + +Follow these rules: + +1. If the answer is not in the context, just say that you don't know. +2. Avoid statements like "Based on the context..." or "The provided information...". + +Query: How many days of annual leave do employees get? + +Answer: + + +--- nothing retrieved, allowEmptyContext(false): the prompt the model receives --- +The user query is outside your knowledge base. +Politely inform the user that you can't answer it. + + +--- nothing retrieved, allowEmptyContext(true): the prompt the model receives --- +What is the capital of Mongolia? + +--- off-topic question, default threshold 0.0, allowEmptyContext(false) --- +chunks placed in the prompt: 4 of 4 +the empty-context safety net fired: false diff --git a/rag/docs/output/07-tenant-filter-and-injection.txt b/rag/docs/output/07-tenant-filter-and-injection.txt new file mode 100644 index 0000000..9a48314 --- /dev/null +++ b/rag/docs/output/07-tenant-filter-and-injection.txt @@ -0,0 +1,42 @@ +# Tenant isolation: metadata filters, and a filter built from user input + +two tenants each upload a handbook with a section 4.1 on annual leave +acme says 20 working days, globex says 25 + +--- SimpleVectorStore (in memory) --- +no filter, top 3: 3 chunk(s) + tenant=acme page=2 "4.1 Annual Leave Entitlement. Full-time ..." + tenant=acme page=3 "4.3 Requesting Leave. All leave requests..." + tenant=globex page=1 "Globex Staff Manual 2026 4.1 Annual Lea..." +eq("tenant_id", "acme") built with FilterExpressionBuilder, top 3: 3 chunk(s) + tenant=acme page=2 "4.1 Annual Leave Entitlement. Full-time ..." + tenant=acme page=3 "4.3 Requesting Leave. All leave requests..." + tenant=acme page=1 "Acme Employee Handbook 2026 3.5 Probati..." +filter string built by concatenation: tenant_id == 'globex' || tenant_id == 'acme' + result, top 5: 5 chunk(s) + tenant=acme page=2 "4.1 Annual Leave Entitlement. Full-time ..." + tenant=acme page=3 "4.3 Requesting Leave. All leave requests..." + tenant=globex page=1 "Globex Staff Manual 2026 4.1 Annual Lea..." + tenant=acme page=1 "Acme Employee Handbook 2026 3.5 Probati..." + tenant=acme page=4 "6.1 Expenses. Receipts are required for ..." +same text passed to FilterExpressionBuilder.eq(), top 5: 0 chunk(s) +double-quote variant passed to FilterExpressionBuilder.eq(), top 5: 0 chunk(s) + +--- PgVectorStore (PostgreSQL + pgvector) --- +no filter, top 3: 3 chunk(s) + tenant=acme page=2 "4.1 Annual Leave Entitlement. Full-time ..." + tenant=acme page=3 "4.3 Requesting Leave. All leave requests..." + tenant=globex page=1 "Globex Staff Manual 2026 4.1 Annual Lea..." +eq("tenant_id", "acme") built with FilterExpressionBuilder, top 3: 3 chunk(s) + tenant=acme page=2 "4.1 Annual Leave Entitlement. Full-time ..." + tenant=acme page=3 "4.3 Requesting Leave. All leave requests..." + tenant=acme page=1 "Acme Employee Handbook 2026 3.5 Probati..." +filter string built by concatenation: tenant_id == 'globex' || tenant_id == 'acme' + result, top 5: 5 chunk(s) + tenant=acme page=2 "4.1 Annual Leave Entitlement. Full-time ..." + tenant=acme page=3 "4.3 Requesting Leave. All leave requests..." + tenant=globex page=1 "Globex Staff Manual 2026 4.1 Annual Lea..." + tenant=acme page=1 "Acme Employee Handbook 2026 3.5 Probati..." + tenant=acme page=4 "6.1 Expenses. Receipts are required for ..." +same text passed to FilterExpressionBuilder.eq(), top 5: 0 chunk(s) +double-quote variant passed to FilterExpressionBuilder.eq(), top 5: 0 chunk(s) diff --git a/rag/docs/output/08-reranking.txt b/rag/docs/output/08-reranking.txt new file mode 100644 index 0000000..7df82db --- /dev/null +++ b/rag/docs/output/08-reranking.txt @@ -0,0 +1,29 @@ +# LLM reranking: calls, order, latency and failure + + +--- one model call per candidate; only topN survive (topN = 2) --- +candidates in: 4, model calls made: 4, chunks out: 2 +order from the vector search, best first: + similarity 0.5891 page 2 + similarity 0.4077 page 3 + similarity 0.2535 page 1 + similarity 0.2023 page 4 +order after reranking, best first: + rerank_score 8 page 2 + rerank_score 6 page 3 +the rating prompt for the page 2 candidate (calls run concurrently, so pick it by content): +Rate how well the PASSAGE helps answer the QUESTION, from 0 (irrelevant) to 10 (answers it). +Reply with a single integer and nothing else. +QUESTION: How many days of annual leave do employees get? +PASSAGE: 4.1 Annual Leave Entitlement. Full-time employees are entitled to 20 working days + +--- latency: 20 candidates, each rating call takes 200 ms (a Thread.sleep in the fake model) --- +the 20 calls one after another take 4000 ms or more: true +LlmReranker, one virtual thread per candidate, takes under 1000 ms: true +a real API adds its own rate limits, which this test cannot show + +--- failure: the model does not reply with a bare integer --- +reply "Score: 8" for every candidate -> failures counted: 4 of 4 +scores assigned: [0, 0] +pages kept, in order: [2, 3] (the vector-search order, because every score is 0) +reply " 9\n" (padded) -> score 9 diff --git a/rag/docs/output/09-faithfulness-check.txt b/rag/docs/output/09-faithfulness-check.txt new file mode 100644 index 0000000..60f5215 --- /dev/null +++ b/rag/docs/output/09-faithfulness-check.txt @@ -0,0 +1,32 @@ +# Generation and the faithfulness check + + +--- 1. the answer is in the chunks --- +status=answered grounded=true sources=4 +answer: Full-time employees are entitled to 20 working days +of annual leave per calendar year. + +--- 2. the model answers with something the chunks do not say --- +status=ungrounded grounded=false sources=4 +answer: Employees get 30 days of annual leave. +the check the judge model was given: + Evaluate whether or not the following claim is supported by the provided document. + Respond with "yes" if the claim is supported, or "no" if it is not. + + +--- 3. the judge says "Yes." instead of "yes" --- +status=ungrounded grounded=false sources=4 +answer: Full-time employees are entitled to 20 working days +of annual leave per calendar year. +with the reply "YES": grounded=true + +--- 4. the judge call itself fails --- +status=ungrounded grounded=false sources=4 +answer: Full-time employees are entitled to 20 working days +of annual leave per calendar year. +rag.faithfulness.judge_failures = 1 + +--- 5. nothing is retrieved (threshold 0.3, off-topic question) --- +status=no_context grounded=false sources=0 +answer: I don't have enough information in the provided documents. +fact-check calls made: 0 diff --git a/rag/docs/output/10-end-to-end.txt b/rag/docs/output/10-end-to-end.txt new file mode 100644 index 0000000..43f814b --- /dev/null +++ b/rag/docs/output/10-end-to-end.txt @@ -0,0 +1,26 @@ +# End to end: HTTP, real PostgreSQL + pgvector, the shipped application.yml + +schema from init.sql: [document_chunks_embedding_idx, document_chunks_pkey] + +--- POST /api/ingest (acme, then globex, then acme again) --- +{"filename":"acme-handbook.pdf","status":"ingested","chunksWritten":4,"chunksReplaced":0} +{"filename":"globex-manual.pdf","status":"ingested","chunksWritten":1,"chunksReplaced":0} +{"filename":"acme-handbook.pdf","status":"skipped","chunksWritten":0,"chunksReplaced":0} + +--- what is in the table --- +rows: 5 +rows per tenant: [acme=4, globex=1] +metadata of one row: {"doc_type": "general", "tenant_id": "globex", "chunk_index": 0, "page_number": 1, "source_file": "globex-manual.pdf", "total_chunks": 1} + +--- POST /api/query --- +tenantId acme: {"answer":"Full-time employees are entitled to 20 working days\nof annual leave per calendar year.","sources":[{"file":"acme-handbook.pdf","page":2,"rerankScore":8,"preview":"4.1 Annual Leave Entitlement. Full-time employees are entitled to 20 working day"},{"file":"acme-handbook.pdf","page":3,"rerankScore":6,"preview":"4.3 Requesting Leave. All leave requests must be submitted through the HR portal"},{"file":"acme-handbook.pdf","page":1,"rerankScore":6,"preview":"Acme Employee Handbook 2026\n\n3.5 Probationary Period. During the three month pro"},{"file":"acme-handbook.pdf","page":4,"rerankScore":4,"preview":"6.1 Expenses. Receipts are required for every expense above 50 euros.\nClaims mus"}],"grounded":true,"status":"answered"} +tenantId globex: {"answer":"Full-time staff are entitled to 25 working days\nof annual leave per calendar year.","sources":[{"file":"globex-manual.pdf","page":1,"rerankScore":6,"preview":"Globex Staff Manual 2026\n\n4.1 Annual Leave Entitlement. Full-time staff are enti"}],"grounded":true,"status":"answered"} + +--- GET /actuator/prometheus (only the rag_ series; the timer's sum and max are left out because they change every run) --- +rag_chunks_ingested_total 5.0 +rag_context_chunks_count 2 +rag_context_chunks_sum 5.0 +rag_context_chunks_max 4.0 +rag_ingestion_skipped_total 1.0 +rag_queries_total{status="answered"} 2.0 +rag_query_duration_seconds_count{status="answered"} 2 diff --git a/rag/docs/output/11-question-answer-advisor.txt b/rag/docs/output/11-question-answer-advisor.txt new file mode 100644 index 0000000..675ac64 --- /dev/null +++ b/rag/docs/output/11-question-answer-advisor.txt @@ -0,0 +1,39 @@ +# QuestionAnswerAdvisor: the smallest RAG + +only the prompts are recorded: what a real model would reply is not something this repository tests + +--- the prompt the model received --- +How many days of annual leave do employees get? + +Context information is below, surrounded by --------------------- + +--------------------- +4.1 Annual Leave Entitlement. Full-time employees are entitled to 20 working days +of annual leave per calendar year. Part-time employees receive leave pro rata. + +4.2 Leave Carryover. Unused annual leave may be carried over for a maximum of 5 days +into the next calendar year and must be used by 31 March. +4.3 Requesting Leave. All leave requests must be submitted through the HR portal +at least two weeks in advance for absences longer than three days. + +4.7 Sick Leave. Sick leave is separate from annual leave and is not deducted from it. +A medical certificate is required after three consecutive days. +--------------------- + +Given the context and provided history information and not prior knowledge, +reply to the user comment. If the answer is not in the context, inform +the user that you can't answer the question. + + +--- the prompt for an off-topic question (nothing passes the threshold) --- +What is the capital of Mongolia? + +Context information is below, surrounded by --------------------- + +--------------------- +--------------------- + +Given the context and provided history information and not prior knowledge, +reply to the user comment. If the answer is not in the context, inform +the user that you can't answer the question. + diff --git a/rag/docs/output/12-api-facts.txt b/rag/docs/output/12-api-facts.txt new file mode 100644 index 0000000..21896e9 --- /dev/null +++ b/rag/docs/output/12-api-facts.txt @@ -0,0 +1,176 @@ +# Spring AI API facts, read with javap from the jars this build resolves + +spring-ai.version: 2.0.1 + +## classes in org/springframework/ai/rag (top level, from spring-ai-rag) +Query +advisor/RetrievalAugmentationAdvisor +advisor/package-info +generation/augmentation/ContextualQueryAugmenter +generation/augmentation/QueryAugmenter +generation/augmentation/package-info +generation/package-info +package-info +postretrieval/document/DocumentPostProcessor +postretrieval/document/package-info +postretrieval/package-info +preretrieval/package-info +preretrieval/query/expansion/MultiQueryExpander +preretrieval/query/expansion/QueryExpander +preretrieval/query/expansion/package-info +preretrieval/query/transformation/CompressionQueryTransformer +preretrieval/query/transformation/QueryTransformer +preretrieval/query/transformation/RewriteQueryTransformer +preretrieval/query/transformation/TranslationQueryTransformer +preretrieval/query/transformation/package-info +retrieval/join/ConcatenationDocumentJoiner +retrieval/join/DocumentJoiner +retrieval/join/package-info +retrieval/search/DocumentRetriever +retrieval/search/VectorStoreDocumentRetriever +retrieval/search/package-info +util/PromptAssert +util/package-info + +## classes anywhere in spring-ai-rag whose name contains "rerank": 0 + +## classes in any jar on this project's classpath whose name contains "SemanticSearchCache" or "SemanticCache": 0 + +## org.springframework.ai.transformer.splitter.TokenTextSplitter +public class org.springframework.ai.transformer.splitter.TokenTextSplitter extends org.springframework.ai.transformer.splitter.TextSplitter { + public org.springframework.ai.transformer.splitter.TokenTextSplitter(); + public org.springframework.ai.transformer.splitter.TokenTextSplitter(boolean); + public org.springframework.ai.transformer.splitter.TokenTextSplitter(com.knuddels.jtokkit.api.EncodingType); + public org.springframework.ai.transformer.splitter.TokenTextSplitter(com.knuddels.jtokkit.api.EncodingType, boolean); + public org.springframework.ai.transformer.splitter.TokenTextSplitter(int, int, int, int, boolean, List); + public static org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder builder(); +} + +## org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder +public final class org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder { + public org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder withEncodingType(com.knuddels.jtokkit.api.EncodingType); + public org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder withChunkSize(int); + public org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder withMinChunkSizeChars(int); + public org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder withMinChunkLengthToEmbed(int); + public org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder withMaxNumChunks(int); + public org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder withKeepSeparator(boolean); + public org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder withPunctuationMarks(List); + public org.springframework.ai.transformer.splitter.TokenTextSplitter build(); +} + +## org.springframework.ai.reader.pdf.PagePdfDocumentReader +public class org.springframework.ai.reader.pdf.PagePdfDocumentReader implements org.springframework.ai.document.DocumentReader { + public static final String METADATA_START_PAGE_NUMBER; + public static final String METADATA_END_PAGE_NUMBER; + public static final String METADATA_FILE_NAME; + public org.springframework.ai.reader.pdf.PagePdfDocumentReader(String); + public org.springframework.ai.reader.pdf.PagePdfDocumentReader(org.springframework.core.io.Resource); + public org.springframework.ai.reader.pdf.PagePdfDocumentReader(String, org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig); + public org.springframework.ai.reader.pdf.PagePdfDocumentReader(org.springframework.core.io.Resource, org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig); + public List get(); + public Object get(); +} + +## org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig$Builder +public final class org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig$Builder { + public org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig$Builder withPageExtractedTextFormatter(org.springframework.ai.reader.ExtractedTextFormatter); + public org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig$Builder withPagesPerDocument(int); + public org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig$Builder withPageTopMargin(int); + public org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig$Builder withPageBottomMargin(int); + public org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig$Builder withReversedParagraphPosition(boolean); + public org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig$Builder addPageRange(int, int); + public org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig build(); +} + +## org.springframework.ai.rag.postretrieval.document.DocumentPostProcessor +public interface org.springframework.ai.rag.postretrieval.document.DocumentPostProcessor extends function.BiFunction, List> { + public abstract List process(org.springframework.ai.rag.Query, List); + public default List apply(org.springframework.ai.rag.Query, List); + public default Object apply(Object, Object); +} + +## org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever +public final class org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever implements org.springframework.ai.rag.retrieval.search.DocumentRetriever { + public static final String FILTER_EXPRESSION; + public org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever(org.springframework.ai.vectorstore.VectorStore, Double, Integer, function.Supplier); + public List retrieve(org.springframework.ai.rag.Query); + public static org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever$Builder builder(); +} + +## org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever$Builder +public final class org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever$Builder { + public org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever$Builder vectorStore(org.springframework.ai.vectorstore.VectorStore); + public org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever$Builder similarityThreshold(Double); + public org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever$Builder topK(Integer); + public org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever$Builder filterExpression(org.springframework.ai.vectorstore.filter.Filter$Expression); + public org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever$Builder filterExpression(function.Supplier); + public org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever build(); +} + +## org.springframework.ai.rag.generation.augmentation.ContextualQueryAugmenter$Builder +public final class org.springframework.ai.rag.generation.augmentation.ContextualQueryAugmenter$Builder { + public org.springframework.ai.rag.generation.augmentation.ContextualQueryAugmenter$Builder(); + public org.springframework.ai.rag.generation.augmentation.ContextualQueryAugmenter$Builder promptTemplate(org.springframework.ai.chat.prompt.PromptTemplate); + public org.springframework.ai.rag.generation.augmentation.ContextualQueryAugmenter$Builder emptyContextPromptTemplate(org.springframework.ai.chat.prompt.PromptTemplate); + public org.springframework.ai.rag.generation.augmentation.ContextualQueryAugmenter$Builder allowEmptyContext(Boolean); + public org.springframework.ai.rag.generation.augmentation.ContextualQueryAugmenter$Builder documentFormatter(function.Function, String>); + public org.springframework.ai.rag.generation.augmentation.ContextualQueryAugmenter build(); +} + +## org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder +public final class org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder { + public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder queryTransformers(List); + public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder queryTransformers(org.springframework.ai.rag.preretrieval.query.transformation.QueryTransformer...); + public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder queryExpander(org.springframework.ai.rag.preretrieval.query.expansion.QueryExpander); + public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder documentRetriever(org.springframework.ai.rag.retrieval.search.DocumentRetriever); + public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder documentJoiner(org.springframework.ai.rag.retrieval.join.DocumentJoiner); + public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder documentPostProcessors(List); + public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder documentPostProcessors(org.springframework.ai.rag.postretrieval.document.DocumentPostProcessor...); + public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder queryAugmenter(org.springframework.ai.rag.generation.augmentation.QueryAugmenter); + public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder taskExecutor(org.springframework.core.task.TaskExecutor); + public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder scheduler(reactor.core.scheduler.Scheduler); + public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder order(Integer); + public org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor build(); +} + +## org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor$Builder +public final class org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor$Builder { + public org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor$Builder promptTemplate(org.springframework.ai.chat.prompt.PromptTemplate); + public org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor$Builder searchRequest(org.springframework.ai.vectorstore.SearchRequest); + public org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor$Builder protectFromBlocking(boolean); + public org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor$Builder scheduler(reactor.core.scheduler.Scheduler); + public org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor$Builder order(int); + public org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor build(); +} + +## org.springframework.ai.chat.evaluation.FactCheckingEvaluator +public class org.springframework.ai.chat.evaluation.FactCheckingEvaluator implements org.springframework.ai.evaluation.Evaluator { + public static org.springframework.ai.chat.evaluation.FactCheckingEvaluator forBespokeMinicheck(org.springframework.ai.chat.client.ChatClient$Builder); + public org.springframework.ai.evaluation.EvaluationResponse evaluate(org.springframework.ai.evaluation.EvaluationRequest); + public static org.springframework.ai.chat.evaluation.FactCheckingEvaluator$Builder builder(org.springframework.ai.chat.client.ChatClient$Builder); +} + +## org.springframework.ai.vectorstore.VectorStore +public interface org.springframework.ai.vectorstore.VectorStore extends org.springframework.ai.document.DocumentWriter,org.springframework.ai.vectorstore.VectorStoreRetriever { + public default String getName(); + public abstract void add(List); + public default void accept(List); + public abstract void delete(List); + public abstract void delete(org.springframework.ai.vectorstore.filter.Filter$Expression); + public default void delete(String); + public default Optional getNativeClient(); + public default void accept(Object); +} + +## org.springframework.ai.vectorstore.SearchRequest$Builder +public final class org.springframework.ai.vectorstore.SearchRequest$Builder { + public org.springframework.ai.vectorstore.SearchRequest$Builder(); + public org.springframework.ai.vectorstore.SearchRequest$Builder query(String); + public org.springframework.ai.vectorstore.SearchRequest$Builder topK(int); + public org.springframework.ai.vectorstore.SearchRequest$Builder similarityThreshold(double); + public org.springframework.ai.vectorstore.SearchRequest$Builder similarityThresholdAll(); + public org.springframework.ai.vectorstore.SearchRequest$Builder filterExpression(org.springframework.ai.vectorstore.filter.Filter$Expression); + public org.springframework.ai.vectorstore.SearchRequest$Builder filterExpression(String); + public org.springframework.ai.vectorstore.SearchRequest build(); +} + diff --git a/rag/docs/output/13-dependencies.txt b/rag/docs/output/13-dependencies.txt new file mode 100644 index 0000000..8b37515 --- /dev/null +++ b/rag/docs/output/13-dependencies.txt @@ -0,0 +1,45 @@ +# Dependency tree, filtered + +## Spring Boot and Spring AI versions +spring-boot-starter-parent 4.1.1 +spring-ai-bom 2.0.1 + +## where spring-jdbc comes from (it is not under any Spring AI artifact) +41:+- org.springframework.boot:spring-boot-starter-jdbc:jar:4.1.1:compile +42:| +- org.springframework.boot:spring-boot-jdbc:jar:4.1.1:compile +47:| | \- org.springframework:spring-jdbc:jar:7.0.9:compile +48:| \- com.zaxxer:HikariCP:jar:7.0.2:compile + +## what the pgvector starter brings ++- org.springframework.ai:spring-ai-starter-vector-store-pgvector:jar:2.0.1:compile +| +- org.springframework.ai:spring-ai-autoconfigure-vector-store-pgvector:jar:2.0.1:compile +| +- org.springframework.ai:spring-ai-autoconfigure-vector-store-observation:jar:2.0.1:compile +| \- org.springframework.ai:spring-ai-pgvector-store:jar:2.0.1:compile +| +- org.postgresql:postgresql:jar:42.7.13:compile +| | \- org.checkerframework:checker-qual:jar:3.55.1:runtime +| \- com.pgvector:pgvector:jar:0.1.6:compile ++- org.springframework.ai:spring-ai-rag:jar:2.0.1:compile + +## every Spring AI artifact on the classpath +org.springframework.ai:spring-ai-autoconfigure-model-chat-client:jar:2.0.1 +org.springframework.ai:spring-ai-autoconfigure-model-chat-memory:jar:2.0.1 +org.springframework.ai:spring-ai-autoconfigure-model-chat-observation:jar:2.0.1 +org.springframework.ai:spring-ai-autoconfigure-model-embedding-observation:jar:2.0.1 +org.springframework.ai:spring-ai-autoconfigure-model-image-observation:jar:2.0.1 +org.springframework.ai:spring-ai-autoconfigure-model-openai:jar:2.0.1 +org.springframework.ai:spring-ai-autoconfigure-model-tool:jar:2.0.1 +org.springframework.ai:spring-ai-autoconfigure-retry:jar:2.0.1 +org.springframework.ai:spring-ai-autoconfigure-vector-store-observation:jar:2.0.1 +org.springframework.ai:spring-ai-autoconfigure-vector-store-pgvector:jar:2.0.1 +org.springframework.ai:spring-ai-client-chat:jar:2.0.1 +org.springframework.ai:spring-ai-commons:jar:2.0.1 +org.springframework.ai:spring-ai-model:jar:2.0.1 +org.springframework.ai:spring-ai-openai:jar:2.0.1 +org.springframework.ai:spring-ai-pdf-document-reader:jar:2.0.1 +org.springframework.ai:spring-ai-pgvector-store:jar:2.0.1 +org.springframework.ai:spring-ai-rag:jar:2.0.1 +org.springframework.ai:spring-ai-starter-model-openai:jar:2.0.1 +org.springframework.ai:spring-ai-starter-vector-store-pgvector:jar:2.0.1 +org.springframework.ai:spring-ai-template-st:jar:2.0.1 +org.springframework.ai:spring-ai-vector-store-advisor:jar:2.0.1 +org.springframework.ai:spring-ai-vector-store:jar:2.0.1 diff --git a/rag/docs/output/14-legacy-1x.txt b/rag/docs/output/14-legacy-1x.txt new file mode 100644 index 0000000..efe9b59 --- /dev/null +++ b/rag/docs/output/14-legacy-1x.txt @@ -0,0 +1,37 @@ +# The 1.x article's code, against 1.1.0 and 2.0.1 + +## The starter artifact ids in the 1.x article: latest version ever published +spring-ai-openai-spring-boot-starter latest: 1.0.0-M6 +spring-ai-pgvector-store-spring-boot-starter latest: 1.0.0-M6 +the ids that replaced them: +spring-ai-starter-model-openai latest: 2.0.1 +spring-ai-starter-vector-store-pgvector latest: 2.0.1 + +## mvn validate on the article's dependency block with spring-ai-bom 1.1.0 +'dependencies.dependency.version' for org.springframework.ai:spring-ai-openai-spring-boot-starter:jar is missing. @ line 31, column 17 +'dependencies.dependency.version' for org.springframework.ai:spring-ai-pgvector-store-spring-boot-starter:jar is missing. @ line 35, column 17 + +## mvn validate on the article's dependency block with spring-ai-bom 2.0.1 +'dependencies.dependency.version' for org.springframework.ai:spring-ai-openai-spring-boot-starter:jar is missing. @ line 31, column 17 +'dependencies.dependency.version' for org.springframework.ai:spring-ai-pgvector-store-spring-boot-starter:jar is missing. @ line 35, column 17 + +## javac legacy-1x/src/LegacyIngestion.java against Spring AI 1.1.0 +legacy-1x/src/LegacyIngestion.java:17: error: incompatible types: ExtractedTextFormatter is not a functional interface + .withPageExtractedTextFormatter(text -> text.replaceAll("s{3,}", " ")) + ^ + +## javac legacy-1x/src/LegacyIngestion.java against Spring AI 2.0.1 +legacy-1x/src/LegacyIngestion.java:17: error: incompatible types: ExtractedTextFormatter is not a functional interface + .withPageExtractedTextFormatter(text -> text.replaceAll("s{3,}", " ")) + ^ +legacy-1x/src/LegacyIngestion.java:22: error: no suitable constructor found for TokenTextSplitter(int,int,int,int,boolean) + TokenTextSplitter splitter = new TokenTextSplitter(512, 128, 5, 10_000, true); + ^ + +## the 1.x configuration keys in the metadata of spring-ai-autoconfigure-model-openai +1.1.0 spring.ai.openai.chat.options.model current +1.1.0 spring.ai.openai.chat.options.temperature current +1.1.0 spring.ai.openai.embedding.options.model current +2.0.1 spring.ai.openai.chat.options.model deprecated, use spring.ai.openai.chat.model +2.0.1 spring.ai.openai.chat.options.temperature deprecated, use spring.ai.openai.chat.temperature +2.0.1 spring.ai.openai.embedding.options.model deprecated, use spring.ai.openai.embedding.model diff --git a/rag/docs/output/15-config-keys.txt b/rag/docs/output/15-config-keys.txt new file mode 100644 index 0000000..d2ce91c --- /dev/null +++ b/rag/docs/output/15-config-keys.txt @@ -0,0 +1,21 @@ +# spring.ai.* keys in application.yml against the jars' configuration metadata + +property names in the jars' metadata: 1528 + +--- keys in the shipped application.yml --- +spring.ai.openai.api-key current +spring.ai.openai.chat.model current +spring.ai.openai.chat.temperature current +spring.ai.openai.embedding.model current +spring.ai.vectorstore.pgvector.dimensions current +spring.ai.vectorstore.pgvector.distance-type current +spring.ai.vectorstore.pgvector.index-type current +spring.ai.vectorstore.pgvector.initialize-schema current +spring.ai.vectorstore.pgvector.schema-name current +spring.ai.vectorstore.pgvector.table-name current + +--- keys written the 1.x way --- +spring.ai.openai.chat.options.model DEPRECATED, use spring.ai.openai.chat.model +spring.ai.openai.chat.options.temperature DEPRECATED, use spring.ai.openai.chat.temperature +spring.ai.openai.embedding.options.model DEPRECATED, use spring.ai.openai.embedding.model +spring.ai.openai.chat.optoins.model UNKNOWN diff --git a/rag/init.sql b/rag/init.sql new file mode 100644 index 0000000..cfd961c --- /dev/null +++ b/rag/init.sql @@ -0,0 +1,20 @@ +-- The schema the application expects. docker-compose.yml runs this on first start, and +-- scripts/pg-up.sh runs it against a local PostgreSQL. initialize-schema is false in +-- application.yml, so the application never creates or alters any of this itself. + +CREATE EXTENSION IF NOT EXISTS vector; + +CREATE SCHEMA IF NOT EXISTS rag; + +-- metadata is json, the type Spring AI's PgVectorStore writes and its filter queries cast from. +-- 1536 is the size of a text-embedding-3-small vector; change it together with the model. +CREATE TABLE IF NOT EXISTS rag.document_chunks ( + id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + content text, + metadata json, + embedding vector(1536) +); + +-- HNSW with cosine distance, matching index-type and distance-type in application.yml. +CREATE INDEX IF NOT EXISTS document_chunks_embedding_idx + ON rag.document_chunks USING hnsw (embedding vector_cosine_ops); diff --git a/rag/legacy-1x/pom.xml b/rag/legacy-1x/pom.xml new file mode 100644 index 0000000..c00a8fb --- /dev/null +++ b/rag/legacy-1x/pom.xml @@ -0,0 +1,44 @@ + + + + 4.0.0 + com.ankurm + legacy-1x + 0 + pom + + + 1.1.0 + + + + + + org.springframework.ai + spring-ai-bom + ${spring-ai.version} + pom + import + + + + + + + org.springframework.ai + spring-ai-openai-spring-boot-starter + + + org.springframework.ai + spring-ai-pgvector-store-spring-boot-starter + + + org.springframework.ai + spring-ai-pdf-document-reader + + + diff --git a/rag/legacy-1x/src/LegacyIngestion.java b/rag/legacy-1x/src/LegacyIngestion.java new file mode 100644 index 0000000..5e1dc1b --- /dev/null +++ b/rag/legacy-1x/src/LegacyIngestion.java @@ -0,0 +1,25 @@ +import java.util.List; + +import org.springframework.ai.document.Document; +import org.springframework.ai.reader.pdf.PagePdfDocumentReader; +import org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig; +import org.springframework.ai.transformer.splitter.TokenTextSplitter; +import org.springframework.core.io.Resource; + +/** + * The two Spring AI calls from the 1.x article's ingestion service, unchanged (including the + * regex, which lost its backslash when the article was first published). Compiled, not run. + */ +class LegacyIngestion { + + List ingest(Resource pdfResource) { + PdfDocumentReaderConfig readerConfig = PdfDocumentReaderConfig.builder() + .withPageExtractedTextFormatter(text -> text.replaceAll("s{3,}", " ")) + .withPagesPerDocument(1) + .build(); + List pages = new PagePdfDocumentReader(pdfResource, readerConfig).get(); + + TokenTextSplitter splitter = new TokenTextSplitter(512, 128, 5, 10_000, true); + return splitter.apply(pages); + } +} diff --git a/rag/pom.xml b/rag/pom.xml new file mode 100644 index 0000000..214102d --- /dev/null +++ b/rag/pom.xml @@ -0,0 +1,106 @@ + + + 4.0.0 + + + org.springframework.boot + spring-boot-starter-parent + 4.1.1 + + + + com.ankurm + rag + 1.0.0 + rag + Production RAG with Spring AI 2.0: chunking, ingestion, retrieval, reranking, a faithfulness check, metrics + + + 25 + + 2.0.1 + + + + + + org.springframework.ai + spring-ai-bom + ${spring-ai.version} + pom + import + + + + + + + org.springframework.boot + spring-boot-starter-webmvc + + + org.springframework.boot + spring-boot-starter-actuator + + + + org.springframework.boot + spring-boot-starter-jdbc + + + + + org.springframework.ai + spring-ai-starter-model-openai + + + + org.springframework.ai + spring-ai-starter-vector-store-pgvector + + + + org.springframework.ai + spring-ai-rag + + + + org.springframework.ai + spring-ai-vector-store-advisor + + + org.springframework.ai + spring-ai-pdf-document-reader + + + io.micrometer + micrometer-registry-prometheus + + + + org.springframework.boot + spring-boot-starter-test + test + + + + + + + org.springframework.boot + spring-boot-maven-plugin + + + org.apache.maven.plugins + maven-surefire-plugin + + + -Duser.timezone=UTC -Dstdout.encoding=UTF-8 -Dfile.encoding=UTF-8 + + + + + diff --git a/rag/scripts/capture-dependencies.sh b/rag/scripts/capture-dependencies.sh new file mode 100755 index 0000000..8ddf743 --- /dev/null +++ b/rag/scripts/capture-dependencies.sh @@ -0,0 +1,24 @@ +#!/usr/bin/env bash +# Writes docs/output/13-dependencies.txt: which artifact brings what, straight from Maven. +set -euo pipefail +cd "$(dirname "$0")/.." +mkdir -p docs/output target +mvn -B dependency:tree -DoutputFile=target/tree.txt >/dev/null +OUT=docs/output/13-dependencies.txt +{ + echo "# Dependency tree, filtered" + echo + echo "## Spring Boot and Spring AI versions" + echo "spring-boot-starter-parent $(grep -A2 'spring-boot-starter-parent' pom.xml | grep -m1 -o '[^<]*' | sed 's/.*>//')" + echo "spring-ai-bom $(grep -m1 -o '[^<]*' pom.xml | sed 's/.*>//')" + echo + echo "## where spring-jdbc comes from (it is not under any Spring AI artifact)" + grep -n -E 'spring-jdbc|spring-boot-starter-jdbc|spring-boot-jdbc|HikariCP' target/tree.txt + echo + echo "## what the pgvector starter brings" + grep -A8 'spring-ai-starter-vector-store-pgvector' target/tree.txt | sed -n 1,8p + echo + echo "## every Spring AI artifact on the classpath" + grep -o 'org.springframework.ai:[a-z0-9-]*:jar:[0-9.]*' target/tree.txt | sort -u +} > "$OUT" +echo "wrote $OUT" diff --git a/rag/scripts/capture-javap.sh b/rag/scripts/capture-javap.sh new file mode 100755 index 0000000..239592b --- /dev/null +++ b/rag/scripts/capture-javap.sh @@ -0,0 +1,39 @@ +#!/usr/bin/env bash +# Writes docs/output/12-api-facts.txt: the public API of every Spring AI class this repository +# relies on, read from the jars the build resolves (not from documentation). +set -euo pipefail +cd "$(dirname "$0")/.." +mkdir -p docs/output target +mvn -B -q dependency:build-classpath -Dmdep.outputFile=target/classpath.txt >/dev/null +CP="$(cat target/classpath.txt)" +OUT=docs/output/12-api-facts.txt +jar_of() { tr ':' '\n' < target/classpath.txt | grep "/$1-[0-9]" | head -1; } +jp() { echo "## $1"; javap -cp "$CP" -public "$1" | grep -v '^Compiled' | sed 's/java\.util\.//g; s/java\.lang\.//g'; echo; } + +{ + echo "# Spring AI API facts, read with javap from the jars this build resolves" + echo + echo "spring-ai.version: $(grep -m1 -o '[^<]*' pom.xml | sed 's/.*>//')" + echo + echo "## classes in org/springframework/ai/rag (top level, from spring-ai-rag)" + unzip -Z1 "$(jar_of spring-ai-rag)" | grep '\.class$' | grep -v '\$' | sed 's|org/springframework/ai/rag/||; s|\.class||' | sort + echo + echo "## classes anywhere in spring-ai-rag whose name contains \"rerank\": $(unzip -Z1 "$(jar_of spring-ai-rag)" | grep -ic rerank || true)" + echo + echo "## classes in any jar on this project's classpath whose name contains \"SemanticSearchCache\" or \"SemanticCache\": $(for j in $(tr ':' '\n' < target/classpath.txt); do unzip -Z1 "$j" 2>/dev/null; done | grep -Eic 'SemanticSearchCache|SemanticCache' || true)" + echo + jp org.springframework.ai.transformer.splitter.TokenTextSplitter + jp 'org.springframework.ai.transformer.splitter.TokenTextSplitter$Builder' + jp org.springframework.ai.reader.pdf.PagePdfDocumentReader + jp 'org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig$Builder' + jp org.springframework.ai.rag.postretrieval.document.DocumentPostProcessor + jp org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever + jp 'org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever$Builder' + jp 'org.springframework.ai.rag.generation.augmentation.ContextualQueryAugmenter$Builder' + jp 'org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor$Builder' + jp 'org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor$Builder' + jp org.springframework.ai.chat.evaluation.FactCheckingEvaluator + jp org.springframework.ai.vectorstore.VectorStore + jp 'org.springframework.ai.vectorstore.SearchRequest$Builder' +} > "$OUT" +echo "wrote $OUT" diff --git a/rag/scripts/capture-legacy-compile.sh b/rag/scripts/capture-legacy-compile.sh new file mode 100755 index 0000000..9b1b67c --- /dev/null +++ b/rag/scripts/capture-legacy-compile.sh @@ -0,0 +1,68 @@ +#!/usr/bin/env bash +# Writes docs/output/14-legacy-1x.txt: what happens to the code in the 1.x version of the article. +# Needs network access to Maven Central. It runs deliberately failing builds, so it never exits non-zero +# because of them. Needs unzip and python3 for the configuration-metadata part. +set -uo pipefail +cd "$(dirname "$0")/.." +mkdir -p docs/output target/legacy +OUT=docs/output/14-legacy-1x.txt +CENTRAL=https://repo1.maven.org/maven2/org/springframework/ai + +latest() { curl -s "$CENTRAL/$1/maven-metadata.xml" | grep -o '[^<]*' | sed 's/.*>//'; } +first_errors() { grep -E 'ERROR.*(is missing|Could not|error:)|error:' | sed 's/^\[ERROR\] *//' | sort -u | head -"${1:-6}"; } + +{ + echo "# The 1.x article's code, against 1.1.0 and 2.0.1" + echo + echo "## The starter artifact ids in the 1.x article: latest version ever published" + echo "spring-ai-openai-spring-boot-starter latest: $(latest spring-ai-openai-spring-boot-starter)" + echo "spring-ai-pgvector-store-spring-boot-starter latest: $(latest spring-ai-pgvector-store-spring-boot-starter)" + echo "the ids that replaced them:" + echo "spring-ai-starter-model-openai latest: $(latest spring-ai-starter-model-openai)" + echo "spring-ai-starter-vector-store-pgvector latest: $(latest spring-ai-starter-vector-store-pgvector)" + + for v in 1.1.0 2.0.1; do + echo + echo "## mvn validate on the article's dependency block with spring-ai-bom $v" + mvn -B -f legacy-1x/pom.xml validate -Dspring-ai.version=$v 2>&1 | first_errors 3 + done + + # Compile the two Spring AI calls against the 1.1.0 jars, then against 2.0.1. + for m in spring-ai-commons spring-ai-pdf-document-reader; do + [ -f target/legacy/$m-1.1.0.jar ] || curl -s -o target/legacy/$m-1.1.0.jar "$CENTRAL/$m/1.1.0/$m-1.1.0.jar" + done + mvn -B -q dependency:build-classpath -Dmdep.outputFile=target/classpath.txt >/dev/null 2>&1 + OTHERS="$(tr ':' '\n' < target/classpath.txt | grep -v '/spring-ai-' | paste -sd:)" + CP21="$(cat target/classpath.txt)" + CP11="target/legacy/spring-ai-commons-1.1.0.jar:target/legacy/spring-ai-pdf-document-reader-1.1.0.jar:$OTHERS" + + for v in 1.1.0 2.0.1; do + echo + echo "## javac legacy-1x/src/LegacyIngestion.java against Spring AI $v" + if [ "$v" = 1.1.0 ]; then CP="$CP11"; else CP="$CP21"; fi + rm -rf target/legacy/out && mkdir -p target/legacy/out + if RESULT="$(javac -proc:none -Xmaxerrs 10 -d target/legacy/out -cp "$CP" legacy-1x/src/LegacyIngestion.java 2>&1)"; then + echo "(compiles)" + else + echo "$RESULT" | grep -A2 'error:' | grep -v '^--$' + fi + done + + # The three 1.x configuration keys, read from the configuration metadata inside the auto-configuration jar. + echo + echo "## the 1.x configuration keys in the metadata of spring-ai-autoconfigure-model-openai" + for v in 1.1.0 2.0.1; do + J=target/legacy/spring-ai-autoconfigure-model-openai-$v.jar + [ -f "$J" ] || curl -s -o "$J" "$CENTRAL/spring-ai-autoconfigure-model-openai/$v/spring-ai-autoconfigure-model-openai-$v.jar" + unzip -p "$J" META-INF/spring-configuration-metadata.json | python3 -c ' +import json, sys +v = sys.argv[1] +props = {p["name"]: p for p in json.load(sys.stdin)["properties"]} +for k in ("spring.ai.openai.chat.options.model", "spring.ai.openai.chat.options.temperature", "spring.ai.openai.embedding.options.model"): + p = props.get(k) + state = "unknown" if p is None else ("deprecated, use " + p["deprecation"]["replacement"] if "deprecation" in p else "current") + print("%-6s %-46s %s" % (v, k, state)) +' "$v" + done +} > "$OUT" +echo "wrote $OUT" diff --git a/rag/scripts/pg-up.sh b/rag/scripts/pg-up.sh new file mode 100755 index 0000000..d46f17d --- /dev/null +++ b/rag/scripts/pg-up.sh @@ -0,0 +1,39 @@ +#!/usr/bin/env bash +# Starts a throwaway PostgreSQL with the pgvector extension, with no Docker, and prepares the +# ragdb database. Use it when docker compose is not available. It is what produced the +# transcripts in docs/output/ in this repository's authoring environment. +# +# Requires PostgreSQL 14+ and the pgvector package (Debian/Ubuntu: apt install postgresql-16 postgresql-16-pgvector). +# Environment: PGDATA (default /tmp/pgrag/data), PGPORT (default 5439). +# Afterwards: export RAG_PG_URL=jdbc:postgresql://127.0.0.1:${PGPORT:-5439}/ragdb +set -euo pipefail + +PGDATA="${PGDATA:-/tmp/pgrag/data}" +PGPORT="${PGPORT:-5439}" +SOCKDIR="$(dirname "$PGDATA")" +BIN="$(ls -d /usr/lib/postgresql/*/bin 2>/dev/null | sort -V | tail -1)" +[ -x "$BIN/initdb" ] || { echo "PostgreSQL server binaries not found under /usr/lib/postgresql" >&2; exit 1; } +HERE="$(cd "$(dirname "$0")/.." && pwd)" + +# PostgreSQL refuses to run as root, so hand the work to the postgres user in that case. +as_pg() { if [ "$(id -u)" = 0 ]; then su postgres -c "$*"; else bash -c "$*"; fi; } + +mkdir -p "$SOCKDIR" +[ "$(id -u)" = 0 ] && chown postgres "$SOCKDIR" +if [ ! -d "$PGDATA/base" ]; then + as_pg "'$BIN/initdb' -D '$PGDATA' -A trust >'$SOCKDIR/initdb.log' 2>&1" +fi +if ! as_pg "'$BIN/pg_ctl' -D '$PGDATA' status" >/dev/null 2>&1; then + as_pg "'$BIN/pg_ctl' -D '$PGDATA' -o \"-p $PGPORT -c listen_addresses=127.0.0.1 -c unix_socket_directories=$SOCKDIR\" -l '$SOCKDIR/pg.log' -w start" +fi + +PSQL="$BIN/psql -h $SOCKDIR -p $PGPORT -U postgres -v ON_ERROR_STOP=1 -q" +as_pg "$PSQL -d postgres -tc \"select 1 from pg_roles where rolname='raguser'\"" | grep -q 1 \ + || as_pg "$PSQL -d postgres -c \"create role raguser login superuser password 'ragpass'\"" +as_pg "$PSQL -d postgres -tc \"select 1 from pg_database where datname='ragdb'\"" | grep -q 1 \ + || as_pg "$PSQL -d postgres -c 'create database ragdb owner raguser'" +as_pg "$PSQL -d ragdb -c 'create extension if not exists vector'" +as_pg "$PSQL -d ragdb -f '$HERE/init.sql'" + +echo "PostgreSQL is up on 127.0.0.1:$PGPORT (database ragdb, user raguser, password ragpass)" +echo "export RAG_PG_URL=jdbc:postgresql://127.0.0.1:$PGPORT/ragdb" diff --git a/rag/scripts/run-all.sh b/rag/scripts/run-all.sh new file mode 100755 index 0000000..922a8e0 --- /dev/null +++ b/rag/scripts/run-all.sh @@ -0,0 +1,20 @@ +#!/usr/bin/env bash +# Regenerates everything under docs/output/. +# +# scripts/run-all.sh uses RAG_PG_URL if set, otherwise starts PostgreSQL with scripts/pg-up.sh +# docker compose up -d && RAG_PG_URL=jdbc:postgresql://localhost:5432/ragdb scripts/run-all.sh +# +# No API key is needed: the tests use scripted stand-ins for the chat and embedding models. +set -euo pipefail +cd "$(dirname "$0")/.." + +if [ -z "${RAG_PG_URL:-}" ]; then + scripts/pg-up.sh + export RAG_PG_URL="jdbc:postgresql://127.0.0.1:${PGPORT:-5439}/ragdb" +fi + +mvn -B -q test # transcripts 01-11 and 15 (each test writes one and asserts the same numbers) +scripts/capture-javap.sh # 12 +scripts/capture-dependencies.sh # 13 +scripts/capture-legacy-compile.sh # 14 +ls docs/output diff --git a/rag/src/main/java/com/ankurm/rag/RagApplication.java b/rag/src/main/java/com/ankurm/rag/RagApplication.java new file mode 100644 index 0000000..09c61b5 --- /dev/null +++ b/rag/src/main/java/com/ankurm/rag/RagApplication.java @@ -0,0 +1,18 @@ +package com.ankurm.rag; + +import org.springframework.boot.SpringApplication; +import org.springframework.boot.autoconfigure.SpringBootApplication; +import org.springframework.boot.context.properties.ConfigurationPropertiesScan; + +/** + * Entry point. The pipeline is described in + * chapter 1. + */ +@SpringBootApplication +@ConfigurationPropertiesScan +public class RagApplication { + + public static void main(String[] args) { + SpringApplication.run(RagApplication.class, args); + } +} diff --git a/rag/src/main/java/com/ankurm/rag/chunk/RecursiveChunker.java b/rag/src/main/java/com/ankurm/rag/chunk/RecursiveChunker.java new file mode 100644 index 0000000..e7f1a88 --- /dev/null +++ b/rag/src/main/java/com/ankurm/rag/chunk/RecursiveChunker.java @@ -0,0 +1,98 @@ +package com.ankurm.rag.chunk; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.regex.Pattern; + +import org.springframework.ai.document.Document; +import org.springframework.ai.document.DocumentTransformer; + +/** + * Cuts text at the coarsest boundary that keeps a piece under {@code maxChars}: a blank line first, + * then a line break, then a sentence end, then a space, and only as a last resort in the middle of a + * word. Neighbouring chunks share up to {@code overlapChars} characters, which + * {@link org.springframework.ai.transformer.splitter.TokenTextSplitter} does not offer. + * + *

Sizes are in characters, not tokens. See + * chapter 2. + */ +public class RecursiveChunker implements DocumentTransformer { + + /** Coarsest first. Each separator stays attached to the end of the piece it closes. */ + private static final List SEPARATORS = List.of("\n\n", "\n", ". ", " "); + + private final int maxChars; + private final int overlapChars; + + public RecursiveChunker(int maxChars, int overlapChars) { + if (overlapChars >= maxChars) { + throw new IllegalArgumentException("overlap must be smaller than the chunk size"); + } + this.maxChars = maxChars; + this.overlapChars = overlapChars; + } + + @Override + public List apply(List documents) { + List chunks = new ArrayList<>(); + for (Document document : documents) { + List texts = merge(split(document.getText(), 0)); + for (int i = 0; i < texts.size(); i++) { + Map metadata = new HashMap<>(document.getMetadata()); + metadata.put("chunk_index", i); + metadata.put("chunk_total", texts.size()); + chunks.add(new Document(texts.get(i), metadata)); + } + } + return chunks; + } + + /** Breaks {@code text} into pieces that are each at most {@code maxChars} long. */ + private List split(String text, int level) { + if (text.length() <= maxChars) { + return List.of(text); + } + List pieces = new ArrayList<>(); + if (level >= SEPARATORS.size()) { + for (int start = 0; start < text.length(); start += maxChars) { + pieces.add(text.substring(start, Math.min(text.length(), start + maxChars))); + } + return pieces; + } + String separator = SEPARATORS.get(level); + for (String part : text.split("(?<=" + Pattern.quote(separator) + ")")) { + pieces.addAll(split(part, level + 1)); + } + return pieces; + } + + /** Packs consecutive pieces into chunks, starting each new chunk with the tail of the last one. */ + private List merge(List pieces) { + List chunks = new ArrayList<>(); + StringBuilder current = new StringBuilder(); + for (String piece : pieces) { + if (current.length() + piece.length() > maxChars && !current.isEmpty()) { + chunks.add(current.toString().strip()); + String tail = overlapTail(current, Math.min(overlapChars, maxChars - piece.length())); + current = new StringBuilder(tail); + } + current.append(piece); + } + if (!current.toString().isBlank()) { + chunks.add(current.toString().strip()); + } + return chunks; + } + + /** The last {@code n} characters, moved forward to a word start so a chunk never begins mid-word. */ + private static String overlapTail(CharSequence text, int n) { + if (n <= 0) { + return ""; + } + String tail = text.subSequence(Math.max(0, text.length() - n), text.length()).toString(); + int space = tail.indexOf(' '); + return space < 0 ? "" : tail.substring(space + 1); + } +} diff --git a/rag/src/main/java/com/ankurm/rag/chunk/SemanticChunker.java b/rag/src/main/java/com/ankurm/rag/chunk/SemanticChunker.java new file mode 100644 index 0000000..9d2b2a1 --- /dev/null +++ b/rag/src/main/java/com/ankurm/rag/chunk/SemanticChunker.java @@ -0,0 +1,63 @@ +package com.ankurm.rag.chunk; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import org.springframework.ai.document.Document; +import org.springframework.ai.document.DocumentTransformer; +import org.springframework.ai.embedding.EmbeddingModel; + +/** + * Cuts where the topic changes: every sentence is embedded, and a new chunk starts wherever two + * neighbouring sentences are further apart than {@code maxDistance} (cosine distance, 0 = identical). + * + *

All sentences of a document go to the embedding model in one batched call. Whether the cuts are + * good depends on the embedding model; see + * chapter 2. + */ +public class SemanticChunker implements DocumentTransformer { + + private final EmbeddingModel embeddingModel; + private final double maxDistance; + + public SemanticChunker(EmbeddingModel embeddingModel, double maxDistance) { + this.embeddingModel = embeddingModel; + this.maxDistance = maxDistance; + } + + @Override + public List apply(List documents) { + List chunks = new ArrayList<>(); + for (Document document : documents) { + String[] sentences = document.getText().strip().split("(?<=[.!?])\\s+"); + if (sentences.length == 0) { + continue; + } + List vectors = embeddingModel.embed(List.of(sentences)); + + int start = 0; + for (int i = 1; i <= sentences.length; i++) { + boolean end = i == sentences.length; + if (end || 1.0 - cosine(vectors.get(i - 1), vectors.get(i)) > maxDistance) { + Map metadata = new HashMap<>(document.getMetadata()); + metadata.put("chunk_index", chunks.size()); + chunks.add(new Document(String.join(" ", List.of(sentences).subList(start, i)), metadata)); + start = i; + } + } + } + return chunks; + } + + static double cosine(float[] a, float[] b) { + double dot = 0, normA = 0, normB = 0; + for (int i = 0; i < a.length; i++) { + dot += a[i] * b[i]; + normA += a[i] * a[i]; + normB += b[i] * b[i]; + } + return dot / (Math.sqrt(normA) * Math.sqrt(normB) + 1e-12); + } +} diff --git a/rag/src/main/java/com/ankurm/rag/config/RagConfig.java b/rag/src/main/java/com/ankurm/rag/config/RagConfig.java new file mode 100644 index 0000000..bb5d767 --- /dev/null +++ b/rag/src/main/java/com/ankurm/rag/config/RagConfig.java @@ -0,0 +1,61 @@ +package com.ankurm.rag.config; + +import com.ankurm.rag.query.LlmReranker; +import io.micrometer.core.instrument.MeterRegistry; +import org.springframework.ai.chat.client.ChatClient; +import org.springframework.ai.chat.evaluation.FactCheckingEvaluator; +import org.springframework.ai.document.DocumentTransformer; +import org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor; +import org.springframework.ai.rag.generation.augmentation.ContextualQueryAugmenter; +import org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever; +import org.springframework.ai.transformer.splitter.TokenTextSplitter; +import org.springframework.ai.vectorstore.VectorStore; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; + +/** Wires the pipeline. Each bean is one stage; replace one and the others do not notice. */ +@Configuration +public class RagConfig { + + /** Stage 1, chunking. Swap in {@code RecursiveChunker} or {@code SemanticChunker} here. */ + @Bean + DocumentTransformer chunker(RagProperties properties) { + return TokenTextSplitter.builder() + .withChunkSize(properties.chunking().chunkSize()) + .withMinChunkSizeChars(properties.chunking().minChunkSizeChars()) + .build(); + } + + /** Stage 3, reranking. Built from the builder so it gets its own client, not the RAG one. */ + @Bean + LlmReranker reranker(ChatClient.Builder builder, RagProperties properties, MeterRegistry metrics) { + return new LlmReranker(builder.build(), properties.rerank().topN(), metrics); + } + + /** Stages 2 to 4: retrieve, rerank, and put the survivors into the prompt. */ + @Bean + RetrievalAugmentationAdvisor retrievalAdvisor(VectorStore vectorStore, LlmReranker reranker, + RagProperties properties) { + return RetrievalAugmentationAdvisor.builder() + .documentRetriever(VectorStoreDocumentRetriever.builder() + .vectorStore(vectorStore) + .topK(properties.retrieval().topK()) + .similarityThreshold(properties.retrieval().similarityThreshold()) + .build()) + .documentPostProcessors(reranker) + .queryAugmenter(ContextualQueryAugmenter.builder().allowEmptyContext(false).build()) + .build(); + } + + /** The chat client that answers questions. The advisor runs on every call it makes. */ + @Bean + ChatClient ragChatClient(ChatClient.Builder builder, RetrievalAugmentationAdvisor advisor) { + return builder.defaultAdvisors(advisor).build(); + } + + /** The judge for the faithfulness check. */ + @Bean + FactCheckingEvaluator factChecker(ChatClient.Builder builder) { + return FactCheckingEvaluator.builder(builder).build(); + } +} diff --git a/rag/src/main/java/com/ankurm/rag/config/RagProperties.java b/rag/src/main/java/com/ankurm/rag/config/RagProperties.java new file mode 100644 index 0000000..3aca6d6 --- /dev/null +++ b/rag/src/main/java/com/ankurm/rag/config/RagProperties.java @@ -0,0 +1,27 @@ +package com.ankurm.rag.config; + +import org.springframework.boot.context.properties.ConfigurationProperties; +import org.springframework.boot.context.properties.bind.DefaultValue; + +/** + * The numbers you tune. Every default is a starting point, not a measured optimum: this repository + * has no relevance benchmark, so nothing here claims to be the best value for your documents. + */ +@ConfigurationProperties("rag") +public record RagProperties( + @DefaultValue Retrieval retrieval, + @DefaultValue Rerank rerank, + @DefaultValue Chunking chunking) { + + /** How wide the first search is, and how similar a chunk must be to be a candidate at all. */ + public record Retrieval(@DefaultValue("20") int topK, @DefaultValue("0.0") double similarityThreshold) { + } + + /** How many chunks survive reranking and reach the prompt. */ + public record Rerank(@DefaultValue("5") int topN) { + } + + /** Settings for {@link org.springframework.ai.transformer.splitter.TokenTextSplitter}. */ + public record Chunking(@DefaultValue("512") int chunkSize, @DefaultValue("350") int minChunkSizeChars) { + } +} diff --git a/rag/src/main/java/com/ankurm/rag/ingest/IngestionResult.java b/rag/src/main/java/com/ankurm/rag/ingest/IngestionResult.java new file mode 100644 index 0000000..d89a499 --- /dev/null +++ b/rag/src/main/java/com/ankurm/rag/ingest/IngestionResult.java @@ -0,0 +1,17 @@ +package com.ankurm.rag.ingest; + +/** What {@link IngestionService} did with one file. */ +public record IngestionResult(String filename, String status, int chunksWritten, int chunksReplaced) { + + public static IngestionResult ingested(String filename, int chunks) { + return new IngestionResult(filename, "ingested", chunks, 0); + } + + public static IngestionResult updated(String filename, int chunks, int replaced) { + return new IngestionResult(filename, "updated", chunks, replaced); + } + + public static IngestionResult skipped(String filename) { + return new IngestionResult(filename, "skipped", 0, 0); + } +} diff --git a/rag/src/main/java/com/ankurm/rag/ingest/IngestionService.java b/rag/src/main/java/com/ankurm/rag/ingest/IngestionService.java new file mode 100644 index 0000000..9ff975f --- /dev/null +++ b/rag/src/main/java/com/ankurm/rag/ingest/IngestionService.java @@ -0,0 +1,103 @@ +package com.ankurm.rag.ingest; + +import java.io.IOException; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; +import java.util.HashMap; +import java.util.HexFormat; +import java.util.List; +import java.util.Map; + +import io.micrometer.core.instrument.MeterRegistry; +import org.springframework.ai.document.Document; +import org.springframework.ai.document.DocumentTransformer; +import org.springframework.ai.reader.pdf.PagePdfDocumentReader; +import org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig; +import org.springframework.ai.vectorstore.VectorStore; +import org.springframework.core.io.Resource; +import org.springframework.stereotype.Service; + +/** + * Reads a PDF page by page, tags every page with where it came from, cuts pages into chunks and + * writes the chunks to the vector store. + * + *

Ingesting the same bytes twice does nothing. Ingesting a changed file deletes the chunks the + * old version produced before adding the new ones. See + * chapter 3. + */ +@Service +public class IngestionService { + + /** Metadata key carrying the file a chunk came from; the delete-by-source filter depends on it. */ + public static final String SOURCE_FILE = "source_file"; + + private final VectorStore vectorStore; + private final DocumentTransformer chunker; + private final IngestionTracker tracker; + private final MeterRegistry metrics; + + public IngestionService(VectorStore vectorStore, DocumentTransformer chunker, + IngestionTracker tracker, MeterRegistry metrics) { + this.vectorStore = vectorStore; + this.chunker = chunker; + this.tracker = tracker; + this.metrics = metrics; + } + + public IngestionResult ingestPdf(Resource pdf, String filename, Map callerMetadata) { + String hash = sha256(pdf); + var known = tracker.find(filename); + if (known.isPresent() && known.get().hash().equals(hash)) { + metrics.counter("rag.ingestion.skipped").increment(); + return IngestionResult.skipped(filename); + } + + List pages = new PagePdfDocumentReader(pdf, PdfDocumentReaderConfig.builder() + .withPagesPerDocument(1) + .build()).get(); + + List tagged = pages.stream().map(page -> { + Map metadata = new HashMap<>(page.getMetadata()); + metadata.put(SOURCE_FILE, filename); + metadata.put("source_hash", hash); + metadata.putAll(callerMetadata); + return new Document(tidy(page.getText()), metadata); + }).toList(); + + List chunks = chunker.apply(tagged); + + int replaced = 0; + if (known.isPresent()) { + // The file changed: remove what the old version wrote, by id, before adding the new chunks. + replaced = known.get().chunkIds().size(); + vectorStore.delete(known.get().chunkIds()); + } + vectorStore.add(chunks); + tracker.record(filename, hash, chunks.stream().map(Document::getId).toList()); + + metrics.counter("rag.chunks.ingested").increment(chunks.size()); + return known.isPresent() + ? IngestionResult.updated(filename, chunks.size(), replaced) + : IngestionResult.ingested(filename, chunks.size()); + } + + /** + * The PDF reader pads text with runs of spaces to reproduce the page layout. Those runs cost tokens + * and change what gets embedded, so collapse them and the blank-line stacks around them. + */ + public static String tidy(String text) { + return text.replaceAll("[ \\t]+", " ") + .replaceAll(" ?\\n ?", "\n") + .replaceAll("\\n{3,}", "\n\n") + .strip(); + } + + private static String sha256(Resource resource) { + try { + return HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256") + .digest(resource.getInputStream().readAllBytes())); + } catch (IOException | NoSuchAlgorithmException e) { + throw new IllegalStateException("cannot hash " + resource, e); + } + } +} diff --git a/rag/src/main/java/com/ankurm/rag/ingest/IngestionTracker.java b/rag/src/main/java/com/ankurm/rag/ingest/IngestionTracker.java new file mode 100644 index 0000000..5fb60e4 --- /dev/null +++ b/rag/src/main/java/com/ankurm/rag/ingest/IngestionTracker.java @@ -0,0 +1,33 @@ +package com.ankurm.rag.ingest; + +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.concurrent.ConcurrentHashMap; + +import org.springframework.stereotype.Component; + +/** + * Remembers, per file name, the SHA-256 of the bytes that were ingested and the ids of the chunks + * they produced. + * + *

It lives in memory, so a restart forgets everything and the next start re-embeds every file. + * Back it with a table if that matters to you; the chunk ids are what make a targeted delete + * possible when a file changes. + */ +@Component +public class IngestionTracker { + + public record Entry(String hash, List chunkIds) { + } + + private final Map entries = new ConcurrentHashMap<>(); + + public Optional find(String filename) { + return Optional.ofNullable(entries.get(filename)); + } + + public void record(String filename, String hash, List chunkIds) { + entries.put(filename, new Entry(hash, List.copyOf(chunkIds))); + } +} diff --git a/rag/src/main/java/com/ankurm/rag/query/LlmReranker.java b/rag/src/main/java/com/ankurm/rag/query/LlmReranker.java new file mode 100644 index 0000000..7dfe705 --- /dev/null +++ b/rag/src/main/java/com/ankurm/rag/query/LlmReranker.java @@ -0,0 +1,80 @@ +package com.ankurm.rag.query; + +import java.util.Comparator; +import java.util.List; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; + +import io.micrometer.core.instrument.MeterRegistry; +import org.springframework.ai.chat.client.ChatClient; +import org.springframework.ai.document.Document; +import org.springframework.ai.rag.Query; +import org.springframework.ai.rag.postretrieval.document.DocumentPostProcessor; + +/** + * Asks a chat model to rate each candidate chunk from 0 to 10 for the question, keeps the best + * {@code topN}, and returns them best first. + * + *

Spring AI ships the hook ({@link DocumentPostProcessor}) but no reranker, so this is ours. It + * costs one model call per candidate. The calls run on virtual threads so they wait together rather + * than one after another. See chapter 4. + */ +public class LlmReranker implements DocumentPostProcessor { + + /** Metadata key holding the score the model gave. */ + public static final String SCORE = "rerank_score"; + + private static final String PROMPT = """ + Rate how well the PASSAGE helps answer the QUESTION, from 0 (irrelevant) to 10 (answers it). + Reply with a single integer and nothing else. + QUESTION: {question} + PASSAGE: {passage} + SCORE:"""; + + private final ChatClient client; + private final int topN; + private final MeterRegistry metrics; + + public LlmReranker(ChatClient client, int topN, MeterRegistry metrics) { + this.client = client; + this.topN = topN; + this.metrics = metrics; + } + + @Override + public List process(Query query, List candidates) { + try (var executor = Executors.newVirtualThreadPerTaskExecutor()) { + List> futures = candidates.stream() + .map(candidate -> executor.submit(() -> withScore(query.text(), candidate))) + .toList(); + return futures.stream() + .map(LlmReranker::join) + .sorted(Comparator.comparingInt((Document d) -> (int) d.getMetadata().get(SCORE)).reversed()) + .limit(topN) + .toList(); + } + } + + private Document withScore(String question, Document candidate) { + int score = 0; + try { + String reply = client.prompt() + .user(u -> u.text(PROMPT).param("question", question).param("passage", candidate.getText())) + .call().content(); + score = Math.max(0, Math.min(10, Integer.parseInt(reply.strip()))); + } catch (RuntimeException e) { + // An unparseable or failed rating counts as 0 rather than failing the whole question. + metrics.counter("rag.rerank.failures").increment(); + } + return candidate.mutate().metadata(SCORE, score).build(); + } + + private static Document join(Future future) { + try { + return future.get(); + } catch (InterruptedException | ExecutionException e) { + throw new IllegalStateException(e); + } + } +} diff --git a/rag/src/main/java/com/ankurm/rag/query/RagQueryService.java b/rag/src/main/java/com/ankurm/rag/query/RagQueryService.java new file mode 100644 index 0000000..2167263 --- /dev/null +++ b/rag/src/main/java/com/ankurm/rag/query/RagQueryService.java @@ -0,0 +1,90 @@ +package com.ankurm.rag.query; + +import java.util.List; + +import io.micrometer.core.instrument.MeterRegistry; +import io.micrometer.core.instrument.Timer; +import org.springframework.ai.chat.client.ChatClient; +import org.springframework.ai.chat.client.ChatClientResponse; +import org.springframework.ai.chat.evaluation.FactCheckingEvaluator; +import org.springframework.ai.document.Document; +import org.springframework.ai.evaluation.EvaluationRequest; +import org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor; +import org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever; +import org.springframework.ai.vectorstore.filter.Filter; +import org.springframework.stereotype.Service; + +/** + * Asks one question: retrieve, rerank and augment (all inside the {@link RetrievalAugmentationAdvisor} + * the chat client was built with), generate, then check the answer against the chunks it was given. + * + *

See chapter 5. + */ +@Service +public class RagQueryService { + + private final ChatClient chatClient; + private final FactCheckingEvaluator factChecker; + private final MeterRegistry metrics; + + public RagQueryService(ChatClient ragChatClient, FactCheckingEvaluator factChecker, MeterRegistry metrics) { + this.chatClient = ragChatClient; + this.factChecker = factChecker; + this.metrics = metrics; + } + + /** @param filter restricts the search, for example to one tenant; {@code null} searches everything */ + public RagResponse ask(String question, Filter.Expression filter) { + Timer.Sample sample = Timer.start(metrics); + String status = "error"; + try { + ChatClientResponse response = chatClient.prompt() + .user(question) + .advisors(a -> { + if (filter != null) { + a.param(VectorStoreDocumentRetriever.FILTER_EXPRESSION, filter); + } + }) + .call() + .chatClientResponse(); + + String answer = response.chatResponse().getResult().getOutput().getText(); + @SuppressWarnings("unchecked") + List context = (List) response.context() + .getOrDefault(RetrievalAugmentationAdvisor.DOCUMENT_CONTEXT, List.of()); + metrics.summary("rag.context.chunks").record(context.size()); + + List sources = context.stream().map(RagQueryService::toSource).toList(); + if (context.isEmpty()) { + status = "no_context"; + return new RagResponse(answer, sources, false, status); + } + boolean grounded = isGrounded(question, answer, context); + status = grounded ? "answered" : "ungrounded"; + return new RagResponse(answer, sources, grounded, status); + } finally { + sample.stop(metrics.timer("rag.query.duration", "status", status)); + metrics.counter("rag.queries", "status", status).increment(); + } + } + + private boolean isGrounded(String question, String answer, List context) { + try { + return factChecker.evaluate(new EvaluationRequest(question, context, answer)).isPass(); + } catch (RuntimeException e) { + // If the judge itself fails, the answer is unchecked, which is not the same as supported. + metrics.counter("rag.faithfulness.judge_failures").increment(); + return false; + } + } + + private static RagResponse.Source toSource(Document d) { + String text = d.getText(); + Object score = d.getMetadata().get(LlmReranker.SCORE); + return new RagResponse.Source( + String.valueOf(d.getMetadata().getOrDefault("source_file", "unknown")), + d.getMetadata().get("page_number"), + score instanceof Integer i ? i : -1, + text.substring(0, Math.min(80, text.length()))); + } +} diff --git a/rag/src/main/java/com/ankurm/rag/query/RagResponse.java b/rag/src/main/java/com/ankurm/rag/query/RagResponse.java new file mode 100644 index 0000000..a1963e2 --- /dev/null +++ b/rag/src/main/java/com/ankurm/rag/query/RagResponse.java @@ -0,0 +1,14 @@ +package com.ankurm.rag.query; + +import java.util.List; + +/** + * The answer, where it came from, and whether the guard found it supported by that context. + * {@code status} is {@code answered}, {@code ungrounded} or {@code no_context}. + */ +public record RagResponse(String answer, List sources, boolean grounded, String status) { + + /** One chunk that reached the prompt. {@code rerankScore} is the 0-10 rating, or -1 if not reranked. */ + public record Source(String file, Object page, int rerankScore, String preview) { + } +} diff --git a/rag/src/main/java/com/ankurm/rag/web/RagController.java b/rag/src/main/java/com/ankurm/rag/web/RagController.java new file mode 100644 index 0000000..2674a2e --- /dev/null +++ b/rag/src/main/java/com/ankurm/rag/web/RagController.java @@ -0,0 +1,50 @@ +package com.ankurm.rag.web; + +import java.io.IOException; +import java.util.Map; + +import com.ankurm.rag.ingest.IngestionResult; +import com.ankurm.rag.ingest.IngestionService; +import com.ankurm.rag.query.RagQueryService; +import com.ankurm.rag.query.RagResponse; +import org.springframework.ai.vectorstore.filter.FilterExpressionBuilder; +import org.springframework.core.io.ByteArrayResource; +import org.springframework.web.bind.annotation.PostMapping; +import org.springframework.web.bind.annotation.RequestBody; +import org.springframework.web.bind.annotation.RequestMapping; +import org.springframework.web.bind.annotation.RequestParam; +import org.springframework.web.bind.annotation.RestController; +import org.springframework.web.multipart.MultipartFile; + +/** Two endpoints: upload a PDF, ask a question. */ +@RestController +@RequestMapping("/api") +public class RagController { + + public record QueryRequest(String question, String tenantId) { + } + + private final IngestionService ingestion; + private final RagQueryService queries; + + public RagController(IngestionService ingestion, RagQueryService queries) { + this.ingestion = ingestion; + this.queries = queries; + } + + @PostMapping("/ingest") + public IngestionResult ingest(@RequestParam("file") MultipartFile file, + @RequestParam(defaultValue = "default") String tenantId, + @RequestParam(defaultValue = "general") String docType) throws IOException { + return ingestion.ingestPdf(new ByteArrayResource(file.getBytes()), file.getOriginalFilename(), + Map.of("tenant_id", tenantId, "doc_type", docType)); + } + + @PostMapping("/query") + public RagResponse query(@RequestBody QueryRequest request) { + // The tenant goes in as a value of an expression, never as text glued into a filter string. + var filter = request.tenantId() == null ? null : new FilterExpressionBuilder() + .eq("tenant_id", request.tenantId()).build(); + return queries.ask(request.question(), filter); + } +} diff --git a/rag/src/main/resources/application.yml b/rag/src/main/resources/application.yml new file mode 100644 index 0000000..ee090ab --- /dev/null +++ b/rag/src/main/resources/application.yml @@ -0,0 +1,46 @@ +spring: + application: + name: spring-ai-rag + threads: + virtual: + enabled: true + datasource: + url: ${RAG_PG_URL:jdbc:postgresql://localhost:5432/ragdb} + username: ${RAG_PG_USER:raguser} + password: ${RAG_PG_PASSWORD:ragpass} + ai: + openai: + api-key: ${OPENAI_API_KEY} + # In 2.0.1 the "options" level (chat.options.model) is marked deprecated in the jars' metadata; the + # settings now sit directly under chat: and embedding:. Spring Boot ignores keys it does not know, + # so a misspelt key fails nothing: ConfigKeysTest checks every spring.ai key against the jars. + chat: + model: gpt-4o + temperature: 0.1 + embedding: + model: text-embedding-3-small + vectorstore: + pgvector: + schema-name: rag + table-name: document_chunks + index-type: HNSW + distance-type: COSINE_DISTANCE + dimensions: 1536 + # false: the schema comes from init.sql, not from the application. + initialize-schema: false + +rag: + retrieval: + top-k: 20 + similarity-threshold: 0.0 + rerank: + top-n: 5 + chunking: + chunk-size: 512 + min-chunk-size-chars: 350 + +management: + endpoints: + web: + exposure: + include: health, prometheus diff --git a/rag/src/test/java/com/ankurm/rag/ChunkingTest.java b/rag/src/test/java/com/ankurm/rag/ChunkingTest.java new file mode 100644 index 0000000..e8c5cf1 --- /dev/null +++ b/rag/src/test/java/com/ankurm/rag/ChunkingTest.java @@ -0,0 +1,158 @@ +package com.ankurm.rag; + +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +import com.ankurm.rag.chunk.RecursiveChunker; +import com.ankurm.rag.chunk.SemanticChunker; +import com.ankurm.rag.support.HashingEmbeddingModel; +import com.ankurm.rag.support.Transcript; +import com.knuddels.jtokkit.Encodings; +import com.knuddels.jtokkit.api.Encoding; +import com.knuddels.jtokkit.api.EncodingType; +import org.junit.jupiter.api.Test; +import org.springframework.ai.document.Document; +import org.springframework.ai.transformer.splitter.TokenTextSplitter; + +import static org.assertj.core.api.Assertions.assertThat; + +/** Chapter 2: what the chunkers do to a document, measured. Transcripts 02 and 03. */ +class ChunkingTest { + + private static final Encoding CL100K = Encodings.newDefaultEncodingRegistry().getEncoding(EncodingType.CL100K_BASE); + + /** Forty numbered sentences, about 880 tokens in all, so a document has a known size. */ + private static String longText() { + StringBuilder sb = new StringBuilder(); + for (int i = 1; i <= 40; i++) { + sb.append("Sentence ").append(i).append(" describes the policy for item ").append(i) + .append(" and states that the rule applies to every full-time employee."); + sb.append(i % 5 == 0 ? "\n\n" : " "); + } + return sb.toString().strip(); + } + + private static Document doc(String text) { + return new Document(text, Map.of("source_file", "long.txt", "page_number", 1)); + } + + private static int tokens(String text) { + return CL100K.countTokens(text); + } + + @Test + void tokenTextSplitter() { + String text = longText(); + try (Transcript t = new Transcript("02-token-text-splitter.txt", "What TokenTextSplitter does to a 40-sentence document")) { + t.line("document: %d characters, %d cl100k_base tokens", text.length(), tokens(text)); + + t.section("new TokenTextSplitter() (defaults: 800 tokens, 350 min chars, 5 min length to embed)"); + List byDefault = new TokenTextSplitter().apply(List.of(doc(text))); + t.line("chunks: %d, sizes in tokens: %s", byDefault.size(), + byDefault.stream().map(c -> tokens(c.getText())).toList()); + assertThat(byDefault).hasSize(2); + assertThat(tokens(byDefault.get(0).getText())).isLessThanOrEqualTo(800); + + t.section("chunk size 100 tokens"); + TokenTextSplitter small = TokenTextSplitter.builder().withChunkSize(100).build(); + List chunks = small.apply(List.of(doc(text))); + t.line("chunks: %d", chunks.size()); + for (int i = 0; i < chunks.size(); i++) { + String c = chunks.get(i).getText(); + t.line("chunk %d: %3d tokens, %4d chars, starts \"%s\", ends \"%s\"", i, tokens(c), c.length(), + c.substring(0, Math.min(22, c.length())).replace("\n", "\\n"), + c.substring(Math.max(0, c.length() - 22)).replace("\n", "\\n")); + } + var chunkMetadata = new java.util.TreeMap<>(chunks.get(1).getMetadata()); + chunkMetadata.remove("parent_document_id"); // a random UUID, different on every run + t.line("metadata of chunk 1 (parent_document_id, a random UUID, left out): %s", chunkMetadata); + + t.section("is there any overlap between neighbouring chunks?"); + int shared = 0; + for (int i = 1; i < chunks.size(); i++) { + String opening = chunks.get(i).getText().substring(0, 30); + if (chunks.get(i - 1).getText().contains(opening)) { + shared++; + } + } + t.line("boundaries where the first 30 characters of a chunk already appear in the chunk before it: %d of %d", shared, chunks.size() - 1); + assertThat(shared).isZero(); + assertThat(chunks.stream().mapToInt(c -> tokens(c.getText())).max().orElseThrow()).isLessThanOrEqualTo(100); + + t.section("minChunkSizeChars: where the cut lands"); + for (int minChars : new int[] {350, 50}) { + List cut = TokenTextSplitter.builder().withChunkSize(100).withMinChunkSizeChars(minChars).build() + .apply(List.of(doc(text))); + long atSentenceEnd = cut.stream().limit(cut.size() - 1L) + .filter(c -> c.getText().stripTrailing().endsWith(".")).count(); + t.line("minChunkSizeChars=%3d: %d chunks, %d of the first %d end on a full stop", + minChars, cut.size(), atSentenceEnd, cut.size() - 1); + } + } + } + + @Test + void recursiveChunker() { + String text = longText(); + try (Transcript t = new Transcript("03-recursive-and-semantic-chunkers.txt", "RecursiveChunker and SemanticChunker")) { + t.section("RecursiveChunker(maxChars=400, overlapChars=80)"); + List chunks = new RecursiveChunker(400, 80).apply(List.of(doc(text))); + t.line("chunks: %d, longest: %d characters", chunks.size(), chunks.stream().mapToInt(c -> c.getText().length()).max().orElse(0)); + for (int i = 0; i < chunks.size(); i++) { + String c = chunks.get(i).getText(); + t.line("chunk %d: %3d chars, starts \"%s\", ends \"%s\"", i, c.length(), + c.substring(0, 24).replace("\n", "\\n"), c.substring(c.length() - 24).replace("\n", "\\n")); + } + t.line("metadata of chunk 2: %s", new java.util.TreeMap<>(chunks.get(2).getMetadata())); + + assertThat(chunks).allSatisfy(c -> assertThat(c.getText().length()).isLessThanOrEqualTo(400)); + int overlapping = 0; + for (int i = 1; i < chunks.size(); i++) { + String[] words = chunks.get(i).getText().split(" "); + String firstWords = String.join(" ", List.of(words).subList(0, 3)); + if (chunks.get(i - 1).getText().contains(firstWords)) { + overlapping++; + } + } + t.line("boundaries where the next chunk opens with words the previous one ended with: %d of %d", overlapping, chunks.size() - 1); + assertThat(overlapping).isGreaterThan(0); + + t.section("one 900-character word-salad with no separators falls back to a hard cut"); + List hard = new RecursiveChunker(400, 0).apply(List.of(doc("x".repeat(900)))); + t.line("chunks: %s", hard.stream().map(c -> c.getText().length()).toList()); + assertThat(hard).hasSize(3); + + t.section("SemanticChunker(distance 0.9) on three topics, four sentences each"); + String topics = String.join(" ", + "Annual leave is twenty days per year.", "Unused annual leave carries over until March.", + "Leave requests go through the HR portal.", "Annual leave accrues monthly.", + "Expenses need a receipt above fifty euros.", "Expense claims must be filed within thirty days.", + "Receipts for expenses are uploaded as photos.", "Approved expenses are paid with salary.", + "Remote work is allowed two days per week.", "Managers agree the remote work days.", + "Remote work needs a quiet workspace.", "Remote work days are recorded in the calendar."); + HashingEmbeddingModel embeddings = new HashingEmbeddingModel(256); + List semantic = new SemanticChunker(embeddings, 0.9).apply(List.of(doc(topics))); + t.line("chunks: %d", semantic.size()); + for (Document c : semantic) { + t.line(" [%s] %s", c.getMetadata().get("chunk_index"), c.getText()); + } + t.line("texts sent to the embedding model: %d (12 sentences, one batched call)", embeddings.textsEmbedded()); + assertThat(semantic).hasSize(3); + assertThat(semantic.get(0).getText()).startsWith("Annual leave is twenty days"); + assertThat(semantic.get(1).getText()).startsWith("Expenses need a receipt"); + assertThat(semantic.get(2).getText()).startsWith("Remote work is allowed"); + } + } + + /** The same fixture, kept so the numbers in transcript 03 do not depend on test order. */ + @Test + void chunkersKeepMetadata() { + List chunks = new RecursiveChunker(400, 80).apply(List.of(doc(longText()))); + assertThat(chunks).allSatisfy(c -> { + assertThat(c.getMetadata()).containsEntry("source_file", "long.txt").containsEntry("page_number", 1); + assertThat(c.getMetadata()).containsKeys("chunk_index", "chunk_total"); + }); + assertThat(new ArrayList<>(chunks).size()).isEqualTo((int) chunks.get(0).getMetadata().get("chunk_total")); + } +} diff --git a/rag/src/test/java/com/ankurm/rag/ConfigKeysTest.java b/rag/src/test/java/com/ankurm/rag/ConfigKeysTest.java new file mode 100644 index 0000000..bc421d9 --- /dev/null +++ b/rag/src/test/java/com/ankurm/rag/ConfigKeysTest.java @@ -0,0 +1,85 @@ +package com.ankurm.rag; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.TreeMap; + +import com.ankurm.rag.support.Transcript; +import org.junit.jupiter.api.Test; +import org.springframework.beans.factory.config.YamlPropertiesFactoryBean; +import org.springframework.core.io.ClassPathResource; +import org.springframework.core.io.Resource; +import org.springframework.core.io.support.PathMatchingResourcePatternResolver; +import tools.jackson.databind.JsonNode; +import tools.jackson.databind.json.JsonMapper; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * Spring Boot ignores configuration keys it does not know, so a wrong key fails neither the build + * nor the start-up: the setting is just never applied. This test checks every {@code spring.ai.*} + * key in the shipped {@code application.yml} against the configuration metadata inside the jars on + * the classpath, and fails on a key that is unknown or deprecated. Transcript 15. + */ +class ConfigKeysTest { + + /** name to the replacement it is deprecated in favour of, or "" when the name is current. */ + private static Map metadata() throws IOException { + JsonMapper mapper = JsonMapper.builder().build(); + Map names = new TreeMap<>(); + for (Resource resource : new PathMatchingResourcePatternResolver() + .getResources("classpath*:META-INF/spring-configuration-metadata.json")) { + JsonNode root = mapper.readTree(resource.getInputStream().readAllBytes()); + for (JsonNode property : root.get("properties")) { + JsonNode deprecation = property.get("deprecation"); + String replacement = deprecation == null || deprecation.get("replacement") == null + ? "(no replacement named)" : deprecation.get("replacement").stringValue(); + boolean deprecated = property.get("deprecated") != null && property.get("deprecated").asBoolean(); + names.merge(property.get("name").stringValue(), deprecated ? replacement : "", (a, b) -> a.isEmpty() ? a : b); + } + } + return names; + } + + private static String status(Map known, String key) { + if (known.containsKey(key)) { + String replacement = known.get(key); + return replacement.isEmpty() ? "current" : "DEPRECATED, use " + replacement; + } + boolean underMap = known.keySet().stream() + .anyMatch(name -> (name.endsWith("custom-headers") || name.endsWith("extra-body")) && key.startsWith(name + ".")); + return underMap ? "current" : "UNKNOWN"; + } + + @Test + void everySpringAiKeyIsCurrent() throws IOException { + Map known = metadata(); + YamlPropertiesFactoryBean yaml = new YamlPropertiesFactoryBean(); + yaml.setResources(new ClassPathResource("application.yml")); + + List problems = new ArrayList<>(); + try (Transcript t = new Transcript("15-config-keys.txt", "spring.ai.* keys in application.yml against the jars' configuration metadata")) { + t.line("property names in the jars' metadata: %d", known.size()); + t.section("keys in the shipped application.yml"); + for (String key : yaml.getObject().keySet().stream().map(Object::toString).sorted().toList()) { + if (key.startsWith("spring.ai.")) { + String status = status(known, key); + t.line("%-52s %s", key, status); + if (!status.equals("current")) { + problems.add(key + " " + status); + } + } + } + t.section("keys written the 1.x way"); + for (String key : List.of("spring.ai.openai.chat.options.model", "spring.ai.openai.chat.options.temperature", + "spring.ai.openai.embedding.options.model", "spring.ai.openai.chat.optoins.model")) { + t.line("%-52s %s", key, status(known, key)); + } + assertThat(status(known, "spring.ai.openai.chat.options.model")).startsWith("DEPRECATED"); + assertThat(status(known, "spring.ai.openai.chat.optoins.model")).isEqualTo("UNKNOWN"); + } + assertThat(problems).isEmpty(); + } +} diff --git a/rag/src/test/java/com/ankurm/rag/EndToEndTest.java b/rag/src/test/java/com/ankurm/rag/EndToEndTest.java new file mode 100644 index 0000000..204ab57 --- /dev/null +++ b/rag/src/test/java/com/ankurm/rag/EndToEndTest.java @@ -0,0 +1,171 @@ +package com.ankurm.rag; + +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; +import java.util.Map; + +import com.ankurm.rag.support.FakeChatModel; +import com.ankurm.rag.support.HashingEmbeddingModel; +import com.ankurm.rag.support.SamplePdf; +import com.ankurm.rag.support.Stores; +import com.ankurm.rag.support.Transcript; +import org.junit.jupiter.api.Test; +import com.ankurm.rag.ingest.IngestionService; +import com.ankurm.rag.query.LlmReranker; +import com.ankurm.rag.query.RagQueryService; +import org.springframework.ai.chat.client.ChatClient; +import org.springframework.ai.chat.evaluation.FactCheckingEvaluator; +import org.springframework.ai.chat.model.ChatModel; +import org.springframework.ai.document.DocumentTransformer; +import org.springframework.ai.embedding.EmbeddingModel; +import org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor; +import org.springframework.ai.vectorstore.VectorStore; +import org.springframework.ai.vectorstore.pgvector.PgVectorStore; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.context.ApplicationContext; +import org.springframework.boot.test.context.SpringBootTest; +import org.springframework.boot.test.context.TestConfiguration; +import org.springframework.boot.test.web.server.LocalServerPort; +import org.springframework.context.annotation.Bean; +import org.springframework.core.io.ByteArrayResource; +import org.springframework.core.io.Resource; +import org.springframework.http.MediaType; +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.test.context.DynamicPropertyRegistry; +import org.springframework.test.context.DynamicPropertySource; +import org.springframework.util.LinkedMultiValueMap; +import org.springframework.web.client.RestClient; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * The whole application, started for real: the shipped {@code application.yml}, the {@code init.sql} + * schema in a real PostgreSQL with pgvector, real HTTP requests to a real port, real PDFs. Only the + * two model beans are replaced, by the scripted fakes, so it runs with no API key. Transcript 10. + */ +@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT, properties = { + "spring.ai.model.chat=none", + "spring.ai.model.embedding=none", + "spring.ai.openai.api-key=not-used-in-tests" +}) +class EndToEndTest { + + @TestConfiguration + static class Fakes { + @Bean + FakeChatModel chatModel() { + return new FakeChatModel(); + } + + @Bean + HashingEmbeddingModel embeddingModel() { + return new HashingEmbeddingModel(Stores.DIMS); + } + } + + @DynamicPropertySource + static void database(DynamicPropertyRegistry registry) throws Exception { + JdbcTemplate jdbc = Stores.jdbc(); + jdbc.execute("DROP SCHEMA IF EXISTS rag CASCADE"); + jdbc.execute(Files.readString(Path.of("init.sql"))); + registry.add("spring.datasource.url", () -> Stores.PG_URL); + registry.add("spring.datasource.username", () -> Stores.PG_USER); + registry.add("spring.datasource.password", () -> Stores.PG_PASSWORD); + } + + @LocalServerPort + int port; + + @Autowired + JdbcTemplate jdbc; + + @Autowired + ApplicationContext context; + + @Test + void pipelineBeans() { + try (Transcript t = new Transcript("01-pipeline-beans.txt", "The pipeline as Spring beans (the shipped configuration, fake models)")) { + t.line("%-28s %-28s %s", "stage / role", "bean name", "actual class"); + List> types = List.of(DocumentTransformer.class, EmbeddingModel.class, VectorStore.class, + RetrievalAugmentationAdvisor.class, LlmReranker.class, ChatModel.class, ChatClient.class, + FactCheckingEvaluator.class, IngestionService.class, RagQueryService.class); + for (Class type : types) { + for (String name : context.getBeanNamesForType(type)) { + t.line("%-28s %-28s %s", type.getSimpleName(), name, context.getType(name).getSimpleName()); + } + } + assertThat(context.getBeanNamesForType(RetrievalAugmentationAdvisor.class)).hasSize(1); + assertThat(context.getBean(VectorStore.class)).isInstanceOf(PgVectorStore.class); + } + } + + @Test + void uploadAskAndScrape() { + RestClient http = RestClient.create("http://localhost:" + port); + + try (Transcript t = new Transcript("10-end-to-end.txt", "End to end: HTTP, real PostgreSQL + pgvector, the shipped application.yml")) { + t.line("schema from init.sql: %s", jdbc.queryForList( + "select indexname from pg_indexes where schemaname = 'rag' order by indexname", String.class)); + + t.section("POST /api/ingest (acme, then globex, then acme again)"); + Resource acmePdf = new ByteArrayResource(bytes(SamplePdf.of(SamplePdf.ACME))); + t.line("%s", upload(http, "acme-handbook.pdf", acmePdf, "acme")); + t.line("%s", upload(http, "globex-manual.pdf", SamplePdf.of(SamplePdf.GLOBEX), "globex")); + String again = upload(http, "acme-handbook.pdf", acmePdf, "acme"); + t.line("%s", again); + + t.section("what is in the table"); + t.line("rows: %d", jdbc.queryForObject("select count(*) from rag.document_chunks", Integer.class)); + t.line("rows per tenant: %s", jdbc.queryForList( + "select metadata::jsonb ->> 'tenant_id' as tenant, count(*) as n from rag.document_chunks group by 1 order by 1") + .stream().map(r -> r.get("tenant") + "=" + r.get("n")).toList()); + t.line("metadata of one row: %s", jdbc.queryForObject( + "select metadata::jsonb - 'parent_document_id' - 'source_hash' from rag.document_chunks where content like 'Globex%' limit 1", + String.class)); + + t.section("POST /api/query"); + String acme = query(http, "How many days of annual leave do employees get?", "acme"); + String globex = query(http, "How many days of annual leave do employees get?", "globex"); + t.line("tenantId acme: %s", acme); + t.line("tenantId globex: %s", globex); + + t.section("GET /actuator/prometheus (only the rag_ series; the timer's sum and max are left out because they change every run)"); + String scrape = http.get().uri("/actuator/prometheus").retrieve().body(String.class); + List lines = scrape.lines().filter(l -> l.startsWith("rag_") && !l.startsWith("rag_query_duration_seconds_bucket") + && !l.startsWith("rag_query_duration_seconds_sum") && !l.startsWith("rag_query_duration_seconds_max")).toList(); + lines.forEach(l -> t.line("%s", l)); + + assertThat(again).contains("\"status\":\"skipped\""); + assertThat(acme).contains("20 working days").contains("\"grounded\":true").doesNotContain("25 working days"); + assertThat(globex).contains("25 working days").contains("\"grounded\":true").doesNotContain("20 working days"); + assertThat(scrape).contains("rag_chunks_ingested_total").contains("rag_ingestion_skipped_total"); + } + } + + private static String upload(RestClient http, String filename, Resource pdf, String tenant) { + var body = new LinkedMultiValueMap(); + body.add("file", new ByteArrayResource(bytes(pdf)) { + @Override + public String getFilename() { + return filename; + } + }); + body.add("tenantId", tenant); + return http.post().uri("/api/ingest").contentType(MediaType.MULTIPART_FORM_DATA).body(body) + .retrieve().body(String.class); + } + + private static String query(RestClient http, String question, String tenant) { + return http.post().uri("/api/query").contentType(MediaType.APPLICATION_JSON) + .body(Map.of("question", question, "tenantId", tenant)).retrieve().body(String.class); + } + + private static byte[] bytes(Resource resource) { + try { + return resource.getInputStream().readAllBytes(); + } catch (java.io.IOException e) { + throw new IllegalStateException(e); + } + } +} diff --git a/rag/src/test/java/com/ankurm/rag/FaithfulnessTest.java b/rag/src/test/java/com/ankurm/rag/FaithfulnessTest.java new file mode 100644 index 0000000..441fc54 --- /dev/null +++ b/rag/src/test/java/com/ankurm/rag/FaithfulnessTest.java @@ -0,0 +1,92 @@ +package com.ankurm.rag; + +import java.util.Map; + +import com.ankurm.rag.config.RagProperties; +import com.ankurm.rag.config.TestPipeline; +import com.ankurm.rag.query.RagResponse; +import com.ankurm.rag.support.FakeChatModel; +import com.ankurm.rag.support.SamplePdf; +import com.ankurm.rag.support.Stores; +import com.ankurm.rag.support.Transcript; +import org.junit.jupiter.api.Test; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * Chapter 5: the check that runs after generation, and the four outcomes {@code RagQueryService} + * reports. Transcript 09. The scripted model decides what the "answer" and the "verdict" are, so + * this tests Spring AI's plumbing and this repository's handling of each outcome. It does not + * test how good a real judge model is at spotting a hallucination. + */ +class FaithfulnessTest { + + private static final String QUESTION = "How many days of annual leave do employees get?"; + private static final String OFF_TOPIC = "What is the capital of Mongolia?"; + + @Test + void fourOutcomes() { + TestPipeline p = TestPipeline.on(Stores::simple); + p.ingestion.ingestPdf(SamplePdf.of(SamplePdf.ACME), "acme-handbook.pdf", Map.of("tenant_id", "acme")); + TestPipeline strict = TestPipeline.on(new RagProperties(new RagProperties.Retrieval(20, 0.3), + new RagProperties.Rerank(5), new RagProperties.Chunking(512, 350)), Stores::simple); + strict.ingestion.ingestPdf(SamplePdf.of(SamplePdf.ACME), "acme-handbook.pdf", Map.of("tenant_id", "acme")); + + try (Transcript t = new Transcript("09-faithfulness-check.txt", "Generation and the faithfulness check")) { + t.section("1. the answer is in the chunks"); + p.chat.reset(); + RagResponse grounded = p.queries.ask(QUESTION, null); + show(t, grounded); + + t.section("2. the model answers with something the chunks do not say"); + p.chat.reset(); + p.chat.forcedAnswer("Employees get 30 days of annual leave."); + RagResponse invented = p.queries.ask(QUESTION, null); + show(t, invented); + t.line("the check the judge model was given:"); + t.line(p.chat.promptsContaining("Evaluate whether or not the following claim is supported").getFirst() + .lines().limit(3).reduce((a, b) -> a + "\n" + b).orElse("")); + + t.section("3. the judge says \"Yes.\" instead of \"yes\""); + p.chat.reset(); + p.chat.forcedVerdict("Yes."); + RagResponse punctuated = p.queries.ask(QUESTION, null); + show(t, punctuated); + p.chat.reset(); + p.chat.forcedVerdict("YES"); + RagResponse upper = p.queries.ask(QUESTION, null); + t.line("with the reply \"YES\": grounded=%s", upper.grounded()); + + t.section("4. the judge call itself fails"); + p.chat.reset(); + p.metrics.clear(); + p.chat.failFactChecks(true); + RagResponse judgeDown = p.queries.ask(QUESTION, null); + show(t, judgeDown); + t.line("rag.faithfulness.judge_failures = %.0f", p.metrics.counter("rag.faithfulness.judge_failures").count()); + + t.section("5. nothing is retrieved (threshold 0.3, off-topic question)"); + strict.chat.reset(); + RagResponse none = strict.queries.ask(OFF_TOPIC, null); + show(t, none); + t.line("fact-check calls made: %d", strict.chat.promptsContaining("Evaluate whether or not").size()); + + assertThat(grounded.status()).isEqualTo("answered"); + assertThat(grounded.grounded()).isTrue(); + assertThat(invented.status()).isEqualTo("ungrounded"); + assertThat(punctuated.grounded()).isFalse(); + assertThat(upper.grounded()).isTrue(); + assertThat(judgeDown.status()).isEqualTo("ungrounded"); + assertThat(p.metrics.counter("rag.faithfulness.judge_failures").count()).isEqualTo(1.0); + assertThat(none.status()).isEqualTo("no_context"); + assertThat(none.sources()).isEmpty(); + assertThat(strict.chat.promptsContaining("Evaluate whether or not")).isEmpty(); + assertThat(none.answer()).isEqualTo(FakeChatModel.REFUSAL); + } + } + + private static void show(Transcript t, RagResponse r) { + t.line("status=%s grounded=%s sources=%d", r.status(), r.grounded(), r.sources().size()); + t.line("answer: %s", r.answer()); + } +} diff --git a/rag/src/test/java/com/ankurm/rag/IngestionTest.java b/rag/src/test/java/com/ankurm/rag/IngestionTest.java new file mode 100644 index 0000000..ec4547d --- /dev/null +++ b/rag/src/test/java/com/ankurm/rag/IngestionTest.java @@ -0,0 +1,157 @@ +package com.ankurm.rag; + +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.regex.Matcher; +import java.util.regex.Pattern; + +import com.ankurm.rag.config.TestPipeline; +import com.ankurm.rag.ingest.IngestionResult; +import com.ankurm.rag.ingest.IngestionService; +import com.ankurm.rag.support.SamplePdf; +import com.ankurm.rag.support.Stores; +import com.ankurm.rag.support.Transcript; +import org.junit.jupiter.api.Test; +import org.springframework.ai.document.Document; +import org.springframework.ai.reader.pdf.PagePdfDocumentReader; +import org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig; +import org.springframework.core.io.Resource; +import org.springframework.jdbc.core.JdbcTemplate; + +import static org.assertj.core.api.Assertions.assertThat; + +/** Chapter 3: what a PDF becomes on the way in, and what re-ingesting does. Transcripts 04 and 05. */ +class IngestionTest { + + private static final Map ACME_TAGS = Map.of("tenant_id", "acme", "doc_type", "handbook"); + + @Test + void pdfPagesAndMetadata() { + Resource pdf = SamplePdf.of(SamplePdf.ACME); + List pages = new PagePdfDocumentReader(pdf, PdfDocumentReaderConfig.builder() + .withPagesPerDocument(1).build()).get(); + + try (Transcript t = new Transcript("04-pdf-pages-and-metadata.txt", "PagePdfDocumentReader: what one PDF page becomes")) { + t.line("pages in the PDF: 4, documents read: %d", pages.size()); + for (Document page : pages) { + t.line("page document metadata: %s", page.getMetadata()); + } + + Document page2 = pages.get(1); + String raw = page2.getText(); + String tidy = IngestionService.tidy(raw); + t.section("page 2 as read (single spaces as dots, runs of 4+ as [n spaces], line ends as a pilcrow)"); + t.line(visible(raw)); + t.section("page 2 after IngestionService.tidy"); + t.line(visible(tidy)); + t.section("size of page 2"); + t.line("as read : %d characters, longest run of spaces %d", raw.length(), longestSpaceRun(raw)); + t.line("tidied : %d characters, longest run of spaces %d", tidy.length(), longestSpaceRun(tidy)); + + assertThat(pages).hasSize(4); + assertThat(pages.get(1).getMetadata()).containsKey("page_number"); + assertThat(tidy.length()).isLessThanOrEqualTo(raw.length()); + assertThat(tidy).contains("20 working days").doesNotContain(" "); + } + } + + @Test + void reingestingTheSameFile() { + JdbcTemplate jdbc = Stores.jdbc(); + TestPipeline p = TestPipeline.on(embeddings -> Stores.pg(jdbc, embeddings)); + Resource v1 = SamplePdf.of(SamplePdf.ACME); + List> edited = new ArrayList<>(SamplePdf.ACME); + edited.set(1, List.of("4.1 Annual Leave Entitlement. Full-time employees are entitled to 22 working days", + "of annual leave per calendar year. Part-time employees receive leave pro rata.", + "", + "4.2 Leave Carryover. Unused annual leave may be carried over for a maximum of 5 days", + "into the next calendar year and must be used by 31 March.")); + Resource v2 = SamplePdf.of(edited); + + try (Transcript t = new Transcript("05-ingestion-idempotency.txt", "Ingesting the same handbook more than once (real PostgreSQL + pgvector)")) { + t.line("rows are counted with: select count(*) from vector_store"); + IngestionResult first = p.ingestion.ingestPdf(v1, "acme-handbook.pdf", ACME_TAGS); + int rowsAfterFirst = rows(jdbc); + int embeddedAfterFirst = p.embeddings.textsEmbedded(); + report(t, "1. first upload", first, rowsAfterFirst, embeddedAfterFirst); + + IngestionResult second = p.ingestion.ingestPdf(v1, "acme-handbook.pdf", ACME_TAGS); + report(t, "2. same bytes again", second, rows(jdbc), p.embeddings.textsEmbedded()); + + IngestionResult third = p.ingestion.ingestPdf(v2, "acme-handbook.pdf", ACME_TAGS); + report(t, "3. page 2 edited (20 -> 22 days)", third, rows(jdbc), p.embeddings.textsEmbedded()); + int stale = jdbc.queryForObject("select count(*) from vector_store where content like '%20 working days%'", Integer.class); + int fresh = jdbc.queryForObject("select count(*) from vector_store where content like '%22 working days%'", Integer.class); + t.line("rows still saying \"20 working days\": %d, rows saying \"22 working days\": %d", stale, fresh); + + Resource v2Again = SamplePdf.of(edited); + assertThat(bytes(v2Again)).isNotEqualTo(bytes(v2)); + IngestionResult reexport = p.ingestion.ingestPdf(v2Again, "acme-handbook.pdf", ACME_TAGS); + report(t, "4. same text, exported again", reexport, rows(jdbc), p.embeddings.textsEmbedded()); + t.line("the two PDFs have identical text and different bytes: %s", !java.util.Arrays.equals(bytes(v2Again), bytes(v2))); + t.line("the file hash is a hash of bytes, so a re-export counts as a change and is re-embedded"); + + t.section("the naive version: vectorStore.add() on every upload, nothing remembered"); + List pages = new PagePdfDocumentReader(v2, PdfDocumentReaderConfig.builder() + .withPagesPerDocument(1).build()).get(); + List chunks = p.chunker.apply(pages.stream() + .map(pg -> new Document(IngestionService.tidy(pg.getText()), pg.getMetadata())).toList()); + int before = rows(jdbc); + p.store.add(chunks); + p.store.add(chunks.stream().map(c -> new Document(c.getText(), c.getMetadata())).toList()); + int after = rows(jdbc); + t.line("the same %d chunks added on two more uploads: rows in table %d -> %d", chunks.size(), before, after); + t.line("rows saying \"22 working days\" now: %d", + jdbc.queryForObject("select count(*) from vector_store where content like '%22 working days%'", Integer.class)); + assertThat(after).isEqualTo(before + 2 * chunks.size()); + assertThat(first.status()).isEqualTo("ingested"); + assertThat(first.chunksWritten()).isEqualTo(rowsAfterFirst); + assertThat(second.status()).isEqualTo("skipped"); + assertThat(third.status()).isEqualTo("updated"); + assertThat(third.chunksReplaced()).isEqualTo(first.chunksWritten()); + assertThat(reexport.status()).isEqualTo("updated"); + assertThat(stale).isZero(); + assertThat(fresh).isEqualTo(1); + assertThat(p.embeddings.textsEmbedded()).isGreaterThan(embeddedAfterFirst); + } + } + + private static void report(Transcript t, String label, IngestionResult r, int rows, int embedded) { + t.line("%-34s -> status=%-8s chunksWritten=%d chunksReplaced=%d | rows in table=%d, texts embedded so far=%d", + label, r.status(), r.chunksWritten(), r.chunksReplaced(), rows, embedded); + } + + private static byte[] bytes(Resource resource) { + try { + return resource.getInputStream().readAllBytes(); + } catch (java.io.IOException e) { + throw new IllegalStateException(e); + } + } + + private static int rows(JdbcTemplate jdbc) { + return jdbc.queryForObject("select count(*) from vector_store", Integer.class); + } + + /** Short runs of spaces become dots, runs of four or more become "[n spaces]", line ends a pilcrow. */ + private static String visible(String text) { + Matcher m = Pattern.compile(" +").matcher(text); + StringBuilder sb = new StringBuilder(); + while (m.find()) { + int n = m.end() - m.start(); + m.appendReplacement(sb, n >= 4 ? "[" + n + " spaces]" : "\u00b7".repeat(n)); + } + m.appendTail(sb); + return sb.toString().replace("\n", "\u00b6\n"); + } + + private static int longestSpaceRun(String text) { + int longest = 0; + Matcher m = Pattern.compile(" +").matcher(text); + while (m.find()) { + longest = Math.max(longest, m.end() - m.start()); + } + return longest; + } +} diff --git a/rag/src/test/java/com/ankurm/rag/RerankTest.java b/rag/src/test/java/com/ankurm/rag/RerankTest.java new file mode 100644 index 0000000..2f414f2 --- /dev/null +++ b/rag/src/test/java/com/ankurm/rag/RerankTest.java @@ -0,0 +1,107 @@ +package com.ankurm.rag; + +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +import com.ankurm.rag.config.RagProperties; +import com.ankurm.rag.config.TestPipeline; +import com.ankurm.rag.query.LlmReranker; +import com.ankurm.rag.support.SamplePdf; +import com.ankurm.rag.support.Stores; +import com.ankurm.rag.support.Transcript; +import org.junit.jupiter.api.Test; +import org.springframework.ai.chat.prompt.Prompt; +import org.springframework.ai.document.Document; +import org.springframework.ai.rag.Query; +import org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * Chapter 4, second half: what an LLM reranker costs, how it fails, and why it runs on virtual + * threads. Transcript 08. The chat model here is the scripted fake, so the timings are its + * {@code Thread.sleep}, standing in for network latency, and the ratings are word overlap. What + * this shows is the shape of the cost and the failure handling; it does not show that reranking + * improves answers, and it cannot. + */ +class RerankTest { + + private static final String QUESTION = "How many days of annual leave do employees get?"; + + @Test + void costOrderingAndFailure() { + TestPipeline p = TestPipeline.on(withTopN(2), Stores::simple); + p.ingestion.ingestPdf(SamplePdf.of(SamplePdf.ACME), "acme-handbook.pdf", Map.of("tenant_id", "acme")); + List candidates = VectorStoreDocumentRetriever.builder().vectorStore(p.store).topK(20) + .similarityThreshold(0.0).build().retrieve(new Query(QUESTION)); + + try (Transcript t = new Transcript("08-reranking.txt", "LLM reranking: calls, order, latency and failure")) { + t.section("one model call per candidate; only topN survive (topN = 2)"); + p.chat.reset(); + List kept = p.reranker.process(new Query(QUESTION), candidates); + t.line("candidates in: %d, model calls made: %d, chunks out: %d", candidates.size(), p.chat.calls(), kept.size()); + t.line("order from the vector search, best first:"); + for (Document d : candidates) { + t.line(" similarity %.4f page %s", d.getScore(), d.getMetadata().get("page_number")); + } + t.line("order after reranking, best first:"); + for (Document d : kept) { + t.line(" rerank_score %s page %s", d.getMetadata().get(LlmReranker.SCORE), d.getMetadata().get("page_number")); + } + t.line("the rating prompt for the page 2 candidate (calls run concurrently, so pick it by content):"); + t.line(p.chat.promptsContaining("4.1 Annual Leave").getFirst().lines().limit(4).reduce((a, b) -> a + "\n" + b).orElse("")); + + t.section("latency: 20 candidates, each rating call takes 200 ms (a Thread.sleep in the fake model)"); + List twenty = new ArrayList<>(); + for (int i = 1; i <= 20; i++) { + twenty.add(new Document("Passage " + i + " says employees get " + i + " days of annual leave.")); + } + p.chat.reset(); + p.chat.latencyMillis(200); + long start = System.nanoTime(); + p.reranker.process(new Query(QUESTION), twenty); + long concurrent = (System.nanoTime() - start) / 1_000_000; + + p.chat.reset(); + p.chat.latencyMillis(200); + start = System.nanoTime(); + for (Document d : twenty) { + p.chat.call(new Prompt("Rate how well the PASSAGE helps answer the QUESTION\nQUESTION: " + QUESTION + + "\nPASSAGE: " + d.getText())); + } + long sequential = (System.nanoTime() - start) / 1_000_000; + t.line("the 20 calls one after another take 4000 ms or more: %s", sequential >= 4000); + t.line("LlmReranker, one virtual thread per candidate, takes under 1000 ms: %s", concurrent < 1000); + t.line("a real API adds its own rate limits, which this test cannot show"); + + t.section("failure: the model does not reply with a bare integer"); + p.chat.reset(); + p.metrics.clear(); + p.chat.forcedRating("Score: 8"); + List degraded = p.reranker.process(new Query(QUESTION), candidates); + double failures = p.metrics.counter("rag.rerank.failures").count(); + t.line("reply \"Score: 8\" for every candidate -> failures counted: %.0f of %d", failures, candidates.size()); + t.line("scores assigned: %s", degraded.stream().map(d -> d.getMetadata().get(LlmReranker.SCORE)).toList()); + t.line("pages kept, in order: %s (the vector-search order, because every score is 0)", + degraded.stream().map(d -> d.getMetadata().get("page_number")).toList()); + p.chat.forcedRating(" 9\n"); + List padded = p.reranker.process(new Query(QUESTION), candidates.subList(0, 1)); + t.line("reply \" 9\\n\" (padded) -> score %s", padded.getFirst().getMetadata().get(LlmReranker.SCORE)); + + assertThat(p.chat.calls()).isPositive(); + assertThat(kept).hasSize(2); + assertThat(concurrent).isLessThan(1000); + assertThat(sequential).isGreaterThanOrEqualTo(4000); + assertThat(failures).isEqualTo(candidates.size()); + assertThat(degraded.stream().map(d -> d.getMetadata().get(LlmReranker.SCORE))).containsOnly(0); + assertThat(degraded.stream().map(d -> d.getMetadata().get("page_number"))).containsExactly(2, 3); + assertThat(padded.getFirst().getMetadata().get(LlmReranker.SCORE)).isEqualTo(9); + } + } + + private static RagProperties withTopN(int topN) { + return new RagProperties(new RagProperties.Retrieval(20, 0.0), new RagProperties.Rerank(topN), + new RagProperties.Chunking(512, 350)); + } +} diff --git a/rag/src/test/java/com/ankurm/rag/RetrievalTest.java b/rag/src/test/java/com/ankurm/rag/RetrievalTest.java new file mode 100644 index 0000000..9a9b139 --- /dev/null +++ b/rag/src/test/java/com/ankurm/rag/RetrievalTest.java @@ -0,0 +1,120 @@ +package com.ankurm.rag; + +import java.util.List; +import java.util.Map; + +import com.ankurm.rag.config.TestPipeline; +import com.ankurm.rag.support.SamplePdf; +import com.ankurm.rag.support.Stores; +import com.ankurm.rag.support.Transcript; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; +import org.springframework.ai.chat.client.ChatClient; +import org.springframework.ai.document.Document; +import org.springframework.ai.rag.Query; +import org.springframework.ai.rag.advisor.RetrievalAugmentationAdvisor; +import org.springframework.ai.rag.generation.augmentation.ContextualQueryAugmenter; +import org.springframework.ai.rag.retrieval.search.VectorStoreDocumentRetriever; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * Chapter 4, first half: what the retriever returns, what the similarity threshold does, what the + * prompt looks like once the chunks are in it, and what happens when nothing is retrieved. + * Transcript 06. Uses the hashing embedding model, so the scores are this repository's, not a real + * model's: the point is the behaviour of the Spring AI components, not the numbers. + */ +class RetrievalTest { + + private static final String LEAVE = "How many days of annual leave do employees get?"; + private static final String OFF_TOPIC = "What is the capital of Mongolia?"; + + private static TestPipeline pipeline; + + @BeforeAll + static void ingest() { + pipeline = TestPipeline.on(Stores::simple); + pipeline.ingestion.ingestPdf(SamplePdf.of(SamplePdf.ACME), "acme-handbook.pdf", Map.of("tenant_id", "acme")); + } + + @Test + void thresholdPromptShapeAndEmptyContext() { + try (Transcript t = new Transcript("06-retrieval-threshold-and-prompt.txt", + "Retrieval: threshold, prompt shape and the empty-context path")) { + t.line("the store holds the 4 pages of one handbook as 4 chunks; topK = 20"); + + t.section("question: \"" + LEAVE + "\""); + List allAccepted = retrieve(0.0, LEAVE); + List filtered = retrieve(0.3, LEAVE); + scores(t, "similarityThreshold 0.0 (the default)", allAccepted); + scores(t, "similarityThreshold 0.3", filtered); + + t.section("question: \"" + OFF_TOPIC + "\""); + List offAll = retrieve(0.0, OFF_TOPIC); + List offFiltered = retrieve(0.3, OFF_TOPIC); + scores(t, "similarityThreshold 0.0 (the default)", offAll); + scores(t, "similarityThreshold 0.3", offFiltered); + + t.section("the prompt the model receives (threshold 0.3, question about leave)"); + String augmented = promptFor(0.3, false, LEAVE); + t.line(augmented); + + t.section("nothing retrieved, allowEmptyContext(false): the prompt the model receives"); + String refusing = promptFor(0.3, false, OFF_TOPIC); + t.line(refusing); + + t.section("nothing retrieved, allowEmptyContext(true): the prompt the model receives"); + String passthrough = promptFor(0.3, true, OFF_TOPIC); + t.line(passthrough); + + t.section("off-topic question, default threshold 0.0, allowEmptyContext(false)"); + String offDefault = promptFor(0.0, false, OFF_TOPIC); + t.line("chunks placed in the prompt: %d of 4", count(offDefault, "Full-time") + count(offDefault, "Expenses") + + count(offDefault, "Requesting Leave") + count(offDefault, "Probationary")); + t.line("the empty-context safety net fired: %s", offDefault.contains("outside your knowledge base")); + + assertThat(allAccepted).hasSize(4); + assertThat(filtered).hasSize(2); + assertThat(filtered.getFirst().getText()).contains("20 working days"); + assertThat(offAll).hasSize(4); + assertThat(offFiltered).isEmpty(); + assertThat(augmented).contains("Context information is below").contains("20 working days"); + assertThat(refusing).contains("outside your knowledge base").doesNotContain("Context information is below"); + assertThat(passthrough).isEqualTo(OFF_TOPIC); + assertThat(offDefault).doesNotContain("outside your knowledge base").contains("Context information is below"); + } + } + + private static List retrieve(double threshold, String question) { + return VectorStoreDocumentRetriever.builder().vectorStore(pipeline.store).topK(20) + .similarityThreshold(threshold).build().retrieve(new Query(question)); + } + + private static String promptFor(double threshold, boolean allowEmptyContext, String question) { + pipeline.chat.reset(); + var advisor = RetrievalAugmentationAdvisor.builder() + .documentRetriever(VectorStoreDocumentRetriever.builder().vectorStore(pipeline.store).topK(20) + .similarityThreshold(threshold).build()) + .queryAugmenter(ContextualQueryAugmenter.builder().allowEmptyContext(allowEmptyContext).build()) + .build(); + ChatClient.builder(pipeline.chat).defaultAdvisors(advisor).build().prompt().user(question).call().content(); + return pipeline.chat.prompts().getFirst().getContents(); + } + + private static void scores(Transcript t, String label, List docs) { + t.line("%s: %d chunk(s)", label, docs.size()); + // Equal scores come back in no promised order, so ties are listed by page to keep this file stable. + List ordered = docs.stream().sorted(java.util.Comparator + .comparing(Document::getScore, java.util.Comparator.reverseOrder()) + .thenComparing(d -> (Integer) d.getMetadata().get("page_number"))).toList(); + for (Document d : ordered) { + String text = d.getText(); + t.line(" score %.4f page %s \"%s...\"", d.getScore(), d.getMetadata().get("page_number"), + text.substring(0, Math.min(44, text.length())).replace('\n', ' ')); + } + } + + private static int count(String text, String needle) { + return text.contains(needle) ? 1 : 0; + } +} diff --git a/rag/src/test/java/com/ankurm/rag/SimpleAdvisorTest.java b/rag/src/test/java/com/ankurm/rag/SimpleAdvisorTest.java new file mode 100644 index 0000000..3151f79 --- /dev/null +++ b/rag/src/test/java/com/ankurm/rag/SimpleAdvisorTest.java @@ -0,0 +1,54 @@ +package com.ankurm.rag; + +import java.util.Map; + +import com.ankurm.rag.config.TestPipeline; +import com.ankurm.rag.support.SamplePdf; +import com.ankurm.rag.support.Stores; +import com.ankurm.rag.support.Transcript; +import org.junit.jupiter.api.Test; +import org.springframework.ai.chat.client.ChatClient; +import org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor; +import org.springframework.ai.vectorstore.SearchRequest; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * Chapter 1: the smallest RAG that works, one advisor and no other class. Transcript 11. + * It is where the article starts, and the comparison at the end shows what it does not do. + */ +class SimpleAdvisorTest { + + private static final String QUESTION = "How many days of annual leave do employees get?"; + private static final String OFF_TOPIC = "What is the capital of Mongolia?"; + + @Test + void smallestWorkingRag() { + TestPipeline p = TestPipeline.on(Stores::simple); + p.ingestion.ingestPdf(SamplePdf.of(SamplePdf.ACME), "acme-handbook.pdf", Map.of("tenant_id", "acme")); + p.chat.reset(); + + ChatClient client = ChatClient.builder(p.chat) + .defaultAdvisors(QuestionAnswerAdvisor.builder(p.store) + .searchRequest(SearchRequest.builder().topK(2).similarityThreshold(0.3).build()) + .build()) + .build(); + + try (Transcript t = new Transcript("11-question-answer-advisor.txt", "QuestionAnswerAdvisor: the smallest RAG")) { + client.prompt().user(QUESTION).call().content(); + String prompt = p.chat.prompts().getFirst().getContents(); + t.line("only the prompts are recorded: what a real model would reply is not something this repository tests"); + t.section("the prompt the model received"); + t.line(prompt); + + p.chat.reset(); + client.prompt().user(OFF_TOPIC).call().content(); + String offTopicPrompt = p.chat.prompts().getFirst().getContents(); + t.section("the prompt for an off-topic question (nothing passes the threshold)"); + t.line(offTopicPrompt); + + assertThat(prompt).contains("20 working days").contains(QUESTION); + assertThat(offTopicPrompt).contains(OFF_TOPIC); + } + } +} diff --git a/rag/src/test/java/com/ankurm/rag/TenantFilterTest.java b/rag/src/test/java/com/ankurm/rag/TenantFilterTest.java new file mode 100644 index 0000000..4a6ca19 --- /dev/null +++ b/rag/src/test/java/com/ankurm/rag/TenantFilterTest.java @@ -0,0 +1,97 @@ +package com.ankurm.rag; + +import java.util.List; +import java.util.Map; + +import com.ankurm.rag.config.TestPipeline; +import com.ankurm.rag.support.SamplePdf; +import com.ankurm.rag.support.Stores; +import com.ankurm.rag.support.Transcript; +import org.junit.jupiter.api.Test; +import org.springframework.ai.document.Document; +import org.springframework.ai.vectorstore.SearchRequest; +import org.springframework.ai.vectorstore.VectorStore; +import org.springframework.ai.vectorstore.filter.FilterExpressionBuilder; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * Chapter 3 and 4: keeping one tenant's chunks away from another's, and how a filter can be built + * so that user input cannot rewrite it. Runs the same checks on the in-memory store and on real + * PostgreSQL, because the two translate a filter differently. Transcript 07. + */ +class TenantFilterTest { + + private static final String QUESTION = "How many days of annual leave do employees get?"; + /** A caller who belongs to globex sends this as their tenant id. */ + private static final String HOSTILE_SINGLE = "globex' || tenant_id == 'acme"; + private static final String HOSTILE_DOUBLE = "globex\" || @.tenant_id == \"acme"; + + @Test + void filtersOnBothStores() { + try (Transcript t = new Transcript("07-tenant-filter-and-injection.txt", + "Tenant isolation: metadata filters, and a filter built from user input")) { + t.line("two tenants each upload a handbook with a section 4.1 on annual leave"); + t.line("acme says 20 working days, globex says 25"); + + var simple = TestPipeline.on(Stores::simple); + load(simple); + check(t, "SimpleVectorStore (in memory)", simple.store); + + var jdbc = Stores.jdbc(); + var pg = TestPipeline.on(e -> Stores.pg(jdbc, e)); + load(pg); + check(t, "PgVectorStore (PostgreSQL + pgvector)", pg.store); + } + } + + private static void load(TestPipeline p) { + p.ingestion.ingestPdf(SamplePdf.of(SamplePdf.ACME), "acme-handbook.pdf", Map.of("tenant_id", "acme")); + p.ingestion.ingestPdf(SamplePdf.of(SamplePdf.GLOBEX), "globex-manual.pdf", Map.of("tenant_id", "globex")); + } + + private static void check(Transcript t, String storeName, VectorStore store) { + t.section(storeName); + + List unfiltered = store.similaritySearch(SearchRequest.builder().query(QUESTION).topK(3).build()); + describe(t, "no filter, top 3", unfiltered); + + var acmeOnly = new FilterExpressionBuilder().eq("tenant_id", "acme").build(); + List scoped = store.similaritySearch( + SearchRequest.builder().query(QUESTION).topK(3).filterExpression(acmeOnly).build()); + describe(t, "eq(\"tenant_id\", \"acme\") built with FilterExpressionBuilder, top 3", scoped); + + // The mistake: build the filter as text from a value the caller controls. + String glued = "tenant_id == '" + HOSTILE_SINGLE + "'"; + t.line("filter string built by concatenation: %s", glued); + List injected = store.similaritySearch( + SearchRequest.builder().query(QUESTION).topK(5).filterExpression(glued).build()); + describe(t, " result, top 5", injected); + + // The same hostile text as a value handed to the builder is only ever a value. + List asValue = store.similaritySearch(SearchRequest.builder().query(QUESTION).topK(5) + .filterExpression(new FilterExpressionBuilder().eq("tenant_id", HOSTILE_SINGLE).build()).build()); + describe(t, "same text passed to FilterExpressionBuilder.eq(), top 5", asValue); + List asValueDouble = store.similaritySearch(SearchRequest.builder().query(QUESTION).topK(5) + .filterExpression(new FilterExpressionBuilder().eq("tenant_id", HOSTILE_DOUBLE).build()).build()); + describe(t, "double-quote variant passed to FilterExpressionBuilder.eq(), top 5", asValueDouble); + + assertThat(tenants(unfiltered)).contains("globex"); + assertThat(tenants(scoped)).containsOnly("acme").hasSize(3); + assertThat(tenants(injected)).contains("acme", "globex"); + assertThat(asValue).isEmpty(); + assertThat(asValueDouble).isEmpty(); + } + + private static List tenants(List docs) { + return docs.stream().map(d -> String.valueOf(d.getMetadata().get("tenant_id"))).toList(); + } + + private static void describe(Transcript t, String label, List docs) { + t.line("%s: %d chunk(s)", label, docs.size()); + for (Document d : docs) { + t.line(" tenant=%-6s page=%s \"%s...\"", d.getMetadata().get("tenant_id"), d.getMetadata().get("page_number"), + d.getText().substring(0, 40).replace('\n', ' ')); + } + } +} diff --git a/rag/src/test/java/com/ankurm/rag/config/TestPipeline.java b/rag/src/test/java/com/ankurm/rag/config/TestPipeline.java new file mode 100644 index 0000000..e6d981a --- /dev/null +++ b/rag/src/test/java/com/ankurm/rag/config/TestPipeline.java @@ -0,0 +1,63 @@ +package com.ankurm.rag.config; + +import java.util.function.Function; + +import com.ankurm.rag.config.RagProperties.Chunking; +import com.ankurm.rag.config.RagProperties.Rerank; +import com.ankurm.rag.config.RagProperties.Retrieval; +import com.ankurm.rag.ingest.IngestionService; +import com.ankurm.rag.ingest.IngestionTracker; +import com.ankurm.rag.query.LlmReranker; +import com.ankurm.rag.query.RagQueryService; +import com.ankurm.rag.support.FakeChatModel; +import com.ankurm.rag.support.HashingEmbeddingModel; +import com.ankurm.rag.support.Stores; +import io.micrometer.core.instrument.simple.SimpleMeterRegistry; +import org.springframework.ai.chat.client.ChatClient; +import org.springframework.ai.document.DocumentTransformer; +import org.springframework.ai.embedding.EmbeddingModel; +import org.springframework.ai.vectorstore.VectorStore; + +/** + * The whole pipeline, wired with the same {@link RagConfig} the application uses but with the fake + * chat and embedding models, so a test can run it with no API key. It lives in the same package as + * {@code RagConfig} because the {@code @Bean} methods are package-private. + */ +public final class TestPipeline { + + public final FakeChatModel chat = new FakeChatModel(); + public final HashingEmbeddingModel embeddings = new HashingEmbeddingModel(Stores.DIMS); + public final SimpleMeterRegistry metrics = new SimpleMeterRegistry(); + public final IngestionTracker tracker = new IngestionTracker(); + public final RagProperties properties; + public final VectorStore store; + public final RagConfig config = new RagConfig(); + public final DocumentTransformer chunker; + public final LlmReranker reranker; + public final IngestionService ingestion; + public final RagQueryService queries; + + private TestPipeline(RagProperties properties, Function storeFactory) { + this.properties = properties; + this.store = storeFactory.apply(embeddings); + ChatClient.Builder builder = ChatClient.builder(chat); + this.reranker = config.reranker(builder, properties, metrics); + this.chunker = config.chunker(properties); + this.ingestion = new IngestionService(store, chunker, tracker, metrics); + var advisor = config.retrievalAdvisor(store, reranker, properties); + this.queries = new RagQueryService(config.ragChatClient(ChatClient.builder(chat), advisor), + config.factChecker(ChatClient.builder(chat)), metrics); + } + + public static RagProperties defaults() { + return new RagProperties(new Retrieval(20, 0.0), new Rerank(5), new Chunking(512, 350)); + } + + public static TestPipeline on(Function storeFactory) { + return new TestPipeline(defaults(), storeFactory); + } + + public static TestPipeline on(RagProperties properties, Function storeFactory) { + return new TestPipeline(properties, storeFactory); + } +} diff --git a/rag/src/test/java/com/ankurm/rag/support/FakeChatModel.java b/rag/src/test/java/com/ankurm/rag/support/FakeChatModel.java new file mode 100644 index 0000000..be7ab13 --- /dev/null +++ b/rag/src/test/java/com/ankurm/rag/support/FakeChatModel.java @@ -0,0 +1,174 @@ +package com.ankurm.rag.support; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.concurrent.CopyOnWriteArrayList; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.regex.Matcher; +import java.util.regex.Pattern; + +import org.springframework.ai.chat.messages.AssistantMessage; +import org.springframework.ai.chat.model.ChatModel; +import org.springframework.ai.chat.model.ChatResponse; +import org.springframework.ai.chat.model.Generation; +import org.springframework.ai.chat.prompt.Prompt; + +/** + * A scripted chat model. It plays three roles, told apart by what the prompt says, so one bean + * serves the answering client, the reranker and the fact checker: + * + *

    + *
  • a rating prompt gets the number of question words found in the passage, scaled to 0-10;
    • + *
  • a fact-check prompt gets "yes" if every content word of the claim is in the document;
    • + *
  • anything else is a question: the reply is the sentence of the supplied context that + * shares the most words with the question, or the refusal if there is no context.
    • + *
+ * + *

It records every prompt it receives, which is what the tests assert on: what Spring AI + * sent is the part of a RAG pipeline that Spring AI is responsible for. What a real model + * would reply is not tested anywhere in this repository. + */ +public class FakeChatModel implements ChatModel { + + public static final String REFUSAL = "I don't have enough information in the provided documents."; + + private static final Pattern QUESTION_LINE = Pattern.compile("QUESTION: (.*)"); + private static final Pattern PASSAGE_LINE = Pattern.compile("PASSAGE: (.*)", Pattern.DOTALL); + private static final Pattern QUERY_LINE = Pattern.compile("Query: (.*)"); + private static final Pattern CONTEXT_BLOCK = Pattern.compile("Context information is below\\.\\s*-+\\s*(.*?)\\s*-+\\s*Given", + Pattern.DOTALL); + + private final List prompts = new CopyOnWriteArrayList<>(); + private final AtomicInteger calls = new AtomicInteger(); + + /** Milliseconds each call sleeps, to stand in for network latency. */ + private volatile long latencyMillis; + /** When set, question replies are this text instead of a sentence taken from the context. */ + private volatile String forcedAnswer; + /** When set, every fact-check reply is this text. */ + private volatile String forcedVerdict; + /** When set, every rating reply is this text. */ + private volatile String forcedRating; + /** When true, a fact-check call throws, as a network failure or a rate limit would. */ + private volatile boolean failFactChecks; + + public List prompts() { + return Collections.unmodifiableList(prompts); + } + + public int calls() { + return calls.get(); + } + + public void reset() { + prompts.clear(); + calls.set(0); + latencyMillis = 0; + forcedAnswer = null; + forcedVerdict = null; + forcedRating = null; + failFactChecks = false; + } + + public void latencyMillis(long millis) { + this.latencyMillis = millis; + } + + public void forcedAnswer(String answer) { + this.forcedAnswer = answer; + } + + public void forcedVerdict(String verdict) { + this.forcedVerdict = verdict; + } + + public void failFactChecks(boolean fail) { + this.failFactChecks = fail; + } + + public void forcedRating(String rating) { + this.forcedRating = rating; + } + + @Override + public ChatResponse call(Prompt prompt) { + prompts.add(prompt); + calls.incrementAndGet(); + if (latencyMillis > 0) { + try { + Thread.sleep(latencyMillis); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + } + return new ChatResponse(List.of(new Generation(new AssistantMessage(reply(prompt.getContents()))))); + } + + private String reply(String text) { + if (text.contains("Rate how well the PASSAGE")) { + return forcedRating != null ? forcedRating : rating(text); + } + if (text.contains("Evaluate whether or not the following claim is supported")) { + if (failFactChecks) { + throw new IllegalStateException("simulated: 429 Too Many Requests from the judge model"); + } + return forcedVerdict != null ? forcedVerdict : verdict(text); + } + if (forcedAnswer != null) { + return forcedAnswer; + } + return answer(text); + } + + private static String rating(String text) { + Matcher q = QUESTION_LINE.matcher(text); + Matcher p = PASSAGE_LINE.matcher(text); + if (!q.find() || !p.find()) { + return "0"; + } + List question = HashingEmbeddingModel.words(q.group(1)); + List passage = HashingEmbeddingModel.words(p.group(1)); + long hits = question.stream().filter(passage::contains).count(); + return String.valueOf(question.isEmpty() ? 0 : Math.round(10.0 * hits / question.size())); + } + + private static String verdict(String text) { + int doc = text.indexOf("Document:"); + int claim = text.indexOf("Claim:"); + List document = HashingEmbeddingModel.words(text.substring(doc + 9, claim)); + List claimWords = HashingEmbeddingModel.words(text.substring(claim + 6)); + return document.containsAll(claimWords) ? "yes" : "no"; + } + + private static String answer(String text) { + Matcher context = CONTEXT_BLOCK.matcher(text); + if (!context.find() || context.group(1).isBlank()) { + return REFUSAL; + } + Matcher query = QUERY_LINE.matcher(text); + List questionWords = HashingEmbeddingModel.words(query.find() ? query.group(1) : ""); + String best = REFUSAL; + long bestHits = -1; + for (String sentence : context.group(1).split("(?<=[.!?])\\s+")) { + List words = HashingEmbeddingModel.words(sentence); + long hits = questionWords.stream().filter(words::contains).count(); + if (hits > bestHits) { + bestHits = hits; + best = sentence.strip(); + } + } + return best; + } + + /** Convenience for tests: the user text of every prompt whose text contains {@code marker}. */ + public List promptsContaining(String marker) { + List out = new ArrayList<>(); + for (Prompt p : prompts) { + if (p.getContents().contains(marker)) { + out.add(p.getContents()); + } + } + return out; + } +} diff --git a/rag/src/test/java/com/ankurm/rag/support/HashingEmbeddingModel.java b/rag/src/test/java/com/ankurm/rag/support/HashingEmbeddingModel.java new file mode 100644 index 0000000..de667f3 --- /dev/null +++ b/rag/src/test/java/com/ankurm/rag/support/HashingEmbeddingModel.java @@ -0,0 +1,90 @@ +package com.ankurm.rag.support; + +import java.util.ArrayList; +import java.util.List; +import java.util.Locale; +import java.util.Set; + +import org.springframework.ai.document.Document; +import org.springframework.ai.embedding.Embedding; +import org.springframework.ai.embedding.EmbeddingModel; +import org.springframework.ai.embedding.EmbeddingRequest; +import org.springframework.ai.embedding.EmbeddingResponse; + +/** + * A deterministic stand-in for a real embedding model: each word is hashed into one of {@code dims} + * slots and the vector is normalised. Two texts are close when they share words, which is enough to + * make retrieval behave the way it should, and nothing like a real model's sense of meaning. + * + *

Nothing here calls a network, so a test that uses it can be run by anyone with no API key. + * It also counts the texts it was asked to embed, which is how the ingestion tests show that + * skipping an unchanged file spends nothing. + */ +public class HashingEmbeddingModel implements EmbeddingModel { + + private static final Set STOPWORDS = Set.of("a", "an", "the", "of", "to", "in", "is", "are", "and", + "or", "for", "on", "at", "be", "by", "do", "does", "how", "what", "many", "can", "it", "its"); + + private final int dims; + private int textsEmbedded; + + public HashingEmbeddingModel(int dims) { + this.dims = dims; + } + + public synchronized int textsEmbedded() { + return textsEmbedded; + } + + @Override + public EmbeddingResponse call(EmbeddingRequest request) { + List out = new ArrayList<>(); + for (int i = 0; i < request.getInstructions().size(); i++) { + out.add(new Embedding(vector(request.getInstructions().get(i)), i)); + } + return new EmbeddingResponse(out); + } + + @Override + public float[] embed(Document document) { + return vector(document.getText()); + } + + @Override + public int dimensions() { + return dims; + } + + /** The words the model keeps for a text, lower-cased with a plural "s" removed. */ + public static List words(String text) { + List words = new ArrayList<>(); + for (String w : text.toLowerCase(Locale.ROOT).split("[^a-z0-9]+")) { + if (w.isEmpty() || STOPWORDS.contains(w)) { + continue; + } + words.add(w.length() > 3 && w.endsWith("s") ? w.substring(0, w.length() - 1) : w); + } + return words; + } + + private synchronized float[] vector(String text) { + textsEmbedded++; + float[] v = new float[dims]; + for (String w : words(text)) { + v[Math.floorMod(w.hashCode() * 0x9E3779B1, dims)] += 1f; + } + double norm = 0; + for (float x : v) { + norm += x * x; + } + norm = Math.sqrt(norm); + if (norm == 0) { + v[0] = 1f; + return v; + } + for (int i = 0; i < dims; i++) { + v[i] /= (float) norm; + } + return v; + } +} diff --git a/rag/src/test/java/com/ankurm/rag/support/SamplePdf.java b/rag/src/test/java/com/ankurm/rag/support/SamplePdf.java new file mode 100644 index 0000000..ba813e3 --- /dev/null +++ b/rag/src/test/java/com/ankurm/rag/support/SamplePdf.java @@ -0,0 +1,76 @@ +package com.ankurm.rag.support; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.util.List; + +import org.apache.pdfbox.pdmodel.PDDocument; +import org.apache.pdfbox.pdmodel.PDPage; +import org.apache.pdfbox.pdmodel.PDPageContentStream; +import org.apache.pdfbox.pdmodel.font.PDType1Font; +import org.apache.pdfbox.pdmodel.font.Standard14Fonts; +import org.springframework.core.io.ByteArrayResource; +import org.springframework.core.io.Resource; + +/** + * Builds small handbook PDFs on the fly, one page per entry, so the tests read a real PDF through + * the real {@code PagePdfDocumentReader} without a binary file in the repository. + */ +public final class SamplePdf { + + /** Two companies with different rules, so a tenant filter has something to keep apart. */ + public static final List> ACME = List.of( + List.of("Acme Employee Handbook 2026", "", + "3.5 Probationary Period. During the three month probationary period annual leave", + "accrues at half the normal rate. Probation can be extended once by four weeks."), + List.of("4.1 Annual Leave Entitlement. Full-time employees are entitled to 20 working days", + "of annual leave per calendar year. Part-time employees receive leave pro rata.", + "", + "4.2 Leave Carryover. Unused annual leave may be carried over for a maximum of 5 days", + "into the next calendar year and must be used by 31 March."), + List.of("4.3 Requesting Leave. All leave requests must be submitted through the HR portal", + "at least two weeks in advance for absences longer than three days.", + "", + "4.7 Sick Leave. Sick leave is separate from annual leave and is not deducted from it.", + "A medical certificate is required after three consecutive days."), + List.of("6.1 Expenses. Receipts are required for every expense above 50 euros.", + "Claims must be filed within 30 days of the expense.", + "", + "7.2 Remote Work. Employees may work remotely up to two days per week", + "with the agreement of their manager.")); + + public static final List> GLOBEX = List.of( + List.of("Globex Staff Manual 2026", "", + "4.1 Annual Leave Entitlement. Full-time staff are entitled to 25 working days", + "of annual leave per calendar year. Unused annual leave cannot be carried over.", + "", + "5.4 Overtime. Overtime must be approved in advance and is paid at 1.5 times the base rate.")); + + private SamplePdf() { + } + + public static Resource of(List> pages) { + try (PDDocument document = new PDDocument(); ByteArrayOutputStream bytes = new ByteArrayOutputStream()) { + PDType1Font font = new PDType1Font(Standard14Fonts.FontName.HELVETICA); + for (List lines : pages) { + PDPage page = new PDPage(); + document.addPage(page); + try (PDPageContentStream stream = new PDPageContentStream(document, page)) { + stream.setFont(font, 11); + stream.beginText(); + stream.setLeading(16); + stream.newLineAtOffset(50, 730); + for (String line : lines) { + stream.showText(line); + stream.newLine(); + } + stream.endText(); + } + } + document.save(bytes); + return new ByteArrayResource(bytes.toByteArray()); + } catch (IOException e) { + throw new IllegalStateException(e); + } + } +} diff --git a/rag/src/test/java/com/ankurm/rag/support/Stores.java b/rag/src/test/java/com/ankurm/rag/support/Stores.java new file mode 100644 index 0000000..d44c66b --- /dev/null +++ b/rag/src/test/java/com/ankurm/rag/support/Stores.java @@ -0,0 +1,52 @@ +package com.ankurm.rag.support; + +import org.springframework.ai.embedding.EmbeddingModel; +import org.springframework.ai.vectorstore.SimpleVectorStore; +import org.springframework.ai.vectorstore.pgvector.PgVectorStore; +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.jdbc.datasource.DriverManagerDataSource; + +/** + * The two vector stores the tests use. {@link #simple} keeps vectors in a map in memory; + * {@link #pg} talks to a real PostgreSQL with the pgvector extension. + * + *

The PostgreSQL location comes from {@code RAG_PG_URL} (default the docker-compose one, + * {@code localhost:5432}); see {@code scripts/pg-up.sh} for starting one without Docker. + */ +public final class Stores { + + public static final String PG_URL = env("RAG_PG_URL", "jdbc:postgresql://localhost:5432/ragdb"); + public static final String PG_USER = env("RAG_PG_USER", "raguser"); + public static final String PG_PASSWORD = env("RAG_PG_PASSWORD", "ragpass"); + /** Matches text-embedding-3-small, the model the shipped application.yml names. */ + public static final int DIMS = 1536; + + private Stores() { + } + + public static SimpleVectorStore simple(EmbeddingModel embeddings) { + return SimpleVectorStore.builder(embeddings).build(); + } + + public static JdbcTemplate jdbc() { + return new JdbcTemplate(new DriverManagerDataSource(PG_URL, PG_USER, PG_PASSWORD)); + } + + /** A pgvector store with a freshly created, empty table and an HNSW cosine index. */ + public static PgVectorStore pg(JdbcTemplate jdbc, EmbeddingModel embeddings) { + PgVectorStore store = PgVectorStore.builder(jdbc, embeddings) + .dimensions(DIMS) + .distanceType(PgVectorStore.PgDistanceType.COSINE_DISTANCE) + .indexType(PgVectorStore.PgIndexType.HNSW) + .removeExistingVectorStoreTable(true) + .initializeSchema(true) + .build(); + store.afterPropertiesSet(); + return store; + } + + private static String env(String name, String fallback) { + String value = System.getenv(name); + return value == null || value.isBlank() ? fallback : value; + } +} diff --git a/rag/src/test/java/com/ankurm/rag/support/Transcript.java b/rag/src/test/java/com/ankurm/rag/support/Transcript.java new file mode 100644 index 0000000..fb127d7 --- /dev/null +++ b/rag/src/test/java/com/ankurm/rag/support/Transcript.java @@ -0,0 +1,52 @@ +package com.ankurm.rag.support; + +import java.io.IOException; +import java.io.PrintWriter; +import java.io.StringWriter; +import java.nio.file.Files; +import java.nio.file.Path; + +/** + * Writes a numbered transcript under {@code docs/output/} and echoes it to the console. + * Every console block quoted in the article comes out of one of these files verbatim. + */ +public final class Transcript implements AutoCloseable { + + private final Path path; + private final StringWriter buffer = new StringWriter(); + private final PrintWriter out = new PrintWriter(buffer); + + public Transcript(String fileName, String title) { + this.path = Path.of("docs", "output", fileName); + out.println("# " + title); + out.println(); + } + + public Transcript line(String format, Object... args) { + out.println(args.length == 0 ? format : String.format(format, args)); + return this; + } + + public Transcript blank() { + out.println(); + return this; + } + + public Transcript section(String heading) { + out.println(); + out.println("--- " + heading + " ---"); + return this; + } + + @Override + public void close() { + out.flush(); + try { + Files.createDirectories(path.getParent()); + Files.writeString(path, buffer.toString()); + } catch (IOException e) { + throw new IllegalStateException("could not write " + path, e); + } + System.out.print(buffer); + } +}