Add rag-ingestion module: PDF, Word and HTML readers and chunking strategies compared by retrieval hit rate; metadata in the embedding, content-hash idempotent ingestion, re-embedding cost of an edit
Co-Authored-By: Claude Sonnet 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_0146opDKBYtADXvkc39zdiyw
This commit is contained in:
Claude
@@ -25,5 +25,6 @@ Runnable companion code for the Spring AI articles on [ankurm.com](https://ankur
|
||||
| [`semantic-cache/`](semantic-cache) | A `CallAdvisor` that serves a stored answer when a new question embeds close to an old one, over a Redis Stack vector index and a real in-process embedding model (all-MiniLM-L6-v2). A labelled set of 30 intents x 4 phrasings plus 20 off-topic questions swept across thresholds (hit rate against wrong answers), 600 requests replayed with an illustrative cost model, and the traps reproduced: Spring AI's score is `(1 + cosine) / 2`, reset-password and reset-2FA are close enough to answer for each other, a tenant filter, expiry, and an index built for another embedding model. The chat model is a script that counts calls; one test uses a real local model. Spring Boot 4.1.1, Spring AI 2.0.1, Java 25. | [Semantic Caching for LLM Calls in Spring Boot with Redis Vector Search](https://ankurm.com/semantic-caching-for-llm-calls-in-spring-boot-with-redis-vector-search/) |
|
||||
| [`langchain4j-interop/`](langchain4j-interop) | The same order assistant in Spring AI 2.0.1 and LangChain4j 1.22.0 in one Spring Boot 4.1.1 app, against a local fake OpenAI server: the bean collision when both OpenAI starters are present, the two requests compared on the wire, a throwing tool and a looping model (what each framework bounds), the meters each records, one MCP tool server used from both, and which lines of the 2025 version of the article still compile. Java 25. | [Spring AI vs LangChain4j in 2026: Is the Pairing Still Worth It?](https://ankurm.com/spring-ai-langchain4j-production-ai-microservices-java/) |
|
||||
| [`agent-skills/`](agent-skills) | `SKILL.md` folders loaded on demand by the `Skill` tool from `spring-ai-agent-utils` 0.13.0: what the model sees before it loads anything (name and description only), the two-call load round trip, unknown, duplicate and description-less skills, and listing size against inlining every body (character counts). Scripted chat model, no live model. Spring Boot 4.1.1, Spring AI 2.0.1, Java 25. | [Agent Skills in Spring AI: Packaging Reusable Capabilities for Your Agents](https://ankurm.com/agent-skills-spring-ai-packaging-reusable-capabilities/) |
|
||||
| [`rag-ingestion/`](rag-ingestion) | PDF, Word and HTML versions of one 12-section runbook through `PagePdfDocumentReader`, `TikaDocumentReader` and `JsoupDocumentReader`, then `TokenTextSplitter` at six sizes, a paragraph chunker, a sentence splitter and a section chunker; retrieval hit@1/3/5 on 37 fixed questions, metadata in the embedding, content-hash idempotent ingestion into pgvector and the re-embedding cost of one edit. Local all-MiniLM-L6-v2; a synthetic corpus, so differences of two or three questions are noise. Spring Boot 4.1.2, Spring AI 2.0.2, Java 25. | [Spring AI RAG Ingestion Pipeline: PDF, Word and HTML Readers with Chunking Strategies Compared](https://ankurm.com/spring-ai-rag-ingestion-pipeline-pdf-word-html-chunking/) |
|
||||
|
||||
Upgrading from Spring AI 1.x: [migration guide](https://ankurm.com/spring-ai-1-to-2-migration-guide/).
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
target/
|
||||
*.iml
|
||||
.idea/
|
||||
@@ -0,0 +1,7 @@
|
||||
# rag-ingestion
|
||||
|
||||
Companion code for [Spring AI RAG Ingestion Pipeline: PDF, Word and HTML Readers with Chunking Strategies Compared](https://ankurm.com/spring-ai-rag-ingestion-pipeline-pdf-word-html-chunking/).
|
||||
|
||||
PDF, Word and HTML versions of one 12-section runbook go through `PagePdfDocumentReader`, `TikaDocumentReader` and `JsoupDocumentReader`, then through several chunkers, and retrieval hit rate is measured on 37 fixed questions. Also: metadata in the embedding, content-hash ingestion, and what one edit costs to re-embed. Spring Boot 4.1.2, Spring AI 2.0.2, Java 25.
|
||||
|
||||
Run `scripts/run-all.sh` (needs PostgreSQL 16 with pgvector; see the script header). Every number in the article is a line in `output/`.
|
||||
@@ -0,0 +1,38 @@
|
||||
# What each reader returns for the same 36-paragraph runbook
|
||||
|
||||
corpus: 10665 characters of body text (title, 12 headings, 36 paragraphs)
|
||||
|
||||
--- PagePdfDocumentReader, default config ---
|
||||
Documents returned: 4 (one per PDF page)
|
||||
metadata of page 1: {file_name=runbook.pdf, page_number=1}
|
||||
pages whose text contains the running footer: 4 of 4
|
||||
characters as read: 30358, after collapsing runs of whitespace: 10812 (the corpus itself is 10665)
|
||||
page 1 ends with: " Helios Operations Runbook - Confidential - Page 1"
|
||||
page 2 starts with: "they receive every page but are not expected to act, and they write up what"
|
||||
paragraphs that no single page contains whole, because a page break cuts them: 2 of 36
|
||||
|
||||
--- PagePdfDocumentReader with pageBottomMargin ---
|
||||
pageBottomMargin= 30: footer on 4 of 4 pages, answer fragments still in the text: 37 of 37
|
||||
pageBottomMargin= 60: footer on 0 of 4 pages, answer fragments still in the text: 37 of 37
|
||||
pageBottomMargin=100: footer on 0 of 4 pages, answer fragments still in the text: 36 of 37
|
||||
no margin: footer on 4 of 4 pages, answer fragments still in the text: 37 of 37
|
||||
|
||||
--- TikaDocumentReader on the .docx ---
|
||||
Documents returned: 1
|
||||
metadata: {source=runbook.docx}
|
||||
characters: 10657, contains the footer text: true
|
||||
starts with: "Helios Cloud Platform Operations Runbook\n1. Backups and Restore\nEvery production databas"
|
||||
blank-line-separated blocks in the text: 1, newline-separated lines: 50
|
||||
answer fragments in the text: 37 of 37
|
||||
|
||||
--- TikaDocumentReader on the same PDF and HTML ---
|
||||
.pdf -> 1 Document, 10861 characters, metadata keys [source]
|
||||
.html -> 1 Document, 10764 characters, contains "Cookie settings": true, contains tracking script text: false
|
||||
|
||||
--- JsoupDocumentReader on the .html ---
|
||||
default config: 1 Document, 10712 characters, contains nav link text "Status": true, contains "Cookie settings": true
|
||||
default metadata: {description=How Helios runs production., title=Helios Cloud Platform Operations Runbook}
|
||||
selector=article: 1 Document, 10615 characters, nav text: false, footer text: false
|
||||
selector=section, groupByElement=true: 12 Documents
|
||||
first section Document: 1188 characters, starts "1. Backups and Restore Every production database is snapshot"
|
||||
with metadataTag("description") and additionalMetadata: {description=How Helios runs production., format=html, title=Helios Cloud Platform Operations Runbook}
|
||||
@@ -0,0 +1,23 @@
|
||||
# TokenTextSplitter chunk size against retrieval hit rate (all-MiniLM-L6-v2, 37 questions)
|
||||
|
||||
source: runbook.docx read by TikaDocumentReader, 2172 cl100k tokens, 37 questions, embedding model all-MiniLM-L6-v2 (max 256 WordPiece tokens)
|
||||
hit@k: the question's answer is inside one of the k best chunks. cut: answers that no chunk contains whole.
|
||||
top3Toks: tokens in the three best chunks, the context a prompt would carry per question.
|
||||
|
||||
--- defaults except chunkSize (minChunkSizeChars=350) ---
|
||||
chunkSize chunks meanToks hit@1 hit@3 hit@5 cut top3Toks
|
||||
32 68 31.9 22/37 29/37 30/37 3 96
|
||||
64 34 63.9 27/37 33/37 33/37 2 191
|
||||
128 19 114.1 31/37 35/37 36/37 0 335
|
||||
256 9 241.1 24/37 31/37 36/37 0 720
|
||||
512 5 434.0 21/37 33/37 37/37 0 1278
|
||||
800 3 723.3 17/37 37/37 37/37 0 2170
|
||||
|
||||
--- same sizes with minChunkSizeChars=10 ---
|
||||
chunkSize chunks meanToks hit@1 hit@3 hit@5 cut top3Toks
|
||||
32 88 24.6 31/37 32/37 32/37 0 76
|
||||
64 42 51.7 27/37 32/37 34/37 1 160
|
||||
128 19 114.1 31/37 35/37 36/37 0 335
|
||||
256 9 241.1 24/37 31/37 36/37 0 720
|
||||
512 5 434.0 21/37 33/37 37/37 0 1278
|
||||
800 3 723.3 17/37 37/37 37/37 0 2170
|
||||
@@ -0,0 +1,18 @@
|
||||
# Chunking strategies on the same runbook (all-MiniLM-L6-v2, 37 questions)
|
||||
|
||||
strategy chunks meanToks hit@1 hit@3 hit@5 cut top3Toks
|
||||
TokenTextSplitter 128 tokens 19 114.1 31/37 35/37 36/37 0 335
|
||||
TokenTextSplitter 64 tokens, minChars=10 42 51.7 27/37 32/37 34/37 1 160
|
||||
SentenceSplitter 128 tokens, no overlap 20 108.5 24/37 33/37 34/37 0 341
|
||||
SentenceSplitter 128 tokens, 1 sentence overlap 24 113.5 30/37 33/37 36/37 0 337
|
||||
SentenceSplitter 64 tokens, no overlap 41 52.8 30/37 34/37 34/37 0 161
|
||||
one chunk per paragraph 37 56.4 34/37 36/37 36/37 0 176
|
||||
one chunk per paragraph + section heading 37 62.7 32/37 35/37 36/37 0 194
|
||||
one chunk per section (JsoupDocumentReader) 12 178.7 31/37 37/37 37/37 0 529
|
||||
|
||||
--- 37 chunks from 36 paragraphs ---
|
||||
last paragraph chunk: section=12. Decommissioning, text="Helios Operations Runbook - Confidential"
|
||||
|
||||
--- what a paragraph chunk with its heading looks like ---
|
||||
metadata: {chunk_index=2, section=1. Backups and Restore, source=runbook.docx}
|
||||
text: 1. Backups and Restore\nPoint-in-time recovery is available for the last 7 days. To request a recovery, the service owne
|
||||
@@ -0,0 +1,14 @@
|
||||
# Metadata and the text that gets embedded
|
||||
|
||||
chunk metadata: {chunk_index=0, content_hash=86f5d5577cede265, format=docx, section=1. Backups and Restore, source=runbook.docx}
|
||||
|
||||
--- the text each model would embed for the same chunk ---
|
||||
MetadataMode.NONE : \n\nEvery production database is snapshotted once a day at 01:30 UTC. Daily snapshots are kept for 35 days, af
|
||||
MetadataMode.EMBED: chunk_index: 0\nformat: docx\nsection: 1. Backups and Restore\nsource: runbook.docx\ncontent_hash: 86f5d5577ce
|
||||
|
||||
--- hit rate, one chunk per paragraph, 37 questions ---
|
||||
configuration chunks meanToks hit@1 hit@3 hit@5 cut top3Toks
|
||||
MetadataMode.NONE 37 56.4 34/37 36/37 36/37 0 176
|
||||
MetadataMode.EMBED, every key embedded 37 56.4 29/37 36/37 36/37 0 175
|
||||
MetadataMode.EMBED, four keys in excludedEmbedMetadataKeys 37 56.4 31/37 35/37 36/37 0 174
|
||||
embedded text with the four keys excluded: section: 1. Backups and Restore\n\nEvery production database is snapshotted once a day at 01:30 UTC. Daily sna
|
||||
@@ -0,0 +1,15 @@
|
||||
# One runbook, three formats, TokenTextSplitter(128), 37 questions
|
||||
|
||||
reader and configuration chunks meanToks hit@1 hit@3 hit@5 cut top3Toks
|
||||
PDF PagePdfDocumentReader, each page split alone 36 106.6 24/37 32/37 33/37 1 333
|
||||
PDF pages joined first, footer still there 34 112.9 26/37 33/37 34/37 1 338
|
||||
PDF pages joined, pageBottomMargin=60 34 110.5 27/37 33/37 34/37 1 330
|
||||
PDF pages joined, margin 60, whitespace collapsed 19 117.7 27/37 36/37 37/37 0 348
|
||||
DOCX TikaDocumentReader 19 114.1 31/37 35/37 36/37 0 335
|
||||
HTML JsoupDocumentReader, default selector (body) 19 114.8 30/37 34/37 36/37 0 342
|
||||
HTML JsoupDocumentReader, selector=article 19 113.8 29/37 34/37 36/37 0 342
|
||||
|
||||
--- tokens the splitter had to cut, by reader ---
|
||||
PDF pages as read : 30358 characters, 3933 tokens
|
||||
PDF whitespace collapsed: 10655 characters, 2251 tokens
|
||||
DOCX : 10657 characters, 2172 tokens
|
||||
@@ -0,0 +1,18 @@
|
||||
# Re-ingesting runbook.docx into pgvector 0.8.7 (all-MiniLM-L6-v2, 384 dims)
|
||||
|
||||
|
||||
--- a naive loop: read, split, add, with random chunk ids ---
|
||||
run 1: rows in table = 19, texts embedded so far = 19
|
||||
run 2: rows in table = 38, texts embedded so far = 38
|
||||
run 3: rows in table = 57, texts embedded so far = 57
|
||||
rows saying "kept for 35 days": 3
|
||||
|
||||
--- TokenTextSplitter(128): windows of tokens ---
|
||||
1. first ingest of v1 chunks=19 embedded=19 unchanged= 0 deleted= 0 | rows=19, texts embedded in total=19
|
||||
2. same file again chunks=19 embedded= 0 unchanged=19 deleted= 0 | rows=19, texts embedded in total=19
|
||||
3. same file a third time chunks=19 embedded= 0 unchanged=19 deleted= 0 | rows=19, texts embedded in total=19
|
||||
4. v2: one paragraph edited chunks=19 embedded= 1 unchanged=18 deleted= 1 | rows=19, texts embedded in total=20
|
||||
rows saying "kept for 35 days": 0, rows saying "kept for 45 days": 1
|
||||
best match for the snapshot question now contains "45 days": true
|
||||
5. v3: section 12 deleted from the file chunks=17 embedded= 1 unchanged=16 deleted= 3 | rows=17, texts embedded in total=22
|
||||
rows saying "180 days in advance" (a fact only section 12 holds): 0
|
||||
@@ -0,0 +1,12 @@
|
||||
# Same sequence with one chunk per paragraph
|
||||
|
||||
|
||||
--- HeadingAwareChunker: one chunk per paragraph ---
|
||||
1. first ingest of v1 chunks=37 embedded=37 unchanged= 0 deleted= 0 | rows=37, texts embedded in total=37
|
||||
2. same file again chunks=37 embedded= 0 unchanged=37 deleted= 0 | rows=37, texts embedded in total=37
|
||||
3. same file a third time chunks=37 embedded= 0 unchanged=37 deleted= 0 | rows=37, texts embedded in total=37
|
||||
4. v2: one paragraph edited chunks=37 embedded= 1 unchanged=36 deleted= 1 | rows=37, texts embedded in total=38
|
||||
rows saying "kept for 35 days": 0, rows saying "kept for 45 days": 1
|
||||
best match for the snapshot question now contains "45 days": true
|
||||
5. v3: section 12 deleted from the file chunks=34 embedded= 0 unchanged=34 deleted= 3 | rows=34, texts embedded in total=39
|
||||
rows saying "180 days in advance" (a fact only section 12 holds): 0
|
||||
@@ -0,0 +1,9 @@
|
||||
# Chunks to embed again after appending one sentence to one paragraph (all 36 paragraphs tried)
|
||||
|
||||
splitter chunks mean median max
|
||||
TokenTextSplitter 128 tokens 19 5.0 2 17
|
||||
TokenTextSplitter 256 tokens 9 2.5 2 10
|
||||
SentenceSplitter 128, no overlap 19 2.4 2 8
|
||||
SentenceSplitter 128, 1 sentence overlap 24 3.1 2 13
|
||||
one chunk per paragraph 36 1.0 1 1
|
||||
columns: chunks in the unedited file; mean, median and max chunks re-embedded over the 36 edits
|
||||
@@ -0,0 +1,29 @@
|
||||
# Filtering retrieval by the metadata written at ingestion
|
||||
|
||||
rows in table: 73 (runbook.docx and runbook.html, one chunk per paragraph)
|
||||
JsoupDocumentReader selector=article: 1 line(s) of text -> 0 chunks from the paragraph chunker
|
||||
selector="article h1, article h2, article p" with separator "\n": 49 lines -> 36 chunks
|
||||
rows per format: [docx=37, html=36]
|
||||
question: "How long is data kept?"
|
||||
|
||||
--- no filter, top 5 ---
|
||||
0.640 html 12. Decommissioning After a customer contract ends, the customer's data is del
|
||||
0.640 docx 12. Decommissioning After a customer contract ends, the customer's data is del
|
||||
0.588 html 1. Backups and Restore Every production database is snapshotted once a day at 01:
|
||||
0.588 docx 1. Backups and Restore Every production database is snapshotted once a day at 01:
|
||||
0.569 html 9. Logging and Monitoring Metrics are stored at 10 second resolution for 7 days, the
|
||||
(5 results)
|
||||
|
||||
--- filter: section == '9. Logging and Monitoring' ---
|
||||
0.569 docx 9. Logging and Monitoring Metrics are stored at 10 second resolution for 7 days, the
|
||||
0.569 html 9. Logging and Monitoring Metrics are stored at 10 second resolution for 7 days, the
|
||||
0.514 docx 9. Logging and Monitoring Application logs are searchable for 90 days in the hot tie
|
||||
0.514 html 9. Logging and Monitoring Application logs are searchable for 90 days in the hot tie
|
||||
0.188 docx 9. Logging and Monitoring To keep paging humane, each on-call shift has a noise budg
|
||||
(5 results)
|
||||
|
||||
--- filter: format == 'html' && section == '1. Backups and Restore' ---
|
||||
0.588 html 1. Backups and Restore Every production database is snapshotted once a day at 01:
|
||||
0.468 html 1. Backups and Restore Point-in-time recovery is available for the last 7 days. T
|
||||
0.254 html 1. Backups and Restore A backup that has never been restored is only a hope. On t
|
||||
(3 results)
|
||||
@@ -0,0 +1,66 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
|
||||
<parent>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-parent</artifactId>
|
||||
<version>4.1.2</version>
|
||||
<relativePath/>
|
||||
</parent>
|
||||
|
||||
<groupId>com.ankurm</groupId>
|
||||
<artifactId>rag-ingestion</artifactId>
|
||||
<version>1.0.0</version>
|
||||
<name>rag-ingestion</name>
|
||||
<description>The ETL side of RAG on Spring AI 2.0: PDF, Word and HTML readers, chunking strategies compared by retrieval hit rate on one fixed question set, metadata enrichment and idempotent re-ingestion into pgvector.</description>
|
||||
|
||||
<properties>
|
||||
<java.version>25</java.version>
|
||||
<spring-ai.version>2.0.2</spring-ai.version>
|
||||
</properties>
|
||||
|
||||
<dependencyManagement>
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-bom</artifactId>
|
||||
<version>${spring-ai.version}</version>
|
||||
<type>pom</type>
|
||||
<scope>import</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
</dependencyManagement>
|
||||
|
||||
<dependencies>
|
||||
<dependency><groupId>org.springframework.ai</groupId><artifactId>spring-ai-pdf-document-reader</artifactId></dependency>
|
||||
<dependency><groupId>org.springframework.ai</groupId><artifactId>spring-ai-tika-document-reader</artifactId></dependency>
|
||||
<dependency><groupId>org.springframework.ai</groupId><artifactId>spring-ai-jsoup-document-reader</artifactId></dependency>
|
||||
<!-- Store module, not the starter: the store is built by hand so every setting is visible. -->
|
||||
<dependency><groupId>org.springframework.ai</groupId><artifactId>spring-ai-pgvector-store</artifactId></dependency>
|
||||
<!-- all-MiniLM-L6-v2 in-process through ONNX Runtime: a real embedding model, no API key. -->
|
||||
<dependency><groupId>org.springframework.ai</groupId><artifactId>spring-ai-transformers</artifactId></dependency>
|
||||
<dependency><groupId>org.postgresql</groupId><artifactId>postgresql</artifactId></dependency>
|
||||
<dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-jdbc</artifactId></dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-test</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-surefire-plugin</artifactId>
|
||||
<configuration>
|
||||
<argLine>-Duser.timezone=UTC -Dstdout.encoding=UTF-8 -Dfile.encoding=UTF-8 -Xmx3g</argLine>
|
||||
</configuration>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
</project>
|
||||
Executable
+10
@@ -0,0 +1,10 @@
|
||||
#!/usr/bin/env bash
|
||||
# Regenerates every file in output/ by running the test suite.
|
||||
# Needs: JDK 25, Maven, internet for the first run (about 90 MB for the MiniLM model, cached in ~/.cache/minilm),
|
||||
# and a PostgreSQL 16 with pgvector 0.8 on localhost:5432 (database ragdemo, user demo / demo) for the two ingestion tests.
|
||||
# Set RAG_DB_URL to point somewhere else.
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
rm -rf output
|
||||
mvn -B -q test
|
||||
ls output
|
||||
Executable
+7
@@ -0,0 +1,7 @@
|
||||
#!/usr/bin/env bash
|
||||
# One way to get the database the ingestion tests expect: a pgvector container on :5432.
|
||||
set -euo pipefail
|
||||
docker run -d --name ragdemo-pg -e POSTGRES_USER=demo -e POSTGRES_PASSWORD=demo -e POSTGRES_DB=ragdemo \
|
||||
-p 5432:5432 pgvector/pgvector:0.8.1-pg16
|
||||
until docker exec ragdemo-pg pg_isready -U demo -d ragdemo >/dev/null 2>&1; do sleep 1; done
|
||||
echo "ready: jdbc:postgresql://localhost:5432/ragdemo (demo / demo)"
|
||||
@@ -0,0 +1,137 @@
|
||||
package com.ankurm.ragingestion;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* The one document every experiment in this module reads: a made-up operations runbook with 12 sections
|
||||
* and 36 paragraphs, plus 37 questions whose answers sit in exactly one paragraph each.
|
||||
*
|
||||
* <p>It is written out as a PDF, a Word file and an HTML page (see the test fixtures), so the readers and
|
||||
* the chunkers are compared on identical content. A question counts as answered when a retrieved chunk
|
||||
* contains its {@code answer} fragment, and {@code CorpusTest} proves every fragment occurs exactly once.
|
||||
*/
|
||||
public final class Corpus {
|
||||
|
||||
public static final String TITLE = "Helios Cloud Platform Operations Runbook";
|
||||
|
||||
public record Section(String title, List<String> paragraphs) {
|
||||
}
|
||||
|
||||
public record Question(String text, String answer) {
|
||||
}
|
||||
|
||||
public static final List<Section> SECTIONS = List.of(
|
||||
new Section("1. Backups and Restore", List.of(
|
||||
"Every production database is snapshotted once a day at 01:30 UTC. Daily snapshots are kept for 35 days, after which they are deleted automatically. Weekly snapshots, taken on Sundays, are kept for 12 weeks. All snapshots are encrypted with AES-256 using keys held in the regional key management service, and each snapshot is copied to the eu-west-2 region within one hour of completion.",
|
||||
"A backup that has never been restored is only a hope. On the first Monday of every quarter the storage team restores the newest snapshot of three randomly chosen databases into an isolated account and runs the integrity checklist against it. A restore test passes only if the restored database starts, replays its write-ahead log without errors and returns the expected row counts for the canary tables.",
|
||||
"Point-in-time recovery is available for the last 7 days. To request a recovery, the service owner opens a ticket in the STORAGE queue with the target timestamp in UTC. The storage team aims to start the recovery within 2 hours of a request and will always restore into a new instance, never over the live one, so that the owner can compare the data before switching traffic.")),
|
||||
new Section("2. Incident Severity and Response", List.of(
|
||||
"Incidents are classified by customer impact. A Sev1 is a full outage of a customer-facing service or any confirmed loss of customer data. A Sev2 is a partial outage or a degradation that affects more than 10 percent of requests. A Sev3 is anything else that needs a human but does not hurt customers right now. The incident commander decides the severity and may raise or lower it at any time during the incident.",
|
||||
"The response targets are fixed: for a Sev1 a responder must be working within 10 minutes of the page, for a Sev2 within 30 minutes and for a Sev3 by the end of the next business day. Sev1 incidents get a status page update within 20 minutes and then every 30 minutes until resolved. The communications lead, not the incident commander, writes those updates.",
|
||||
"Every Sev1 and Sev2 gets a written post-incident review within 5 business days. The review is blameless, lists the contributing factors, and ends with action items that each have an owner and a due date. Action items that are still open after 60 days are escalated to the director of engineering.")),
|
||||
new Section("3. On-Call Rotation", List.of(
|
||||
"Each service has a primary and a secondary on-call engineer. The rotation changes every Thursday at 10:00 UTC, and the outgoing engineer gives a 30 minute verbal handover covering open alerts, risky changes and anything that felt unusual during the week. Nobody may be on call for more than 2 consecutive weeks without written approval from their manager.",
|
||||
"If the primary does not acknowledge a page within 5 minutes it escalates to the secondary, and after a further 5 minutes to the engineering manager. On-call engineers are paid an allowance of 120 euros for each week on the rotation, plus time off in lieu for any night call that lasts longer than 1 hour.",
|
||||
"New engineers shadow the rotation for two full weeks before taking the pager. During shadowing they receive every page but are not expected to act, and they write up what they would have done so that the primary can review it afterwards.")),
|
||||
new Section("4. Deployments and Releases", List.of(
|
||||
"Production deployments are blocked every Friday after 14:00 UTC and for the whole period from 20 December to 3 January. An emergency change during a freeze needs approval from the on-call incident commander and is recorded in the change log with the reason for the exception.",
|
||||
"Every release starts as a canary that receives 5 percent of traffic for 20 minutes. If the error rate of the canary rises more than 2 percent above the baseline in that window the pipeline rolls back automatically without waiting for a human. If the canary stays healthy it is promoted in steps of 25 percent, with a 10 minute pause between steps.",
|
||||
"Database migrations are shipped separately from application code and must be backwards compatible with the previous release, because a rollback restores only the code. A migration that rewrites more than 1 million rows must run as a background job with a throttle and must be announced in the platform channel one day in advance.")),
|
||||
new Section("5. Access Control", List.of(
|
||||
"Engineers reach production only through short-lived certificates issued by the access broker. A certificate is valid for 8 hours and cannot be renewed without signing in again with a hardware security key. Passwords and SMS codes are not accepted for any production system.",
|
||||
"Access rights are reviewed every quarter by the owning team lead, and rights that were not used in the previous 90 days are removed automatically. When an engineer changes teams, their previous team's access is removed on the same day and not at the next review.",
|
||||
"The break-glass account exists for the case where the access broker itself is down. Its token expires after 60 minutes, every use pages the security team immediately, and the engineer who used it must file a written justification within 24 hours.")),
|
||||
new Section("6. Capacity and Autoscaling", List.of(
|
||||
"Services scale on CPU utilisation with a target of 65 percent. After a scale-out the autoscaler waits 180 seconds before it considers another one, and after a scale-in it waits 600 seconds, so that a short burst cannot make the fleet flap up and down.",
|
||||
"Each region keeps reserved headroom of 30 percent above the busiest hour of the last month. Capacity is validated by a load test on the first Wednesday of the month at 3 times the observed peak, and a service that cannot sustain it for 15 minutes gets a capacity ticket against the owning team.",
|
||||
"Stateful services are scaled by hand because adding a node triggers data rebalancing. The owner schedules the change outside peak hours, watches rebalancing progress on the storage dashboard and does not start a second change until the first one has finished.")),
|
||||
new Section("7. Databases", List.of(
|
||||
"The standard database is PostgreSQL 16 with one primary and two replicas in separate availability zones. Replication lag above 15 seconds raises a warning and lag above 60 seconds pages the on-call engineer, because a replica that far behind cannot be promoted safely.",
|
||||
"Automatic failover must complete within 90 seconds. Services reach the database through a connection pooler, so they reconnect on their own, and each service may open at most 200 connections in total. A service that needs more must use the pooler's transaction mode instead of raising the limit.",
|
||||
"Maintenance tasks such as vacuum and index rebuilds run only in the window from 02:00 to 04:00 UTC. Schema changes that need an exclusive lock are not allowed in production at all; the owner must use the online migration tool, which copies the table in the background and swaps it atomically.")),
|
||||
new Section("8. Networking and Certificates", List.of(
|
||||
"All workloads run in private subnets. Outbound traffic leaves through NAT gateways with fixed addresses so that partners can allow-list them, and inbound traffic arrives only through the shared load balancers. The minimum accepted protocol version is TLS 1.3 for every internal and external endpoint.",
|
||||
"Certificates are renewed automatically 30 days before they expire. If a renewal fails, an alert goes to the platform team on the first day and pages them when fewer than 10 days remain, because an expired certificate is the most common cause of a self-inflicted outage.",
|
||||
"DNS records used for failover have a time to live of 60 seconds. All other records use 3600 seconds. Changing a TTL ahead of a planned migration must be done at least one full old-TTL period before the migration starts, so that resolvers have picked up the shorter value.")),
|
||||
new Section("9. Logging and Monitoring", List.of(
|
||||
"Application logs are searchable for 90 days in the hot tier and then archived for a further 400 days in object storage, where they can be restored on request. Logs that contain personal data are redacted at the agent before they leave the host.",
|
||||
"Metrics are stored at 10 second resolution for 7 days, then downsampled to 1 minute resolution for 13 months. Dashboards belong to the service teams, and a dashboard that nobody has opened for 6 months is archived.",
|
||||
"To keep paging humane, each on-call shift has a noise budget of no more than 5 pages. A team that exceeds the budget two shifts in a row must spend the next sprint reducing alert noise before taking new feature work.")),
|
||||
new Section("10. Cost Management", List.of(
|
||||
"Finance and engineering review cloud spend together on the third working day of each month. Any service whose spend is more than 20 percent above its forecast must present a short explanation and a plan at that review.",
|
||||
"Resources without an owner tag are shut down after 14 days of being idle and deleted after another 14 days. The tagging policy is enforced at provisioning time, so an untagged resource can only exist if someone bypassed the pipeline.",
|
||||
"The target for committed-use discounts is to cover 70 percent of steady-state compute. Coverage above 85 percent is discouraged because it removes the ability to shrink the fleet when demand falls.")),
|
||||
new Section("11. Compliance and Vendors", List.of(
|
||||
"The platform is audited for SOC 2 Type II every October, and the auditors receive read-only access to the evidence repository for the whole of that month. Penetration tests by an external firm are carried out twice a year, and every finding rated high or critical must be fixed within 30 days.",
|
||||
"Customers on the Regulated tier have their data stored and processed only in EU regions. A new subprocessor may not receive customer data until a data processing agreement is signed, and the public subprocessor list is updated within 10 business days of any change.",
|
||||
"Employees complete the security awareness course in their first week and again every 12 months. Completion is tracked in the learning system, and access to production is suspended for anyone more than 30 days overdue.")),
|
||||
new Section("12. Decommissioning", List.of(
|
||||
"A service can be deprecated only after its owner has announced the retirement date at least 180 days in advance and has contacted every known consumer directly. The platform team keeps a calendar of retirement dates so that two large migrations do not land in the same month.",
|
||||
"After a customer contract ends, the customer's data is deleted within 30 days and the deletion is confirmed to the customer in writing. Backups that contain the data age out under the normal retention schedule and are not restored for any reason in the meantime.",
|
||||
"When a service is switched off, its DNS records are removed after 14 days of zero traffic, and its credentials are revoked on the same day it stops serving. A final check confirms that no alerts, dashboards or scheduled jobs still reference the retired service.")));
|
||||
|
||||
public static final List<Question> QUESTIONS = List.of(
|
||||
new Question("How long do we keep the daily database snapshots before they are deleted?", "kept for 35 days"),
|
||||
new Question("How often is a restore test performed?", "first Monday of every quarter"),
|
||||
new Question("How far back can a database be recovered to a chosen moment?", "last 7 days"),
|
||||
new Question("How quickly does someone have to start working on the most serious kind of incident?", "within 10 minutes of the page"),
|
||||
new Question("By when must a post-incident review be written?", "within 5 business days"),
|
||||
new Question("How frequently are customers updated on the status page during a major outage?", "every 30 minutes until resolved"),
|
||||
new Question("At what time does the on-call duty change hands each week?", "every Thursday at 10:00 UTC"),
|
||||
new Question("What extra money does an on-call engineer receive?", "120 euros"),
|
||||
new Question("What happens if nobody answers a page for five minutes?", "does not acknowledge a page within 5 minutes"),
|
||||
new Question("On which days and times are releases to production not permitted?", "every Friday after 14:00 UTC"),
|
||||
new Question("What proportion of users sees a new release first, and for how long?", "5 percent of traffic for 20 minutes"),
|
||||
new Question("When does a release get undone without human involvement?", "rises more than 2 percent above the baseline"),
|
||||
new Question("How long does a production login certificate last?", "valid for 8 hours"),
|
||||
new Question("What is the consequence of using the emergency account?", "every use pages the security team"),
|
||||
new Question("After how long does the emergency account's token stop working?", "expires after 60 minutes"),
|
||||
new Question("What CPU load do we aim for when scaling services automatically?", "target of 65 percent"),
|
||||
new Question("How much spare capacity does each region hold?", "reserved headroom of 30 percent"),
|
||||
new Question("How long does the autoscaler wait after removing instances?", "waits 600 seconds"),
|
||||
new Question("When does database replication delay start to raise a warning?", "lag above 15 seconds raises a warning"),
|
||||
new Question("How quickly must a database failover complete?", "within 90 seconds"),
|
||||
new Question("What is the cap on database connections for one service?", "at most 200 connections"),
|
||||
new Question("How long before expiry are TLS certificates replaced?", "renewed automatically 30 days before"),
|
||||
new Question("How long can resolvers cache DNS answers for failover names?", "time to live of 60 seconds"),
|
||||
new Question("What is the oldest TLS version we accept?", "TLS 1.3 for every"),
|
||||
new Question("For how long can I search recent application logs?", "searchable for 90 days"),
|
||||
new Question("What is the page limit for a single on-call shift?", "no more than 5 pages"),
|
||||
new Question("How fine-grained are metrics during their first week?", "10 second resolution"),
|
||||
new Question("When do finance and engineering meet to discuss cloud spend?", "third working day of each month"),
|
||||
new Question("How far above forecast does spending have to go before a team must explain it?", "more than 20 percent above its forecast"),
|
||||
new Question("How much of our steady compute should be covered by committed-use discounts?", "cover 70 percent of steady-state compute"),
|
||||
new Question("In which month does the SOC 2 audit take place?", "SOC 2 Type II every October"),
|
||||
new Question("What is the deadline for fixing serious penetration test findings?", "fixed within 30 days"),
|
||||
new Question("Where may data of customers on the Regulated tier live?", "only in EU regions"),
|
||||
new Question("How much warning must be given before a service is retired?", "at least 180 days in advance"),
|
||||
new Question("How soon after a contract ends is the customer's data erased?", "deleted within 30 days and"),
|
||||
new Question("When are the DNS records of a switched-off service removed?", "removed after 14 days of zero traffic"),
|
||||
new Question("What kind of new subprocessors need a signed agreement before getting data?", "data processing agreement is signed"));
|
||||
|
||||
/** The whole runbook as one string: title, then each section heading and paragraphs separated by blank lines. */
|
||||
public static String fullText() {
|
||||
StringBuilder sb = new StringBuilder(TITLE).append("\n\n");
|
||||
for (Section s : SECTIONS) {
|
||||
sb.append(s.title()).append("\n\n");
|
||||
for (String p : s.paragraphs()) {
|
||||
sb.append(p).append("\n\n");
|
||||
}
|
||||
}
|
||||
return sb.toString();
|
||||
}
|
||||
|
||||
/** Title, then each section heading followed by its paragraphs, in reading order. */
|
||||
public static List<String> blocks() {
|
||||
java.util.List<String> blocks = new java.util.ArrayList<>();
|
||||
blocks.add(TITLE);
|
||||
for (Section s : SECTIONS) {
|
||||
blocks.add(s.title());
|
||||
blocks.addAll(s.paragraphs());
|
||||
}
|
||||
return blocks;
|
||||
}
|
||||
|
||||
private Corpus() {
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
package com.ankurm.ragingestion;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.regex.Pattern;
|
||||
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.ai.document.DocumentTransformer;
|
||||
|
||||
/**
|
||||
* One chunk per paragraph, each tagged with the section heading it sits under. Line-oriented text (what the
|
||||
* Tika reader returns for a Word file) is read as: a short numbered line with no full stop is a heading, any
|
||||
* other line is a paragraph. With {@code prefixHeading} the heading is also written in front of the paragraph,
|
||||
* so the embedding sees which section the paragraph belongs to.
|
||||
*/
|
||||
public class HeadingAwareChunker implements DocumentTransformer {
|
||||
|
||||
private static final Pattern HEADING = Pattern.compile("^\\d+\\. [^.]{1,60}$");
|
||||
|
||||
private final boolean prefixHeading;
|
||||
|
||||
public HeadingAwareChunker(boolean prefixHeading) {
|
||||
this.prefixHeading = prefixHeading;
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<Document> apply(List<Document> documents) {
|
||||
List<Document> out = new ArrayList<>();
|
||||
for (Document doc : documents) {
|
||||
String section = "";
|
||||
int index = 0;
|
||||
for (String raw : doc.getText().split("\\R")) {
|
||||
String line = raw.replaceAll("\\s+", " ").strip();
|
||||
if (line.isEmpty()) {
|
||||
continue;
|
||||
}
|
||||
if (HEADING.matcher(line).matches()) {
|
||||
section = line;
|
||||
continue;
|
||||
}
|
||||
if (section.isEmpty()) {
|
||||
continue; // the document title, before the first section
|
||||
}
|
||||
Map<String, Object> metadata = new LinkedHashMap<>(doc.getMetadata());
|
||||
metadata.put("section", section);
|
||||
metadata.put("chunk_index", index++);
|
||||
out.add(Document.builder().text(prefixHeading ? section + "\n" + line : line).metadata(metadata).build());
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
package com.ankurm.ragingestion;
|
||||
|
||||
import java.nio.file.Path;
|
||||
import java.util.HashSet;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
import java.util.Set;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.ai.document.DocumentReader;
|
||||
import org.springframework.ai.document.DocumentTransformer;
|
||||
import org.springframework.ai.reader.jsoup.JsoupDocumentReader;
|
||||
import org.springframework.ai.reader.jsoup.config.JsoupDocumentReaderConfig;
|
||||
import org.springframework.ai.reader.pdf.PagePdfDocumentReader;
|
||||
import org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig;
|
||||
import org.springframework.ai.reader.tika.TikaDocumentReader;
|
||||
import org.springframework.ai.vectorstore.VectorStore;
|
||||
import org.springframework.core.io.FileSystemResource;
|
||||
import org.springframework.jdbc.core.JdbcTemplate;
|
||||
|
||||
/**
|
||||
* Reads one file, splits it, tags the chunks and writes them to a pgvector table so that running it again
|
||||
* on the same file is harmless:
|
||||
* <ul>
|
||||
* <li>chunks that already exist (same id) are skipped, so nothing is embedded twice;</li>
|
||||
* <li>chunks that exist in the table for this source but not in the new version are deleted;</li>
|
||||
* <li>only the new chunks are embedded and written.</li>
|
||||
* </ul>
|
||||
*/
|
||||
public class IngestionPipeline {
|
||||
|
||||
public record Result(int chunks, int embedded, int unchanged, int deleted) {
|
||||
}
|
||||
|
||||
private final VectorStore store;
|
||||
private final JdbcTemplate jdbc;
|
||||
private final String table;
|
||||
private final DocumentTransformer splitter;
|
||||
|
||||
public IngestionPipeline(VectorStore store, JdbcTemplate jdbc, String table, DocumentTransformer splitter) {
|
||||
this.store = store;
|
||||
this.jdbc = jdbc;
|
||||
this.table = table;
|
||||
this.splitter = splitter;
|
||||
}
|
||||
|
||||
public Result ingest(Path file) {
|
||||
String name = file.getFileName().toString();
|
||||
String format = name.substring(name.lastIndexOf('.') + 1).toLowerCase(Locale.ROOT);
|
||||
|
||||
// 1. read, then join every page or element into one text so chunk boundaries do not depend on the file's layout
|
||||
List<Document> read = reader(file, format).get();
|
||||
String text = read.stream().map(Document::getText).collect(Collectors.joining("\n"));
|
||||
List<Document> whole = List.of(new Document(text));
|
||||
|
||||
// 2. split and tag
|
||||
List<Document> chunks = new MetadataEnricher(name, format).apply(splitter.apply(whole));
|
||||
if (chunks.isEmpty() && !text.isBlank()) {
|
||||
// a splitter that does not understand the reader's text shape returns nothing; say so instead of writing zero rows
|
||||
throw new IllegalStateException(name + ": " + text.length() + " characters read, 0 chunks produced");
|
||||
}
|
||||
|
||||
// 3. work out what is already stored for this source
|
||||
Set<String> existing = new HashSet<>(jdbc.queryForList(
|
||||
"select id::text from " + table + " where metadata->>'source' = ?", String.class, name));
|
||||
Set<String> wanted = chunks.stream().map(Document::getId).collect(Collectors.toSet());
|
||||
|
||||
List<String> stale = existing.stream().filter(id -> !wanted.contains(id)).toList();
|
||||
if (!stale.isEmpty()) {
|
||||
store.delete(stale);
|
||||
}
|
||||
List<Document> fresh = chunks.stream().filter(c -> !existing.contains(c.getId())).toList();
|
||||
if (!fresh.isEmpty()) {
|
||||
store.add(fresh);
|
||||
}
|
||||
return new Result(chunks.size(), fresh.size(), chunks.size() - fresh.size(), stale.size());
|
||||
}
|
||||
|
||||
static DocumentReader reader(Path file, String format) {
|
||||
FileSystemResource resource = new FileSystemResource(file);
|
||||
return switch (format) {
|
||||
case "pdf" -> new PagePdfDocumentReader(resource, PdfDocumentReaderConfig.builder().withPageBottomMargin(60).build());
|
||||
// headings and paragraphs as separate lines: JsoupDocumentReader joins elements with the separator, and a line-oriented chunker needs the lines
|
||||
case "html", "htm" -> new JsoupDocumentReader(resource, JsoupDocumentReaderConfig.builder()
|
||||
.selector("article h1, article h2, article p").separator("\n").build());
|
||||
default -> new TikaDocumentReader(resource);
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
package com.ankurm.ragingestion;
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.security.MessageDigest;
|
||||
import java.security.NoSuchAlgorithmException;
|
||||
import java.util.ArrayList;
|
||||
import java.util.HexFormat;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.UUID;
|
||||
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.ai.document.DocumentTransformer;
|
||||
|
||||
/**
|
||||
* Gives every chunk the same metadata whatever reader produced it: {@code source}, {@code format},
|
||||
* {@code chunk_index} and a {@code content_hash}, and a deterministic id derived from them. The id is the
|
||||
* whole idempotency story: the same chunk of the same file always gets the same id, so writing it twice
|
||||
* is an upsert, and a chunk whose text changed gets a new id.
|
||||
*/
|
||||
public class MetadataEnricher implements DocumentTransformer {
|
||||
|
||||
private final String source;
|
||||
private final String format;
|
||||
|
||||
public MetadataEnricher(String source, String format) {
|
||||
this.source = source;
|
||||
this.format = format;
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<Document> apply(List<Document> chunks) {
|
||||
List<Document> out = new ArrayList<>();
|
||||
for (int i = 0; i < chunks.size(); i++) {
|
||||
Document chunk = chunks.get(i);
|
||||
String hash = sha256(chunk.getText()).substring(0, 16);
|
||||
Map<String, Object> metadata = new LinkedHashMap<>(chunk.getMetadata());
|
||||
metadata.put("source", source);
|
||||
metadata.put("format", format);
|
||||
metadata.put("chunk_index", i);
|
||||
metadata.put("content_hash", hash);
|
||||
String id = UUID.nameUUIDFromBytes((source + "#" + hash).getBytes(StandardCharsets.UTF_8)).toString();
|
||||
out.add(Document.builder().id(id).text(chunk.getText()).metadata(metadata).build());
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
static String sha256(String text) {
|
||||
try {
|
||||
return HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(text.getBytes(StandardCharsets.UTF_8)));
|
||||
} catch (NoSuchAlgorithmException e) {
|
||||
throw new IllegalStateException(e);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
package com.ankurm.ragingestion;
|
||||
|
||||
import java.util.Map;
|
||||
|
||||
import org.springframework.ai.document.MetadataMode;
|
||||
import org.springframework.ai.embedding.EmbeddingModel;
|
||||
import org.springframework.ai.transformers.TransformersEmbeddingModel;
|
||||
|
||||
/**
|
||||
* all-MiniLM-L6-v2 (384 dimensions) running inside the JVM through ONNX Runtime. Both model files come from
|
||||
* Hugging Face and are cached under the user's home directory, so the first run downloads about 90 MB and
|
||||
* later runs are offline. No API key is involved, which is what lets anyone repeat the measurements.
|
||||
*/
|
||||
public final class MiniLm {
|
||||
|
||||
private static final String HF = "https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/";
|
||||
private static EmbeddingModel shared;
|
||||
|
||||
private MiniLm() {
|
||||
}
|
||||
|
||||
/** The shared instance, which embeds the text of a Document only (MetadataMode.NONE, the constructor default). */
|
||||
public static synchronized EmbeddingModel get() {
|
||||
if (shared == null) {
|
||||
shared = create(MetadataMode.NONE);
|
||||
}
|
||||
return shared;
|
||||
}
|
||||
|
||||
/** A fresh instance whose Document embedding content is built with the given metadata mode. */
|
||||
public static EmbeddingModel create(MetadataMode mode) {
|
||||
try {
|
||||
TransformersEmbeddingModel m = new TransformersEmbeddingModel(mode);
|
||||
m.setTokenizerResource(HF + "tokenizer.json");
|
||||
m.setModelResource(HF + "onnx/model.onnx");
|
||||
m.setResourceCacheDirectory(System.getProperty("user.home") + "/.cache/minilm");
|
||||
m.setTokenizerOptions(Map.of("padding", "true", "truncation", "true", "maxLength", "256"));
|
||||
m.afterPropertiesSet();
|
||||
return m;
|
||||
} catch (Exception e) {
|
||||
throw new IllegalStateException(e);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
package com.ankurm.ragingestion;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.ai.document.DocumentTransformer;
|
||||
|
||||
/**
|
||||
* Groups whole sentences into chunks of about {@code targetTokens}, repeating the last
|
||||
* {@code overlapSentences} sentences at the start of the next chunk. Unlike TokenTextSplitter it never cuts
|
||||
* inside a sentence, so an answer that sits in one sentence cannot be torn in half by a chunk boundary.
|
||||
*/
|
||||
public class SentenceSplitter implements DocumentTransformer {
|
||||
|
||||
/** A full stop after a letter, followed by a capital: "UTC. Daily" splits, "1. Backups" does not. */
|
||||
private static final String SENTENCE_END = "(?<=[a-z\\)][.!?])\\s+(?=[A-Z])";
|
||||
|
||||
private final int targetTokens;
|
||||
private final int overlapSentences;
|
||||
|
||||
public SentenceSplitter(int targetTokens, int overlapSentences) {
|
||||
this.targetTokens = targetTokens;
|
||||
this.overlapSentences = overlapSentences;
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<Document> apply(List<Document> documents) {
|
||||
List<Document> out = new ArrayList<>();
|
||||
for (Document doc : documents) {
|
||||
List<String> sentences = new ArrayList<>();
|
||||
for (String line : doc.getText().split("\\R")) {
|
||||
String clean = line.replaceAll("\\s+", " ").strip();
|
||||
if (!clean.isEmpty()) {
|
||||
sentences.addAll(List.of(clean.split(SENTENCE_END)));
|
||||
}
|
||||
}
|
||||
List<String> window = new ArrayList<>();
|
||||
int tokens = 0;
|
||||
int index = 0;
|
||||
for (String sentence : sentences) {
|
||||
int t = Tokens.count(sentence);
|
||||
if (!window.isEmpty() && tokens + t > targetTokens) {
|
||||
out.add(chunk(doc, window, index++));
|
||||
List<String> keep = overlapSentences == 0 ? List.of()
|
||||
: window.subList(Math.max(0, window.size() - overlapSentences), window.size());
|
||||
window = new ArrayList<>(keep);
|
||||
tokens = window.stream().mapToInt(Tokens::count).sum();
|
||||
}
|
||||
window.add(sentence);
|
||||
tokens += t;
|
||||
}
|
||||
if (!window.isEmpty()) {
|
||||
out.add(chunk(doc, window, index));
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
private static Document chunk(Document parent, List<String> sentences, int index) {
|
||||
Map<String, Object> metadata = new LinkedHashMap<>(parent.getMetadata());
|
||||
metadata.put("chunk_index", index);
|
||||
return Document.builder().text(String.join(" ", sentences)).metadata(metadata).build();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
package com.ankurm.ragingestion;
|
||||
|
||||
import com.knuddels.jtokkit.Encodings;
|
||||
import com.knuddels.jtokkit.api.Encoding;
|
||||
import com.knuddels.jtokkit.api.EncodingType;
|
||||
|
||||
/** Token counts in the same encoding TokenTextSplitter uses by default (cl100k_base). */
|
||||
public final class Tokens {
|
||||
|
||||
private static final Encoding ENCODING = Encodings.newDefaultEncodingRegistry().getEncoding(EncodingType.CL100K_BASE);
|
||||
|
||||
private Tokens() {
|
||||
}
|
||||
|
||||
public static int count(String text) {
|
||||
return ENCODING.countTokens(text);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
package com.ankurm.ragingestion;
|
||||
|
||||
import java.nio.file.Path;
|
||||
import java.util.List;
|
||||
|
||||
import com.ankurm.ragingestion.support.Eval;
|
||||
import com.ankurm.ragingestion.support.Fixtures;
|
||||
import com.ankurm.ragingestion.support.Transcript;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.ai.reader.tika.TikaDocumentReader;
|
||||
import org.springframework.ai.transformer.splitter.TokenTextSplitter;
|
||||
import org.springframework.core.io.FileSystemResource;
|
||||
|
||||
/** Retrieval hit rate of TokenTextSplitter at six chunk sizes, with its default minChunkSizeChars and with a small one. */
|
||||
class ChunkSizeSweepTest {
|
||||
|
||||
@TempDir
|
||||
Path dir;
|
||||
|
||||
@Test
|
||||
void chunkSizeSweep() throws Exception {
|
||||
List<Document> text = new TikaDocumentReader(new FileSystemResource(Fixtures.docx(dir))).get();
|
||||
int total = Tokens.count(text.get(0).getText());
|
||||
|
||||
try (Transcript t = new Transcript("02-chunk-size-sweep.txt", "TokenTextSplitter chunk size against retrieval hit rate (all-MiniLM-L6-v2, 37 questions)")) {
|
||||
t.line("source: runbook.docx read by TikaDocumentReader, %d cl100k tokens, 37 questions, embedding model all-MiniLM-L6-v2 (max 256 WordPiece tokens)", total);
|
||||
t.line("hit@k: the question's answer is inside one of the k best chunks. cut: answers that no chunk contains whole.");
|
||||
t.line("top3Toks: tokens in the three best chunks, the context a prompt would carry per question.");
|
||||
|
||||
t.section("defaults except chunkSize (minChunkSizeChars=350)");
|
||||
t.line("chunkSize %s", Eval.Score.HEADER);
|
||||
for (int size : new int[] {32, 64, 128, 256, 512, 800}) {
|
||||
List<Document> chunks = TokenTextSplitter.builder().withChunkSize(size).build().apply(text);
|
||||
t.line("%9d %s", size, Eval.score(chunks, MiniLm.get()).row());
|
||||
}
|
||||
|
||||
t.section("same sizes with minChunkSizeChars=10");
|
||||
t.line("chunkSize %s", Eval.Score.HEADER);
|
||||
for (int size : new int[] {32, 64, 128, 256, 512, 800}) {
|
||||
List<Document> chunks = TokenTextSplitter.builder().withChunkSize(size).withMinChunkSizeChars(10).build().apply(text);
|
||||
t.line("%9d %s", size, Eval.score(chunks, MiniLm.get()).row());
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
package com.ankurm.ragingestion;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
class CorpusTest {
|
||||
|
||||
@Test
|
||||
void everyAnswerFragmentOccursExactlyOnce() {
|
||||
String text = Corpus.fullText();
|
||||
for (Corpus.Question q : Corpus.QUESTIONS) {
|
||||
int first = text.indexOf(q.answer());
|
||||
assertThat(first).as("fragment present: %s", q.answer()).isGreaterThanOrEqualTo(0);
|
||||
assertThat(text.indexOf(q.answer(), first + 1)).as("fragment unique: %s", q.answer()).isEqualTo(-1);
|
||||
}
|
||||
assertThat(Corpus.QUESTIONS).hasSize(37);
|
||||
assertThat(Corpus.SECTIONS.stream().mapToInt(s -> s.paragraphs().size()).sum()).isEqualTo(36);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
package com.ankurm.ragingestion;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.function.Supplier;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
import com.ankurm.ragingestion.support.Transcript;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.ai.document.DocumentTransformer;
|
||||
import org.springframework.ai.transformer.splitter.TokenTextSplitter;
|
||||
|
||||
/**
|
||||
* For each of the 36 paragraphs in turn, append one sentence to it and count how many chunks get a new id,
|
||||
* which is how many texts must be embedded again. No database and no model: ids are hashes of chunk text.
|
||||
*/
|
||||
class EditCostTest {
|
||||
|
||||
private static final String ADDED = " This paragraph was revised in the latest edition of the runbook.";
|
||||
|
||||
@Test
|
||||
void whatOneEditCosts() {
|
||||
List<String> blocks = Corpus.blocks();
|
||||
List<Integer> paragraphBlocks = new ArrayList<>();
|
||||
for (int i = 0; i < blocks.size(); i++) {
|
||||
if (blocks.get(i).length() > 100) {
|
||||
paragraphBlocks.add(i);
|
||||
}
|
||||
}
|
||||
|
||||
Map<String, Supplier<DocumentTransformer>> splitters = new LinkedHashMap<>();
|
||||
splitters.put("TokenTextSplitter 128 tokens", () -> TokenTextSplitter.builder().withChunkSize(128).build());
|
||||
splitters.put("TokenTextSplitter 256 tokens", () -> TokenTextSplitter.builder().withChunkSize(256).build());
|
||||
splitters.put("SentenceSplitter 128, no overlap", () -> new SentenceSplitter(128, 0));
|
||||
splitters.put("SentenceSplitter 128, 1 sentence overlap", () -> new SentenceSplitter(128, 1));
|
||||
splitters.put("one chunk per paragraph", () -> new HeadingAwareChunker(false));
|
||||
|
||||
try (Transcript t = new Transcript("08-edit-cost.txt", "Chunks to embed again after appending one sentence to one paragraph (all 36 paragraphs tried)")) {
|
||||
t.line("%-42s %6s %6s %6s %6s", "splitter", "chunks", "mean", "median", "max");
|
||||
for (var entry : splitters.entrySet()) {
|
||||
DocumentTransformer splitter = entry.getValue().get();
|
||||
Set<String> before = ids(splitter, blocks);
|
||||
List<Integer> costs = new ArrayList<>();
|
||||
for (int index : paragraphBlocks) {
|
||||
List<String> edited = new ArrayList<>(blocks);
|
||||
edited.set(index, blocks.get(index) + ADDED);
|
||||
Set<String> after = ids(splitter, edited);
|
||||
costs.add((int) after.stream().filter(id -> !before.contains(id)).count());
|
||||
}
|
||||
List<Integer> sorted = costs.stream().sorted().toList();
|
||||
t.line("%-42s %6d %6.1f %6d %6d", entry.getKey(), before.size(),
|
||||
costs.stream().mapToInt(Integer::intValue).average().orElse(0), sorted.get(sorted.size() / 2),
|
||||
sorted.get(sorted.size() - 1));
|
||||
}
|
||||
t.line("columns: chunks in the unedited file; mean, median and max chunks re-embedded over the 36 edits");
|
||||
}
|
||||
}
|
||||
|
||||
private static Set<String> ids(DocumentTransformer splitter, List<String> blocks) {
|
||||
Document whole = new Document(String.join("\n", blocks));
|
||||
return new MetadataEnricher("runbook.docx", "docx").apply(splitter.apply(List.of(whole))).stream()
|
||||
.map(Document::getId).collect(Collectors.toSet());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,81 @@
|
||||
package com.ankurm.ragingestion;
|
||||
|
||||
import java.nio.file.Path;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
import com.ankurm.ragingestion.support.Eval;
|
||||
import com.ankurm.ragingestion.support.Fixtures;
|
||||
import com.ankurm.ragingestion.support.Transcript;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.ai.reader.jsoup.JsoupDocumentReader;
|
||||
import org.springframework.ai.reader.jsoup.config.JsoupDocumentReaderConfig;
|
||||
import org.springframework.ai.reader.pdf.PagePdfDocumentReader;
|
||||
import org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig;
|
||||
import org.springframework.ai.reader.tika.TikaDocumentReader;
|
||||
import org.springframework.ai.transformer.splitter.TokenTextSplitter;
|
||||
import org.springframework.core.io.FileSystemResource;
|
||||
|
||||
/** The same words in three file formats, read seven ways, cut by the same 128-token splitter. */
|
||||
class FormatParityTest {
|
||||
|
||||
@TempDir
|
||||
Path dir;
|
||||
|
||||
@Test
|
||||
void sameContentThreeFormats() throws Exception {
|
||||
FileSystemResource pdf = new FileSystemResource(Fixtures.pdf(dir));
|
||||
FileSystemResource docx = new FileSystemResource(Fixtures.docx(dir));
|
||||
FileSystemResource html = new FileSystemResource(Fixtures.html(dir));
|
||||
TokenTextSplitter splitter = TokenTextSplitter.builder().withChunkSize(128).build();
|
||||
|
||||
try (Transcript t = new Transcript("05-format-parity.txt", "One runbook, three formats, TokenTextSplitter(128), 37 questions")) {
|
||||
t.line("%-52s %s", "reader and configuration", Eval.Score.HEADER);
|
||||
|
||||
List<Document> pages = new PagePdfDocumentReader(pdf).get();
|
||||
row(t, "PDF PagePdfDocumentReader, each page split alone", splitter.apply(pages));
|
||||
row(t, "PDF pages joined first, footer still there", splitter.apply(joined(pages)));
|
||||
|
||||
List<Document> trimmed = new PagePdfDocumentReader(pdf, PdfDocumentReaderConfig.builder().withPageBottomMargin(60).build()).get();
|
||||
row(t, "PDF pages joined, pageBottomMargin=60", splitter.apply(joined(trimmed)));
|
||||
row(t, "PDF pages joined, margin 60, whitespace collapsed", splitter.apply(joined(collapse(trimmed))));
|
||||
|
||||
row(t, "DOCX TikaDocumentReader", splitter.apply(new TikaDocumentReader(docx).get()));
|
||||
row(t, "HTML JsoupDocumentReader, default selector (body)", splitter.apply(new JsoupDocumentReader(html).get()));
|
||||
row(t, "HTML JsoupDocumentReader, selector=article",
|
||||
splitter.apply(new JsoupDocumentReader(html, JsoupDocumentReaderConfig.builder().selector("article").build()).get()));
|
||||
|
||||
t.section("tokens the splitter had to cut, by reader");
|
||||
t.line("PDF pages as read : %d characters, %d tokens", chars(pages), tokens(pages));
|
||||
t.line("PDF whitespace collapsed: %d characters, %d tokens", chars(collapse(trimmed)), tokens(collapse(trimmed)));
|
||||
t.line("DOCX : %d characters, %d tokens", chars(new TikaDocumentReader(docx).get()), tokens(new TikaDocumentReader(docx).get()));
|
||||
}
|
||||
}
|
||||
|
||||
private static List<Document> joined(List<Document> docs) {
|
||||
return List.of(new Document(docs.stream().map(Document::getText).collect(Collectors.joining("\n"))));
|
||||
}
|
||||
|
||||
private static List<Document> collapse(List<Document> docs) {
|
||||
List<Document> out = new ArrayList<>();
|
||||
for (Document d : docs) {
|
||||
out.add(new Document(d.getText().replaceAll("[ \\t]+", " ").replaceAll(" ?\\n ?", "\n"), d.getMetadata()));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
private static int chars(List<Document> docs) {
|
||||
return docs.stream().mapToInt(d -> d.getText().length()).sum();
|
||||
}
|
||||
|
||||
private static int tokens(List<Document> docs) {
|
||||
return docs.stream().mapToInt(d -> Tokens.count(d.getText())).sum();
|
||||
}
|
||||
|
||||
private static void row(Transcript t, String name, List<Document> chunks) {
|
||||
t.line("%-52s %s", name, Eval.score(chunks, MiniLm.get()).row());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
package com.ankurm.ragingestion;
|
||||
|
||||
import java.nio.file.Path;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
import com.ankurm.ragingestion.support.CountingEmbeddingModel;
|
||||
import com.ankurm.ragingestion.support.Fixtures;
|
||||
import com.ankurm.ragingestion.support.Transcript;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.ai.document.DocumentTransformer;
|
||||
import org.springframework.ai.reader.tika.TikaDocumentReader;
|
||||
import org.springframework.ai.transformer.splitter.TokenTextSplitter;
|
||||
import org.springframework.ai.vectorstore.SearchRequest;
|
||||
import org.springframework.ai.vectorstore.pgvector.PgVectorStore;
|
||||
import org.springframework.core.io.FileSystemResource;
|
||||
import org.springframework.jdbc.core.JdbcTemplate;
|
||||
import org.springframework.jdbc.datasource.DriverManagerDataSource;
|
||||
|
||||
/** Re-ingesting into a real pgvector table: what a naive loop does, what stable chunk ids do, and what an edit costs. */
|
||||
class IdempotentIngestionTest {
|
||||
|
||||
static final String URL = System.getenv().getOrDefault("RAG_DB_URL", "jdbc:postgresql://localhost:5432/ragdemo");
|
||||
|
||||
@TempDir
|
||||
Path dir;
|
||||
|
||||
@Test
|
||||
void reingestion() throws Exception {
|
||||
JdbcTemplate jdbc = new JdbcTemplate(new DriverManagerDataSource(URL, "demo", "demo"));
|
||||
List<String> blocks = Corpus.blocks();
|
||||
List<String> edited = new ArrayList<>(blocks);
|
||||
// block 2 is the first paragraph of section 1; change the number and add a sentence, the way a real edit looks
|
||||
edited.set(2, blocks.get(2).replace("kept for 35 days", "kept for 45 days")
|
||||
+ " Snapshots of databases tagged restricted are kept for 90 days instead.");
|
||||
List<String> shorter = new ArrayList<>(edited.subList(0, edited.size() - 4)); // drop section 12
|
||||
|
||||
Path v1 = Fixtures.docx(dir, "runbook.docx", blocks);
|
||||
|
||||
try (Transcript t = new Transcript("06-idempotent-reingestion.txt", "Re-ingesting runbook.docx into pgvector 0.8.7 (all-MiniLM-L6-v2, 384 dims)")) {
|
||||
t.section("a naive loop: read, split, add, with random chunk ids");
|
||||
CountingEmbeddingModel naiveModel = new CountingEmbeddingModel(MiniLm.get());
|
||||
PgVectorStore naive = store(jdbc, "ingest_naive", naiveModel);
|
||||
List<Document> read = new TikaDocumentReader(new FileSystemResource(v1)).get();
|
||||
for (int run = 1; run <= 3; run++) {
|
||||
naive.add(TokenTextSplitter.builder().withChunkSize(128).build().apply(read));
|
||||
t.line("run %d: rows in table = %d, texts embedded so far = %d", run, count(jdbc, "ingest_naive"), naiveModel.texts());
|
||||
}
|
||||
t.line("rows saying \"kept for 35 days\": %d", rows(jdbc, "ingest_naive", "%kept for 35 days%"));
|
||||
|
||||
scenario(t, jdbc, "TokenTextSplitter(128): windows of tokens", "ingest_tokens", new TokenTextSplitter(), blocks, edited, shorter);
|
||||
}
|
||||
try (Transcript t = new Transcript("07-paragraph-chunks-reingestion.txt", "Same sequence with one chunk per paragraph")) {
|
||||
scenario(t, jdbc, "HeadingAwareChunker: one chunk per paragraph", "ingest_paragraphs", new HeadingAwareChunker(false), blocks, edited, shorter);
|
||||
}
|
||||
}
|
||||
|
||||
private void scenario(Transcript t, JdbcTemplate jdbc, String title, String table, DocumentTransformer splitter,
|
||||
List<String> v1Blocks, List<String> v2Blocks, List<String> v3Blocks) throws Exception {
|
||||
DocumentTransformer chosen = splitter instanceof TokenTextSplitter ? TokenTextSplitter.builder().withChunkSize(128).build() : splitter;
|
||||
CountingEmbeddingModel model = new CountingEmbeddingModel(MiniLm.get());
|
||||
PgVectorStore store = store(jdbc, table, model);
|
||||
IngestionPipeline pipeline = new IngestionPipeline(store, jdbc, table, chosen);
|
||||
|
||||
t.section(title);
|
||||
step(t, jdbc, table, model, "1. first ingest of v1", pipeline.ingest(Fixtures.docx(dir, "runbook.docx", v1Blocks)));
|
||||
step(t, jdbc, table, model, "2. same file again", pipeline.ingest(Fixtures.docx(dir, "runbook.docx", v1Blocks)));
|
||||
step(t, jdbc, table, model, "3. same file a third time", pipeline.ingest(Fixtures.docx(dir, "runbook.docx", v1Blocks)));
|
||||
step(t, jdbc, table, model, "4. v2: one paragraph edited", pipeline.ingest(Fixtures.docx(dir, "runbook.docx", v2Blocks)));
|
||||
t.line(" rows saying \"kept for 35 days\": %d, rows saying \"kept for 45 days\": %d",
|
||||
rows(jdbc, table, "%kept for 35 days%"), rows(jdbc, table, "%kept for 45 days%"));
|
||||
List<Document> top = store.similaritySearch(SearchRequest.builder()
|
||||
.query("How long do we keep the daily database snapshots before they are deleted?").topK(1).build());
|
||||
t.line(" best match for the snapshot question now contains \"45 days\": %s", top.get(0).getText().contains("45 days"));
|
||||
step(t, jdbc, table, model, "5. v3: section 12 deleted from the file", pipeline.ingest(Fixtures.docx(dir, "runbook.docx", v3Blocks)));
|
||||
t.line(" rows saying \"180 days in advance\" (a fact only section 12 holds): %d", rows(jdbc, table, "%180 days in advance%"));
|
||||
}
|
||||
|
||||
private static void step(Transcript t, JdbcTemplate jdbc, String table, CountingEmbeddingModel model, String label, IngestionPipeline.Result r) {
|
||||
t.line("%-40s chunks=%2d embedded=%2d unchanged=%2d deleted=%2d | rows=%2d, texts embedded in total=%d", label, r.chunks(),
|
||||
r.embedded(), r.unchanged(), r.deleted(), count(jdbc, table), model.texts());
|
||||
}
|
||||
|
||||
private static PgVectorStore store(JdbcTemplate jdbc, String table, CountingEmbeddingModel model) throws Exception {
|
||||
PgVectorStore store = PgVectorStore.builder(jdbc, model).vectorTableName(table).dimensions(384)
|
||||
.removeExistingVectorStoreTable(true).initializeSchema(true).build();
|
||||
store.afterPropertiesSet();
|
||||
return store;
|
||||
}
|
||||
|
||||
static int count(JdbcTemplate jdbc, String table) {
|
||||
return jdbc.queryForObject("select count(*) from " + table, Integer.class);
|
||||
}
|
||||
|
||||
static int rows(JdbcTemplate jdbc, String table, String like) {
|
||||
return jdbc.queryForObject("select count(*) from " + table + " where content like ?", Integer.class, like);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
package com.ankurm.ragingestion;
|
||||
|
||||
import java.nio.file.Path;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
|
||||
import com.ankurm.ragingestion.support.Eval;
|
||||
import com.ankurm.ragingestion.support.Fixtures;
|
||||
import com.ankurm.ragingestion.support.Transcript;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
import org.springframework.ai.document.DefaultContentFormatter;
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.ai.document.MetadataMode;
|
||||
import org.springframework.ai.embedding.EmbeddingModel;
|
||||
import org.springframework.ai.reader.tika.TikaDocumentReader;
|
||||
import org.springframework.core.io.FileSystemResource;
|
||||
|
||||
/** Does metadata you add for filtering end up inside the vector? It depends on the embedding model's MetadataMode. */
|
||||
class MetadataEmbeddingTest {
|
||||
|
||||
@TempDir
|
||||
Path dir;
|
||||
|
||||
@Test
|
||||
void metadataInTheEmbedding() throws Exception {
|
||||
List<Document> word = new TikaDocumentReader(new FileSystemResource(Fixtures.docx(dir))).get();
|
||||
List<Document> chunks = new MetadataEnricher("runbook.docx", "docx").apply(new HeadingAwareChunker(false).apply(word));
|
||||
Document sample = chunks.get(0);
|
||||
|
||||
try (Transcript t = new Transcript("04-metadata-in-the-embedding.txt", "Metadata and the text that gets embedded")) {
|
||||
EmbeddingModel none = MiniLm.create(MetadataMode.NONE);
|
||||
EmbeddingModel embed = MiniLm.create(MetadataMode.EMBED);
|
||||
t.line("chunk metadata: %s", new java.util.TreeMap<>(sample.getMetadata()));
|
||||
|
||||
t.section("the text each model would embed for the same chunk");
|
||||
t.line("MetadataMode.NONE : %s", one(none.getEmbeddingContent(sample), 110));
|
||||
t.line("MetadataMode.EMBED: %s", one(embed.getEmbeddingContent(sample), 110));
|
||||
|
||||
t.section("hit rate, one chunk per paragraph, 37 questions");
|
||||
t.line("%-58s %s", "configuration", Eval.Score.HEADER);
|
||||
t.line("%-58s %s", "MetadataMode.NONE", Eval.score(chunks, none).row());
|
||||
t.line("%-58s %s", "MetadataMode.EMBED, every key embedded", Eval.score(chunks, embed).row());
|
||||
|
||||
DefaultContentFormatter keep = DefaultContentFormatter.builder()
|
||||
.withExcludedEmbedMetadataKeys("source", "format", "chunk_index", "content_hash").build();
|
||||
List<Document> filtered = new ArrayList<>();
|
||||
for (Document c : chunks) {
|
||||
Document copy = c.mutate().build();
|
||||
copy.setContentFormatter(keep);
|
||||
filtered.add(copy);
|
||||
}
|
||||
t.line("%-58s %s", "MetadataMode.EMBED, four keys in excludedEmbedMetadataKeys", Eval.score(filtered, embed).row());
|
||||
t.line("embedded text with the four keys excluded: %s", one(embed.getEmbeddingContent(filtered.get(0)), 110));
|
||||
}
|
||||
}
|
||||
|
||||
private static String one(String s, int n) {
|
||||
String x = s.replace("\n", "\\n");
|
||||
return x.length() <= n ? x : x.substring(0, n);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
package com.ankurm.ragingestion;
|
||||
|
||||
import java.nio.file.Path;
|
||||
import java.util.List;
|
||||
|
||||
import com.ankurm.ragingestion.support.Fixtures;
|
||||
import com.ankurm.ragingestion.support.Transcript;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.ai.vectorstore.SearchRequest;
|
||||
import org.springframework.ai.vectorstore.pgvector.PgVectorStore;
|
||||
import org.springframework.jdbc.core.JdbcTemplate;
|
||||
import org.springframework.jdbc.datasource.DriverManagerDataSource;
|
||||
|
||||
/** What the metadata added at ingestion buys at query time: a filter on section or format, run inside pgvector. */
|
||||
class MetadataFilterTest {
|
||||
|
||||
@TempDir
|
||||
Path dir;
|
||||
|
||||
@Test
|
||||
void filterBySectionAndFormat() throws Exception {
|
||||
JdbcTemplate jdbc = new JdbcTemplate(new DriverManagerDataSource(IdempotentIngestionTest.URL, "demo", "demo"));
|
||||
PgVectorStore store = PgVectorStore.builder(jdbc, MiniLm.get()).vectorTableName("ingest_filter").dimensions(384)
|
||||
.removeExistingVectorStoreTable(true).initializeSchema(true).build();
|
||||
store.afterPropertiesSet();
|
||||
IngestionPipeline pipeline = new IngestionPipeline(store, jdbc, "ingest_filter", new HeadingAwareChunker(false));
|
||||
Path html = Fixtures.html(dir);
|
||||
// the first configuration that came to mind: selector=article. Jsoup flattens it to one line.
|
||||
String flat = new org.springframework.ai.reader.jsoup.JsoupDocumentReader(new org.springframework.core.io.FileSystemResource(html),
|
||||
org.springframework.ai.reader.jsoup.config.JsoupDocumentReaderConfig.builder().selector("article").build()).get().get(0).getText();
|
||||
int flatChunks = new HeadingAwareChunker(false).apply(List.of(new Document(flat))).size();
|
||||
String lined = IngestionPipeline.reader(html, "html").get().get(0).getText();
|
||||
int linedChunks = new HeadingAwareChunker(false).apply(List.of(new Document(lined))).size();
|
||||
pipeline.ingest(Fixtures.docx(dir));
|
||||
pipeline.ingest(html);
|
||||
|
||||
String question = "How long is data kept?";
|
||||
try (Transcript t = new Transcript("09-metadata-filter.txt", "Filtering retrieval by the metadata written at ingestion")) {
|
||||
t.line("rows in table: %d (runbook.docx and runbook.html, one chunk per paragraph)", IdempotentIngestionTest.count(jdbc, "ingest_filter"));
|
||||
t.line("JsoupDocumentReader selector=article: %d line(s) of text -> %d chunks from the paragraph chunker", flat.split("\\R").length, flatChunks);
|
||||
t.line("selector=\"article h1, article h2, article p\" with separator \"\\n\": %d lines -> %d chunks", lined.split("\\R").length, linedChunks);
|
||||
t.line("rows per format: %s", jdbc.queryForList("select metadata->>'format' as f, count(*) as n from ingest_filter group by 1 order by 1")
|
||||
.stream().map(r -> r.get("f") + "=" + r.get("n")).toList());
|
||||
t.line("question: \"%s\"", question);
|
||||
|
||||
t.section("no filter, top 5");
|
||||
show(t, store.similaritySearch(SearchRequest.builder().query(question).topK(5).build()));
|
||||
|
||||
t.section("filter: section == '9. Logging and Monitoring'");
|
||||
show(t, store.similaritySearch(SearchRequest.builder().query(question).topK(5)
|
||||
.filterExpression("section == '9. Logging and Monitoring'").build()));
|
||||
|
||||
t.section("filter: format == 'html' && section == '1. Backups and Restore'");
|
||||
show(t, store.similaritySearch(SearchRequest.builder().query(question).topK(5)
|
||||
.filterExpression("format == 'html' && section == '1. Backups and Restore'").build()));
|
||||
}
|
||||
}
|
||||
|
||||
private static void show(Transcript t, List<Document> docs) {
|
||||
for (Document d : docs) {
|
||||
t.line(" %.3f %-5s %-34s %s", d.getScore(), d.getMetadata().get("format"), d.getMetadata().get("section"),
|
||||
d.getText().substring(0, 58).replace("\n", " "));
|
||||
}
|
||||
t.line(" (%d results)", docs.size());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,143 @@
|
||||
package com.ankurm.ragingestion;
|
||||
|
||||
import java.nio.file.Path;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.TreeMap;
|
||||
|
||||
import com.ankurm.ragingestion.support.Fixtures;
|
||||
import com.ankurm.ragingestion.support.Transcript;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.ai.reader.jsoup.JsoupDocumentReader;
|
||||
import org.springframework.ai.reader.jsoup.config.JsoupDocumentReaderConfig;
|
||||
import org.springframework.ai.reader.pdf.PagePdfDocumentReader;
|
||||
import org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig;
|
||||
import org.springframework.ai.reader.tika.TikaDocumentReader;
|
||||
import org.springframework.core.io.FileSystemResource;
|
||||
|
||||
/** What each reader hands back for the same runbook, written as PDF, Word and HTML. */
|
||||
class ReadersTest {
|
||||
|
||||
@TempDir
|
||||
Path dir;
|
||||
|
||||
@Test
|
||||
void whatEachReaderReturns() throws Exception {
|
||||
Path pdf = Fixtures.pdf(dir);
|
||||
Path docx = Fixtures.docx(dir);
|
||||
Path html = Fixtures.html(dir);
|
||||
String footerMarker = "Confidential";
|
||||
|
||||
try (Transcript t = new Transcript("01-readers.txt", "What each reader returns for the same 36-paragraph runbook")) {
|
||||
t.line("corpus: %d characters of body text (title, 12 headings, 36 paragraphs)", Corpus.fullText().length());
|
||||
|
||||
t.section("PagePdfDocumentReader, default config");
|
||||
List<Document> pages = new PagePdfDocumentReader(new FileSystemResource(pdf)).get();
|
||||
t.line("Documents returned: %d (one per PDF page)", pages.size());
|
||||
t.line("metadata of page 1: %s", new TreeMap<>(pages.get(0).getMetadata()));
|
||||
t.line("pages whose text contains the running footer: %d of %d", count(pages, footerMarker), pages.size());
|
||||
t.line("characters as read: %d, after collapsing runs of whitespace: %d (the corpus itself is %d)", total(pages),
|
||||
pages.stream().mapToInt(d -> norm(d.getText()).length()).sum(), Corpus.fullText().length());
|
||||
t.line("page 1 ends with: \"%s\"", tail(pages.get(0).getText(), 70));
|
||||
t.line("page 2 starts with: \"%s\"", norm(head(pages.get(1).getText(), 90)));
|
||||
t.line("paragraphs that no single page contains whole, because a page break cuts them: %d of 36", cutParagraphs(pages));
|
||||
|
||||
t.section("PagePdfDocumentReader with pageBottomMargin");
|
||||
for (int margin : new int[] {30, 60, 100}) {
|
||||
PdfDocumentReaderConfig cfg = PdfDocumentReaderConfig.builder().withPageBottomMargin(margin).build();
|
||||
List<Document> cut = new PagePdfDocumentReader(new FileSystemResource(pdf), cfg).get();
|
||||
t.line("pageBottomMargin=%3d: footer on %d of %d pages, answer fragments still in the text: %d of %d", margin,
|
||||
count(cut, footerMarker), cut.size(), fragments(cut), Corpus.QUESTIONS.size());
|
||||
}
|
||||
t.line("no margin: footer on %d of %d pages, answer fragments still in the text: %d of %d",
|
||||
count(pages, footerMarker), pages.size(), fragments(pages), Corpus.QUESTIONS.size());
|
||||
|
||||
t.section("TikaDocumentReader on the .docx");
|
||||
List<Document> word = new TikaDocumentReader(new FileSystemResource(docx)).get();
|
||||
t.line("Documents returned: %d", word.size());
|
||||
t.line("metadata: %s", new TreeMap<>(word.get(0).getMetadata()));
|
||||
t.line("characters: %d, contains the footer text: %s", word.get(0).getText().length(),
|
||||
word.get(0).getText().contains(footerMarker));
|
||||
t.line("starts with: \"%s\"", head(word.get(0).getText(), 90));
|
||||
t.line("blank-line-separated blocks in the text: %d, newline-separated lines: %d", word.get(0).getText().split("\n\n+").length,
|
||||
word.get(0).getText().split("\n").length);
|
||||
t.line("answer fragments in the text: %d of %d", fragments(word), Corpus.QUESTIONS.size());
|
||||
|
||||
t.section("TikaDocumentReader on the same PDF and HTML");
|
||||
List<Document> tikaPdf = new TikaDocumentReader(new FileSystemResource(pdf)).get();
|
||||
List<Document> tikaHtml = new TikaDocumentReader(new FileSystemResource(html)).get();
|
||||
t.line(".pdf -> %d Document, %d characters, metadata keys %s", tikaPdf.size(), tikaPdf.get(0).getText().length(),
|
||||
tikaPdf.get(0).getMetadata().keySet());
|
||||
t.line(".html -> %d Document, %d characters, contains \"Cookie settings\": %s, contains tracking script text: %s",
|
||||
tikaHtml.size(), tikaHtml.get(0).getText().length(), tikaHtml.get(0).getText().contains("Cookie settings"),
|
||||
tikaHtml.get(0).getText().contains("noise noise"));
|
||||
|
||||
t.section("JsoupDocumentReader on the .html");
|
||||
List<Document> body = new JsoupDocumentReader(new FileSystemResource(html)).get();
|
||||
t.line("default config: %d Document, %d characters, contains nav link text \"Status\": %s, contains \"Cookie settings\": %s",
|
||||
body.size(), body.get(0).getText().length(), body.get(0).getText().contains("Status"),
|
||||
body.get(0).getText().contains("Cookie settings"));
|
||||
t.line("default metadata: %s", new TreeMap<>(body.get(0).getMetadata()));
|
||||
|
||||
JsoupDocumentReaderConfig article = JsoupDocumentReaderConfig.builder().selector("article").build();
|
||||
List<Document> art = new JsoupDocumentReader(new FileSystemResource(html), article).get();
|
||||
t.line("selector=article: %d Document, %d characters, nav text: %s, footer text: %s", art.size(),
|
||||
art.get(0).getText().length(), art.get(0).getText().contains("Status"),
|
||||
art.get(0).getText().contains("Cookie settings"));
|
||||
|
||||
JsoupDocumentReaderConfig bySection = JsoupDocumentReaderConfig.builder().selector("section").allElements(false)
|
||||
.groupByElement(true).build();
|
||||
List<Document> sections = new JsoupDocumentReader(new FileSystemResource(html), bySection).get();
|
||||
t.line("selector=section, groupByElement=true: %d Documents", sections.size());
|
||||
t.line("first section Document: %d characters, starts \"%s\"", sections.get(0).getText().length(),
|
||||
head(sections.get(0).getText(), 60));
|
||||
|
||||
JsoupDocumentReaderConfig meta = JsoupDocumentReaderConfig.builder().selector("article")
|
||||
.metadataTag("description").additionalMetadata("format", "html").build();
|
||||
Map<String, Object> m = new TreeMap<>(new JsoupDocumentReader(new FileSystemResource(html), meta).get().get(0).getMetadata());
|
||||
t.line("with metadataTag(\"description\") and additionalMetadata: %s", m);
|
||||
}
|
||||
}
|
||||
|
||||
private static int count(List<Document> docs, String needle) {
|
||||
return (int) docs.stream().filter(d -> d.getText().contains(needle)).count();
|
||||
}
|
||||
|
||||
/** Collapse every run of whitespace to one space, the same normalisation the hit-rate check applies. */
|
||||
static String norm(String s) {
|
||||
return s.replaceAll("\\s+", " ").strip();
|
||||
}
|
||||
|
||||
private static int fragments(List<Document> docs) {
|
||||
String all = norm(docs.stream().map(Document::getText).collect(java.util.stream.Collectors.joining(" ")));
|
||||
return (int) Corpus.QUESTIONS.stream().filter(q -> all.contains(q.answer())).count();
|
||||
}
|
||||
|
||||
private static int cutParagraphs(List<Document> pages) {
|
||||
int cut = 0;
|
||||
for (Corpus.Section s : Corpus.SECTIONS) {
|
||||
for (String p : s.paragraphs()) {
|
||||
if (pages.stream().noneMatch(d -> norm(d.getText()).contains(p))) {
|
||||
cut++;
|
||||
}
|
||||
}
|
||||
}
|
||||
return cut;
|
||||
}
|
||||
|
||||
private static int total(List<Document> docs) {
|
||||
return docs.stream().mapToInt(d -> d.getText().length()).sum();
|
||||
}
|
||||
|
||||
private static String head(String s, int n) {
|
||||
String one = s.strip().replace("\n", "\\n");
|
||||
return one.length() <= n ? one : one.substring(0, n);
|
||||
}
|
||||
|
||||
private static String tail(String s, int n) {
|
||||
String one = s.strip().replace("\n", "\\n");
|
||||
return one.length() <= n ? one : one.substring(one.length() - n);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
package com.ankurm.ragingestion;
|
||||
|
||||
import java.nio.file.Path;
|
||||
import java.util.List;
|
||||
|
||||
import com.ankurm.ragingestion.support.Eval;
|
||||
import com.ankurm.ragingestion.support.Fixtures;
|
||||
import com.ankurm.ragingestion.support.Transcript;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.ai.reader.jsoup.JsoupDocumentReader;
|
||||
import org.springframework.ai.reader.jsoup.config.JsoupDocumentReaderConfig;
|
||||
import org.springframework.ai.reader.tika.TikaDocumentReader;
|
||||
import org.springframework.ai.transformer.splitter.TokenTextSplitter;
|
||||
import org.springframework.core.io.FileSystemResource;
|
||||
|
||||
/** Six ways to cut the same runbook, scored on the same 37 questions with the same embedding model. */
|
||||
class StrategyTest {
|
||||
|
||||
@TempDir
|
||||
Path dir;
|
||||
|
||||
@Test
|
||||
void strategies() throws Exception {
|
||||
List<Document> word = new TikaDocumentReader(new FileSystemResource(Fixtures.docx(dir))).get();
|
||||
List<Document> sections = new JsoupDocumentReader(new FileSystemResource(Fixtures.html(dir)),
|
||||
JsoupDocumentReaderConfig.builder().selector("section").allElements(false).groupByElement(true).build()).get();
|
||||
|
||||
try (Transcript t = new Transcript("03-chunking-strategies.txt", "Chunking strategies on the same runbook (all-MiniLM-L6-v2, 37 questions)")) {
|
||||
t.line("%-44s %s", "strategy", Eval.Score.HEADER);
|
||||
row(t, "TokenTextSplitter 128 tokens", TokenTextSplitter.builder().withChunkSize(128).build().apply(word));
|
||||
row(t, "TokenTextSplitter 64 tokens, minChars=10", TokenTextSplitter.builder().withChunkSize(64).withMinChunkSizeChars(10).build().apply(word));
|
||||
row(t, "SentenceSplitter 128 tokens, no overlap", new SentenceSplitter(128, 0).apply(word));
|
||||
row(t, "SentenceSplitter 128 tokens, 1 sentence overlap", new SentenceSplitter(128, 1).apply(word));
|
||||
row(t, "SentenceSplitter 64 tokens, no overlap", new SentenceSplitter(64, 0).apply(word));
|
||||
row(t, "one chunk per paragraph", new HeadingAwareChunker(false).apply(word));
|
||||
row(t, "one chunk per paragraph + section heading", new HeadingAwareChunker(true).apply(word));
|
||||
row(t, "one chunk per section (JsoupDocumentReader)", sections);
|
||||
|
||||
List<Document> paragraphs = new HeadingAwareChunker(false).apply(word);
|
||||
t.section("37 chunks from 36 paragraphs");
|
||||
t.line("last paragraph chunk: section=%s, text=\"%s\"", paragraphs.get(36).getMetadata().get("section"), paragraphs.get(36).getText());
|
||||
|
||||
t.section("what a paragraph chunk with its heading looks like");
|
||||
Document sample = new HeadingAwareChunker(true).apply(word).get(2);
|
||||
t.line("metadata: %s", new java.util.TreeMap<>(sample.getMetadata()));
|
||||
t.line("text: %s", sample.getText().replace("\n", "\\n").substring(0, 120));
|
||||
}
|
||||
}
|
||||
|
||||
private static void row(Transcript t, String name, List<Document> chunks) {
|
||||
t.line("%-44s %s", name, Eval.score(chunks, MiniLm.get()).row());
|
||||
}
|
||||
}
|
||||
+58
@@ -0,0 +1,58 @@
|
||||
package com.ankurm.ragingestion.support;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.ai.embedding.EmbeddingModel;
|
||||
import org.springframework.ai.embedding.EmbeddingRequest;
|
||||
import org.springframework.ai.embedding.EmbeddingResponse;
|
||||
|
||||
/** Wraps a model and counts how many texts it was asked to embed, which is what an embedding API bills for. */
|
||||
public class CountingEmbeddingModel implements EmbeddingModel {
|
||||
|
||||
private final EmbeddingModel delegate;
|
||||
private final AtomicInteger texts = new AtomicInteger();
|
||||
|
||||
public CountingEmbeddingModel(EmbeddingModel delegate) {
|
||||
this.delegate = delegate;
|
||||
}
|
||||
|
||||
public int texts() {
|
||||
return texts.get();
|
||||
}
|
||||
|
||||
@Override
|
||||
public EmbeddingResponse call(EmbeddingRequest request) {
|
||||
texts.addAndGet(request.getInstructions().size());
|
||||
return delegate.call(request);
|
||||
}
|
||||
|
||||
@Override
|
||||
public float[] embed(Document document) {
|
||||
texts.incrementAndGet();
|
||||
return delegate.embed(document);
|
||||
}
|
||||
|
||||
@Override
|
||||
public float[] embed(String text) {
|
||||
texts.incrementAndGet();
|
||||
return delegate.embed(text);
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<float[]> embed(List<String> input) {
|
||||
texts.addAndGet(input.size());
|
||||
return delegate.embed(input);
|
||||
}
|
||||
|
||||
@Override
|
||||
public String getEmbeddingContent(Document document) {
|
||||
return delegate.getEmbeddingContent(document);
|
||||
}
|
||||
|
||||
@Override
|
||||
public int dimensions() {
|
||||
return delegate.dimensions();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
package com.ankurm.ragingestion.support;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
import com.ankurm.ragingestion.Corpus;
|
||||
import com.ankurm.ragingestion.Tokens;
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.ai.embedding.EmbeddingModel;
|
||||
import org.springframework.ai.vectorstore.SearchRequest;
|
||||
import org.springframework.ai.vectorstore.SimpleVectorStore;
|
||||
|
||||
/**
|
||||
* Scores a set of chunks against the 37 questions. A question is answered at rank r when the r-th retrieved
|
||||
* chunk is the first one whose text, with whitespace collapsed, contains the question's answer fragment.
|
||||
*/
|
||||
public final class Eval {
|
||||
|
||||
public record Score(int chunks, double meanTokens, int hit1, int hit3, int hit5, int cut, double contextTokens) {
|
||||
|
||||
public String row() {
|
||||
int n = Corpus.QUESTIONS.size();
|
||||
return String.format("%6d %8.1f %2d/%d %2d/%d %2d/%d %4d %9.0f", chunks, meanTokens, hit1, n, hit3, n, hit5, n,
|
||||
cut, contextTokens);
|
||||
}
|
||||
|
||||
public static final String HEADER = "chunks meanToks hit@1 hit@3 hit@5 cut top3Toks";
|
||||
}
|
||||
|
||||
private Eval() {
|
||||
}
|
||||
|
||||
public static String norm(String s) {
|
||||
return s.replaceAll("\\s+", " ").strip();
|
||||
}
|
||||
|
||||
public static Score score(List<Document> chunks, EmbeddingModel model) {
|
||||
SimpleVectorStore store = SimpleVectorStore.builder(model).build();
|
||||
store.add(chunks);
|
||||
int h1 = 0, h3 = 0, h5 = 0, cut = 0;
|
||||
long ctx = 0;
|
||||
for (Corpus.Question q : Corpus.QUESTIONS) {
|
||||
boolean exists = chunks.stream().anyMatch(c -> norm(c.getText()).contains(q.answer()));
|
||||
if (!exists) {
|
||||
cut++;
|
||||
}
|
||||
List<Document> top = store.similaritySearch(SearchRequest.builder().query(q.text()).topK(5).similarityThresholdAll().build());
|
||||
for (int i = 0; i < top.size(); i++) {
|
||||
if (norm(top.get(i).getText()).contains(q.answer())) {
|
||||
if (i == 0) {
|
||||
h1++;
|
||||
}
|
||||
if (i < 3) {
|
||||
h3++;
|
||||
}
|
||||
h5++;
|
||||
break;
|
||||
}
|
||||
}
|
||||
for (int i = 0; i < Math.min(3, top.size()); i++) {
|
||||
ctx += Tokens.count(top.get(i).getText());
|
||||
}
|
||||
}
|
||||
double mean = chunks.stream().mapToInt(c -> Tokens.count(c.getText())).average().orElse(0);
|
||||
return new Score(chunks.size(), mean, h1, h3, h5, cut, (double) ctx / Corpus.QUESTIONS.size());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,141 @@
|
||||
package com.ankurm.ragingestion.support;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.OutputStream;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
|
||||
import com.ankurm.ragingestion.Corpus;
|
||||
import org.apache.pdfbox.pdmodel.PDDocument;
|
||||
import org.apache.pdfbox.pdmodel.PDPage;
|
||||
import org.apache.pdfbox.pdmodel.PDPageContentStream;
|
||||
import org.apache.pdfbox.pdmodel.font.PDType1Font;
|
||||
import org.apache.pdfbox.pdmodel.font.Standard14Fonts;
|
||||
import org.apache.poi.wp.usermodel.HeaderFooterType;
|
||||
import org.apache.poi.xwpf.usermodel.XWPFDocument;
|
||||
import org.apache.poi.xwpf.usermodel.XWPFParagraph;
|
||||
import org.apache.poi.xwpf.usermodel.XWPFRun;
|
||||
|
||||
/** Writes the runbook from {@link Corpus} as a PDF, a Word file and an HTML page, so no binary is committed. */
|
||||
public final class Fixtures {
|
||||
|
||||
public static final String FOOTER = "Helios Operations Runbook - Confidential - Page ";
|
||||
|
||||
private Fixtures() {
|
||||
}
|
||||
|
||||
public static Path pdf(Path dir) throws IOException {
|
||||
Path file = dir.resolve("runbook.pdf");
|
||||
try (PDDocument doc = new PDDocument()) {
|
||||
PDType1Font body = new PDType1Font(Standard14Fonts.FontName.HELVETICA);
|
||||
PDType1Font bold = new PDType1Font(Standard14Fonts.FontName.HELVETICA_BOLD);
|
||||
float width = 480, lineHeight = 15, top = 790, bottom = 80;
|
||||
List<String[]> lines = new ArrayList<>(); // {font, text}
|
||||
lines.add(new String[] {"B", Corpus.TITLE});
|
||||
lines.add(new String[] {"P", ""});
|
||||
for (Corpus.Section s : Corpus.SECTIONS) {
|
||||
lines.add(new String[] {"B", s.title()});
|
||||
for (String paragraph : s.paragraphs()) {
|
||||
for (String l : wrap(paragraph, body, 11, width)) {
|
||||
lines.add(new String[] {"P", l});
|
||||
}
|
||||
lines.add(new String[] {"P", ""});
|
||||
}
|
||||
}
|
||||
int pageNo = 0;
|
||||
int i = 0;
|
||||
while (i < lines.size()) {
|
||||
PDPage page = new PDPage();
|
||||
doc.addPage(page);
|
||||
pageNo++;
|
||||
try (PDPageContentStream cs = new PDPageContentStream(doc, page)) {
|
||||
float y = top;
|
||||
while (i < lines.size() && y > bottom) {
|
||||
String[] l = lines.get(i++);
|
||||
if (!l[1].isEmpty()) {
|
||||
cs.beginText();
|
||||
cs.setFont("B".equals(l[0]) ? bold : body, "B".equals(l[0]) ? 13 : 11);
|
||||
cs.newLineAtOffset(60, y);
|
||||
cs.showText(l[1]);
|
||||
cs.endText();
|
||||
}
|
||||
y -= lineHeight;
|
||||
}
|
||||
cs.beginText();
|
||||
cs.setFont(body, 9);
|
||||
cs.newLineAtOffset(60, 40);
|
||||
cs.showText(FOOTER + pageNo);
|
||||
cs.endText();
|
||||
}
|
||||
}
|
||||
doc.save(file.toFile());
|
||||
}
|
||||
return file;
|
||||
}
|
||||
|
||||
public static Path docx(Path dir) throws IOException {
|
||||
return docx(dir, "runbook.docx", Corpus.blocks());
|
||||
}
|
||||
|
||||
/** Writes a Word file from blocks: the first is the title, a block starting "N. " is a heading, the rest paragraphs. */
|
||||
public static Path docx(Path dir, String name, java.util.List<String> blocks) throws IOException {
|
||||
Path file = dir.resolve(name);
|
||||
try (XWPFDocument doc = new XWPFDocument(); OutputStream out = Files.newOutputStream(file)) {
|
||||
for (int i = 0; i < blocks.size(); i++) {
|
||||
String block = blocks.get(i);
|
||||
XWPFRun run = doc.createParagraph().createRun();
|
||||
if (i == 0) {
|
||||
run.setBold(true);
|
||||
run.setFontSize(18);
|
||||
} else if (block.matches("^\\d+\\. [^.]{1,60}$")) {
|
||||
run.setBold(true);
|
||||
run.setFontSize(14);
|
||||
}
|
||||
run.setText(block);
|
||||
}
|
||||
doc.createFooter(HeaderFooterType.DEFAULT).createParagraph().createRun().setText("Helios Operations Runbook - Confidential");
|
||||
doc.write(out);
|
||||
}
|
||||
return file;
|
||||
}
|
||||
|
||||
public static Path html(Path dir) throws IOException {
|
||||
Path file = dir.resolve("runbook.html");
|
||||
StringBuilder sb = new StringBuilder("<!DOCTYPE html>\n<html lang=\"en\"><head><meta charset=\"utf-8\">")
|
||||
.append("<title>").append(Corpus.TITLE).append("</title>")
|
||||
.append("<meta name=\"description\" content=\"How Helios runs production.\">")
|
||||
.append("<script>var tracking = 'noise noise noise';</script></head><body>\n")
|
||||
.append("<header><nav><a href=\"/\">Home</a> | <a href=\"/docs\">Docs</a> | <a href=\"/status\">Status</a></nav></header>\n")
|
||||
.append("<main><article>\n<h1>").append(Corpus.TITLE).append("</h1>\n");
|
||||
for (Corpus.Section s : Corpus.SECTIONS) {
|
||||
sb.append("<section><h2>").append(s.title()).append("</h2>\n");
|
||||
for (String p : s.paragraphs()) {
|
||||
sb.append("<p>").append(p.replace("&", "&")).append("</p>\n");
|
||||
}
|
||||
sb.append("</section>\n");
|
||||
}
|
||||
sb.append("</article></main>\n<footer>Helios Operations Runbook - Confidential. Cookie settings | Privacy | Terms</footer>\n</body></html>\n");
|
||||
Files.writeString(file, sb.toString());
|
||||
return file;
|
||||
}
|
||||
|
||||
static List<String> wrap(String text, PDType1Font font, float size, float width) throws IOException {
|
||||
List<String> out = new ArrayList<>();
|
||||
StringBuilder line = new StringBuilder();
|
||||
for (String word : text.split(" ")) {
|
||||
String candidate = line.isEmpty() ? word : line + " " + word;
|
||||
if (font.getStringWidth(candidate) / 1000 * size > width && !line.isEmpty()) {
|
||||
out.add(line.toString());
|
||||
line = new StringBuilder(word);
|
||||
} else {
|
||||
line = new StringBuilder(candidate);
|
||||
}
|
||||
}
|
||||
if (!line.isEmpty()) {
|
||||
out.add(line.toString());
|
||||
}
|
||||
return out;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
package com.ankurm.ragingestion.support;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.PrintWriter;
|
||||
import java.io.StringWriter;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
|
||||
/**
|
||||
* Writes a numbered transcript under {@code output/} and echoes it to the console. Every console block
|
||||
* quoted in the article comes out of one of these files verbatim.
|
||||
*/
|
||||
public final class Transcript implements AutoCloseable {
|
||||
|
||||
private final Path path;
|
||||
private final StringWriter buffer = new StringWriter();
|
||||
private final PrintWriter out = new PrintWriter(buffer);
|
||||
|
||||
public Transcript(String fileName, String title) {
|
||||
this.path = Path.of("output", fileName);
|
||||
out.println("# " + title);
|
||||
out.println();
|
||||
}
|
||||
|
||||
public Transcript line(String format, Object... args) {
|
||||
out.println(args.length == 0 ? format : String.format(format, args));
|
||||
return this;
|
||||
}
|
||||
|
||||
public Transcript blank() {
|
||||
out.println();
|
||||
return this;
|
||||
}
|
||||
|
||||
public Transcript section(String heading) {
|
||||
out.println();
|
||||
out.println("--- " + heading + " ---");
|
||||
return this;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() {
|
||||
out.flush();
|
||||
try {
|
||||
Files.createDirectories(path.getParent());
|
||||
Files.writeString(path, buffer.toString());
|
||||
} catch (IOException e) {
|
||||
throw new IllegalStateException("could not write " + path, e);
|
||||
}
|
||||
System.out.print(buffer);
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user