Claude
1d4625a1c2
Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01B38FGKKam5SCGgwgduVAh3
60 lines
4.0 KiB
Markdown
60 lines
4.0 KiB
Markdown
# 2. Chunking
|
|
|
|
prev: [1. The shape of a RAG pipeline](01-the-shape-of-a-rag-pipeline.md) · [Index](../README.md) · next: [3. Ingestion](03-ingestion.md)
|
|
|
|
A chunk is the unit that gets embedded, stored, retrieved and pasted into the prompt. Cut too big and the
|
|
prompt fills with text unrelated to the question; cut too small and an answer is split across two chunks.
|
|
|
|
This chapter measures what the chunkers do to one fixed document, not which one retrieves best. **There is no
|
|
retrieval-quality benchmark in this repository**, so nothing here says which strategy is best for your
|
|
documents. The fixture is forty numbered sentences, 4,268 characters, 880 `cl100k_base` tokens.
|
|
|
|
## `TokenTextSplitter` in Spring AI 2.0.1
|
|
|
|
Transcript: [output 02](output/02-token-text-splitter.txt). Test: [`ChunkingTest`](../src/test/java/com/ankurm/rag/ChunkingTest.java).
|
|
|
|
- **The defaults are large.** `new TokenTextSplitter()` (800-token chunks) turned the 880-token fixture into
|
|
two chunks of 792 and 88 tokens. A short handbook page becomes one chunk.
|
|
- **It prefers sentence ends.** With `withChunkSize(100)` the result was ten chunks of 88 tokens, every one ending
|
|
on a full stop; it did not cut at exactly 100 tokens.
|
|
- **It has no overlap.** Zero of nine neighbouring boundaries repeated any text. A sentence that straddles a cut
|
|
belongs to one chunk only.
|
|
- **`withMinChunkSizeChars` did nothing on this fixture.** 350 and 50 produced identical ten-chunk results. Reading
|
|
the bytecode of the splitter: when a chunk exceeds the token limit it is cut back to its last sentence-ending
|
|
mark, but only if that mark is more than `minChunkSizeChars` characters into the chunk. On text with a full stop
|
|
every ~100 characters the mark is always far enough in, so neither value matters. Do not read this as "the
|
|
setting is useless", only as "this document does not exercise it".
|
|
- **The constructor changed.** The 1.x five-argument constructor `new TokenTextSplitter(512, 128, 5, 10_000, true)`
|
|
does not exist in 2.0.1; use the builder. See [output 14](output/14-legacy-1x.txt) and [output 12](output/12-api-facts.txt).
|
|
|
|
## `RecursiveChunker` (ours)
|
|
|
|
Transcript: [output 03](output/03-recursive-and-semantic-chunkers.txt). Source: [`RecursiveChunker`](../src/main/java/com/ankurm/rag/chunk/RecursiveChunker.java).
|
|
|
|
Cuts at the coarsest boundary that fits (blank line, line break, sentence end, space, and only then mid-word),
|
|
and repeats the last `overlapChars` characters at the start of the next chunk. With `(400, 80)` the fixture became
|
|
14 chunks, longest 395 characters, and all 13 boundaries carried overlap. Sizes are characters, not tokens. A
|
|
900-character string with no separators is cut hard at `[400, 400, 100]`.
|
|
|
|
The overlap starts at a word boundary, so a chunk often *opens mid-sentence* ("for item 3 and states th..."). That is
|
|
the price of overlap; whether it helps retrieval was not measured.
|
|
|
|
## `SemanticChunker` (ours)
|
|
|
|
Same transcript. Source: [`SemanticChunker`](../src/main/java/com/ankurm/rag/chunk/SemanticChunker.java).
|
|
|
|
Embeds every sentence in **one batched call** and starts a new chunk where the cosine distance between neighbours exceeds a
|
|
threshold. On twelve sentences about three topics it produced three chunks, one per topic, and sent 12 texts to the
|
|
embedding model. That result is from the hashing model, which is good at exactly this (topics share words). With a
|
|
real embedding model the right threshold is different and must be found on your own text; the value 0.9 has no meaning
|
|
outside this test. It costs an embedding call per sentence at ingestion time.
|
|
|
|
## Choosing
|
|
|
|
Start with `TokenTextSplitter` and a size you set on purpose. Move to the recursive chunker when a fact is being cut
|
|
in half at the boundary and you can see it in real retrievals. Consider semantic chunking only when documents mix
|
|
topics without headings and you can afford the extra embedding calls. Whichever you use, change it by replacing the
|
|
one `DocumentTransformer` bean; nothing else in the pipeline notices.
|
|
|
|
prev: [1](01-the-shape-of-a-rag-pipeline.md) · next: [3. Ingestion](03-ingestion.md)
|