Claude
d7e68a14af
Co-Authored-By: Claude Sonnet 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_0146opDKBYtADXvkc39zdiyw
24 lines
1.4 KiB
Plaintext
24 lines
1.4 KiB
Plaintext
# TokenTextSplitter chunk size against retrieval hit rate (all-MiniLM-L6-v2, 37 questions)
|
|
|
|
source: runbook.docx read by TikaDocumentReader, 2172 cl100k tokens, 37 questions, embedding model all-MiniLM-L6-v2 (max 256 WordPiece tokens)
|
|
hit@k: the question's answer is inside one of the k best chunks. cut: answers that no chunk contains whole.
|
|
top3Toks: tokens in the three best chunks, the context a prompt would carry per question.
|
|
|
|
--- defaults except chunkSize (minChunkSizeChars=350) ---
|
|
chunkSize chunks meanToks hit@1 hit@3 hit@5 cut top3Toks
|
|
32 68 31.9 22/37 29/37 30/37 3 96
|
|
64 34 63.9 27/37 33/37 33/37 2 191
|
|
128 19 114.1 31/37 35/37 36/37 0 335
|
|
256 9 241.1 24/37 31/37 36/37 0 720
|
|
512 5 434.0 21/37 33/37 37/37 0 1278
|
|
800 3 723.3 17/37 37/37 37/37 0 2170
|
|
|
|
--- same sizes with minChunkSizeChars=10 ---
|
|
chunkSize chunks meanToks hit@1 hit@3 hit@5 cut top3Toks
|
|
32 88 24.6 31/37 32/37 32/37 0 76
|
|
64 42 51.7 27/37 32/37 34/37 1 160
|
|
128 19 114.1 31/37 35/37 36/37 0 335
|
|
256 9 241.1 24/37 31/37 36/37 0 720
|
|
512 5 434.0 21/37 33/37 37/37 0 1278
|
|
800 3 723.3 17/37 37/37 37/37 0 2170
|