# TokenTextSplitter chunk size against retrieval hit rate (all-MiniLM-L6-v2, 37 questions)

source: runbook.docx read by TikaDocumentReader, 2172 cl100k tokens, 37 questions, embedding model all-MiniLM-L6-v2 (max 256 WordPiece tokens)
hit@k: the question's answer is inside one of the k best chunks. cut: answers that no chunk contains whole.
top3Toks: tokens in the three best chunks, the context a prompt would carry per question.

--- defaults except chunkSize (minChunkSizeChars=350) ---
chunkSize  chunks  meanToks  hit@1  hit@3  hit@5   cut  top3Toks
       32      68      31.9  22/37  29/37  30/37     3         96
       64      34      63.9  27/37  33/37  33/37     2        191
      128      19     114.1  31/37  35/37  36/37     0        335
      256       9     241.1  24/37  31/37  36/37     0        720
      512       5     434.0  21/37  33/37  37/37     0       1278
      800       3     723.3  17/37  37/37  37/37     0       2170

--- same sizes with minChunkSizeChars=10 ---
chunkSize  chunks  meanToks  hit@1  hit@3  hit@5   cut  top3Toks
       32      88      24.6  31/37  32/37  32/37     0         76
       64      42      51.7  27/37  32/37  34/37     1        160
      128      19     114.1  31/37  35/37  36/37     0        335
      256       9     241.1  24/37  31/37  36/37     0        720
      512       5     434.0  21/37  33/37  37/37     0       1278
      800       3     723.3  17/37  37/37  37/37     0       2170