Claude
d7e68a14af
Co-Authored-By: Claude Sonnet 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_0146opDKBYtADXvkc39zdiyw
16 lines
1.1 KiB
Plaintext
16 lines
1.1 KiB
Plaintext
# One runbook, three formats, TokenTextSplitter(128), 37 questions
|
|
|
|
reader and configuration chunks meanToks hit@1 hit@3 hit@5 cut top3Toks
|
|
PDF PagePdfDocumentReader, each page split alone 36 106.6 24/37 32/37 33/37 1 333
|
|
PDF pages joined first, footer still there 34 112.9 26/37 33/37 34/37 1 338
|
|
PDF pages joined, pageBottomMargin=60 34 110.5 27/37 33/37 34/37 1 330
|
|
PDF pages joined, margin 60, whitespace collapsed 19 117.7 27/37 36/37 37/37 0 348
|
|
DOCX TikaDocumentReader 19 114.1 31/37 35/37 36/37 0 335
|
|
HTML JsoupDocumentReader, default selector (body) 19 114.8 30/37 34/37 36/37 0 342
|
|
HTML JsoupDocumentReader, selector=article 19 113.8 29/37 34/37 36/37 0 342
|
|
|
|
--- tokens the splitter had to cut, by reader ---
|
|
PDF pages as read : 30358 characters, 3933 tokens
|
|
PDF whitespace collapsed: 10655 characters, 2251 tokens
|
|
DOCX : 10657 characters, 2172 tokens
|