asmhatre/java-ai-agents
1
0
Fork 0
Code Activity

Add jlama module: run a 4-bit TinyLlama inside the JVM with Jlama on the Vector API, with tokens per second, streaming and an Ollama baseline

Browse Source 
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01G8ikz8xdWuTP5yun8DZ1hk
This commit is contained in:
Claude
2026-10-11 09:24:56 +00:00
parent 10545299f8
commit fdff2b7388
17 changed files with 564 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
jlama/output/02-tokens-per-second.txt
+11
View File
@@ -0,0 +1,11 @@
Same question, 5 measured runs after 1 warm-up run, temperature 0, limit 64 tokens.
run 1: prompt 26 tokens in 4902 ms (5.3 tok/s), generated 24 tokens in 3040 ms (7.9 tok/s)
run 2: prompt 26 tokens in 4762 ms (5.5 tok/s), generated 24 tokens in 3061 ms (7.8 tok/s)
run 3: prompt 26 tokens in 4527 ms (5.7 tok/s), generated 24 tokens in 2856 ms (8.4 tok/s)
run 4: prompt 26 tokens in 4892 ms (5.3 tok/s), generated 24 tokens in 3007 ms (8.0 tok/s)
run 5: prompt 26 tokens in 5203 ms (5.0 tok/s), generated 24 tokens in 2894 ms (8.3 tok/s)
Generation tokens/s: min 7.8, median 8.0, max 8.4
Prompt tokens/s: min 5.0, median 5.3, max 5.7
Prompt + generated tokens in the longest run: 50 (limit was 64)