asmhatre/java-ai-agents
1
0
Fork 0
Code Activity

Add jlama module: run a 4-bit TinyLlama inside the JVM with Jlama on the Vector API, with tokens per second, streaming and an Ollama baseline

Browse Source 
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01G8ikz8xdWuTP5yun8DZ1hk
This commit is contained in:
Claude
2026-10-11 09:24:56 +00:00
parent 10545299f8
commit fdff2b7388
17 changed files with 564 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
jlama/output/06-ollama-baseline.txt
+11
View File
@@ -0,0 +1,11 @@
Ollama 0.40.3, model "tl" created from tinyllama-1.1b-chat-v1.0.Q4_0.gguf (TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF).
Same machine, same question, 1 warm-up run then 5 measured runs, temperature 0, limit 64 tokens.
run 1: prompt 36 tokens (16 served from cache) in 190 ms, generated 40 tokens in 1196 ms (33.4 tok/s)
run 2: prompt 36 tokens (35 served from cache) in 38 ms, generated 40 tokens in 1172 ms (34.1 tok/s)
run 3: prompt 36 tokens (35 served from cache) in 28 ms, generated 40 tokens in 1187 ms (33.7 tok/s)
run 4: prompt 36 tokens (35 served from cache) in 28 ms, generated 40 tokens in 1169 ms (34.2 tok/s)
run 5: prompt 36 tokens (35 served from cache) in 29 ms, generated 40 tokens in 1215 ms (32.9 tok/s)
Generation tokens/s: min 32.9, median 33.7, max 34.2
Answer of the last run: A Java record is a data structure that combines the properties of a class and a struct. It is a type of data structure that allows for the creation of immutable objects with specific properties.