Add jlama module: run a 4-bit TinyLlama inside the JVM with Jlama on the Vector API, with tokens per second, streaming and an Ollama baseline

Co-Authored-By: Claude Sonnet 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01G8ikz8xdWuTP5yun8DZ1hk
2026-10-11 09:24:56 +00:00
parent 10545299f8
commit fdff2b7388
17 changed files with 564 additions and 0 deletions
@@ -0,0 +1,9 @@
+Machine: 2 CPUs, JDK 25.0.4.1+1-LTS, Vector API preferred float width 512 bits
+Model: tjake/TinyLlama-1.1B-Chat-v1.0-Jlama-Q4, 1132 MB on disk
+Load time: 665 ms; process resident memory 119 MB before, 175 MB after
+
+Question: In one sentence, what is a Java record?
+Answer: A Java record is a data structure that allows for the creation of immutable, named, and indexed collections of objects.
+
+Prompt tokens: 26, generated tokens: 24
+Prompt time: 4838 ms, generation time: 2762 ms
@@ -0,0 +1,11 @@
+Same question, 5 measured runs after 1 warm-up run, temperature 0, limit 64 tokens.
+
+run 1: prompt 26 tokens in 4902 ms (5.3 tok/s), generated 24 tokens in 3040 ms (7.9 tok/s)
+run 2: prompt 26 tokens in 4762 ms (5.5 tok/s), generated 24 tokens in 3061 ms (7.8 tok/s)
+run 3: prompt 26 tokens in 4527 ms (5.7 tok/s), generated 24 tokens in 2856 ms (8.4 tok/s)
+run 4: prompt 26 tokens in 4892 ms (5.3 tok/s), generated 24 tokens in 3007 ms (8.0 tok/s)
+run 5: prompt 26 tokens in 5203 ms (5.0 tok/s), generated 24 tokens in 2894 ms (8.3 tok/s)
+
+Generation tokens/s: min 7.8, median 8.0, max 8.4
+Prompt tokens/s:     min 5.0, median 5.3, max 5.7
+Prompt + generated tokens in the longest run: 50 (limit was 64)
@@ -0,0 +1,8 @@
+Callback receives one piece of text per generated token.
+
+Pieces received: 24, generated tokens reported: 24
+Time to first piece: 5230 ms (includes processing the 26-token prompt)
+Time to last piece: 7997 ms; whole call 8114 ms
+
+First pieces: [[A], [ Java], [ record], [ is], [ a], [ data], [ structure], [ that]]
+Joined: A Java record is a data structure that allows for the creation of immutable, named, and indexed collections of objects.
@@ -0,0 +1,13 @@
+Temperature 0.0, asked twice:
+  1: A Java record is a data structure that allows for the creation of immutable, named, and indexed collections of
+  2: A Java record is a data structure that allows for the creation of immutable, named, and indexed collections of
+  identical: true
+
+Temperature 0.9, asked three times:
+  1: In Java, a record is a type of data structure that stores and retrieves data in a logical, indexed
+  2: A Java record is a type of nested class (or interface) that represents a collection of related data objects in
+  3: A Java record is a data structure that contains multiple values and provides a way to represent multiple instances of the same
+  distinct answers: 3
+
+Temperature 0.0, limit 64, asked six more times: 1 distinct answer(s)
+  6x: A Java record is a data structure that allows for the creation of immutable, named, and indexed collections of objects.
@@ -0,0 +1,5 @@
+Second JVM, same classpath, started WITHOUT --add-modules jdk.incubator.vector.
+Exit code: 1
+Relevant lines:
+  Exception in thread "main" java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
+  Caused by: java.lang.ClassNotFoundException: jdk.incubator.vector.FloatVector
@@ -0,0 +1,11 @@
+Ollama 0.40.3, model "tl" created from tinyllama-1.1b-chat-v1.0.Q4_0.gguf (TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF).
+Same machine, same question, 1 warm-up run then 5 measured runs, temperature 0, limit 64 tokens.
+
+run 1: prompt 36 tokens (16 served from cache) in 190 ms, generated 40 tokens in 1196 ms (33.4 tok/s)
+run 2: prompt 36 tokens (35 served from cache) in 38 ms, generated 40 tokens in 1172 ms (34.1 tok/s)
+run 3: prompt 36 tokens (35 served from cache) in 28 ms, generated 40 tokens in 1187 ms (33.7 tok/s)
+run 4: prompt 36 tokens (35 served from cache) in 28 ms, generated 40 tokens in 1169 ms (34.2 tok/s)
+run 5: prompt 36 tokens (35 served from cache) in 29 ms, generated 40 tokens in 1215 ms (32.9 tok/s)
+
+Generation tokens/s: min 32.9, median 33.7, max 34.2
+Answer of the last run: A Java record is a data structure that combines the properties of a class and a struct. It is a type of data structure that allows for the creation of immutable objects with specific properties.
@@ -0,0 +1,4 @@
+The question is 26 prompt tokens long.
+
+Limit 30: generated 3 tokens (prompt + generated = 29)
+Limit 20: IllegalArgumentException: Prompt exceeds max tokens
@@ -0,0 +1,39 @@
+mvn dependency:tree for the jlama module:
+com.ankurm.agents:jlama:jar:1.0.0
+com.github.tjake:jlama-core:jar:0.8.4:compile
+com.fasterxml.jackson.core:jackson-databind:jar:2.15.2:compile
+com.fasterxml.jackson.core:jackson-annotations:jar:2.15.2:compile
+com.fasterxml.jackson.core:jackson-core:jar:2.15.2:compile
+com.google.guava:guava:jar:32.0.1-jre:compile
+com.google.guava:failureaccess:jar:1.0.1:compile
+com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
+com.google.code.findbugs:jsr305:jar:3.0.2:compile
+org.checkerframework:checker-qual:jar:3.33.0:compile
+com.google.errorprone:error_prone_annotations:jar:2.18.0:compile
+com.google.j2objc:j2objc-annotations:jar:2.8:compile
+org.jctools:jctools-core:jar:4.0.1:compile
+com.hubspot.jinjava:jinjava:jar:2.7.2:compile
+org.javassist:javassist:jar:3.24.1-GA:compile
+com.google.re2j:re2j:jar:1.2:compile
+org.apache.commons:commons-lang3:jar:3.9:compile
+commons-net:commons-net:jar:3.9.0:compile
+com.googlecode.java-ipv6:java-ipv6:jar:0.17:compile
+com.google.code.findbugs:annotations:jar:3.0.1:compile
+com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:jar:2.14.0:compile
+org.yaml:snakeyaml:jar:1.33:compile
+ch.obermuhlner:big-math:jar:2.0.0:compile
+net.fellbaum:jemoji:jar:1.4.1:compile
+org.jspecify:jspecify:jar:0.3.0:compile
+net.jafama:jafama:jar:2.3.2:compile
+org.slf4j:slf4j-api:jar:2.0.7:compile
+org.slf4j:slf4j-simple:jar:2.0.17:compile
+org.junit.jupiter:junit-jupiter:jar:6.1.3:test
+org.junit.jupiter:junit-jupiter-api:jar:6.1.3:test
+org.opentest4j:opentest4j:jar:1.3.0:test
+org.junit.platform:junit-platform-commons:jar:6.1.3:test
+org.apiguardian:apiguardian-api:jar:1.1.2:test
+org.junit.jupiter:junit-jupiter-params:jar:6.1.3:test
+org.junit.jupiter:junit-jupiter-engine:jar:6.1.3:test
+org.junit.platform:junit-platform-engine:jar:6.1.3:test
+org.assertj:assertj-core:jar:3.27.6:test
+net.bytebuddy:byte-buddy:jar:1.17.7:test