asmhatre/spring-ai
1
0
Fork 0
Code Activity

Add docker-model-runner module: Spring AI OpenAI starter against Docker Model Runner and Ollama with a benchmark runner (time to first token, decode tokens/second, container memory, tool-call success); real docker/model-runner v1.2.9 CPU image and Ollama 0.40.13 on a shared 2-vCPU sandbox, no GPU

Browse Source 
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_017ic4b4VQVEJWetTT1dYzyJ
This commit is contained in:
asmhatre
2026-10-29 06:24:00 +00:00
co-authored by Claude Sonnet 5.5
parent 6d792f00e0
commit c97bcf9adf
18 changed files with 668 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docker-model-runner/README.md
+35
View File
@@ -0,0 +1,35 @@
# docker-model-runner
Companion code for [Spring AI with Docker Model Runner vs Ollama](https://ankurm.com/spring-ai-docker-model-runner-vs-ollama/), part of the [Spring AI series](../README.md) on ankurm.com.
One Spring AI `OpenAiChatModel` (the `spring-ai-starter-model-openai` starter) pointed at two local servers by changing only `spring.ai.openai.base-url` and the model name. A benchmark runner sends the same prompts to each and reports time to first token, decode tokens/second, container memory (`docker stats`) and tool-call success over N runs.
Measured here: the real `docker/model-runner:latest` image (v1.2.9, llama.cpp CPU backend) and `ollama/ollama:latest` (0.40.13), both serving Qwen2.5-0.5B-Instruct at Q4_K_M, on a shared 2-vCPU sandbox with no GPU. Not measured: Docker Desktop's `docker model` CLI path and any GPU path.
| Component | Version |
|---|---|
| Spring Boot | 4.1.2 |
| Spring AI | 2.0.2 |
| Java | 25 (LTS) |
## Run
```bash
scripts/start-dmr.sh && scripts/start-ollama.sh # EXTRA_CA=/path/ca.crt if HTTPS is intercepted
# (the model sizes in run-all.sh are the 0.5B ones; pull the same models in both engines)
scripts/run-all.sh # writes output/dmr.txt and output/ollama.txt
mvn test # WireTest writes output/01-wire.txt
```
On Docker Desktop with `docker model` instead: `docker model pull hf.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M`, enable host-side TCP on port 12434, then `scripts/bench.sh dmr http://localhost:12434/engines/v1 <model id from /engines/v1/models> <container name>` (memory sampling needs a container name; skip it if there is none).
## Files
| File | What |
|---|---|
| `src/main/resources/application.yml` | the whole engine configuration |
| `Benchmark.java`, `WeatherTools.java` | throughput and tool-call harness |
| `WireTest.java` | records what the starter sends: path, key header, model field |
| `output/01-wire.txt` | request line, header and model name seen by a fake server |
| `output/02-engines.txt`, `03-dmr-extra-fields.txt` | `/models` listings, image ids, usage and timing fields |
| `output/dmr.txt`, `output/ollama.txt` | benchmark transcripts |
docker-model-runner/output/01-wire.txt
+8
View File
@@ -0,0 +1,8 @@
base-url = http://127.0.0.1:<port>/engines/v1
spring.ai.openai.chat.options.model = configured-model
call 1, model.call(new Prompt("ping"))
POST /engines/v1/chat/completions | Authorization: Bearer not-needed | model: configured-model
call 2, same prompt with OpenAiChatOptions.builder().temperature(0.0).build()
POST /engines/v1/chat/completions | Authorization: Bearer not-needed | model: gpt-5-mini
docker-model-runner/output/02-engines.txt
+12
View File
@@ -0,0 +1,12 @@
# GET http://localhost:12434/engines/v1/models (Docker Model Runner, docker/model-runner v1.2.9 image)
{"object":"list","data":[{"id":"huggingface.co/qwen/qwen2.5-0.5b-instruct-gguf:Q4_K_M","object":"model","created":1726813224,"owned_by":"docker","dmr":{"context_window":32768,"architecture":"qwen2","parameters":"630.17M","quantization":"MOSTLY_Q4_K_M","size":"462.96MiB"}}]}
# GET http://localhost:11434/v1/models (Ollama)
{"object":"list","data":[{"id":"qwen2.5:0.5b","object":"model","created":1793253801,"owned_by":"library"},{"id":"qwen2.5:1.5b","object":"model","created":1793253450,"owned_by":"library"}]}
# docker image digests
docker/model-runner:latest sha256:60e6a14d8cc331fab3e51d7b78bba98274b4d28573f4e0b345e475d52d167869
ollama/ollama:latest sha256:b2f0122c476bc9fe7ce679d8bb4bd555351e25e2554eefb156f61cb0a89cf429
ollama version is 0.40.13
docker-model-runner/output/03-dmr-extra-fields.txt
+4
View File
@@ -0,0 +1,4 @@
usage {'completion_tokens': 16, 'prompt_tokens': 31, 'total_tokens': 47, 'prompt_tokens_details': {'cached_tokens': 24}}
timings keys ['cache_n', 'predicted_ms', 'predicted_n', 'predicted_per_second', 'predicted_per_token_ms', 'prompt_ms', 'prompt_n', 'prompt_per_second', 'prompt_per_token_ms']
ollama usage {'prompt_tokens': 31, 'prompt_tokens_details': {'cached_tokens': 24}, 'completion_tokens': 10, 'total_tokens': 41}
ollama top-level keys ['choices', 'created', 'id', 'model', 'object', 'system_fingerprint', 'usage']
docker-model-runner/output/dmr.txt
+56
View File
@@ -0,0 +1,56 @@
engine label: dmr
java: 25.0.5+8-LTS
first request (includes model load): ttft 5018 ms, 48 completion tokens
throughput, 5 prompts x 3 repeats, temperature 0, max_tokens 48
p rep ttft_ms decode_tps wall_tps prompt compl
1 1 514 19.97 16.70 41 48
1 2 74 18.03 17.88 41 48
1 3 91 17.50 17.26 41 48
2 1 350 16.35 14.85 45 48
2 2 488 13.79 12.29 45 48
2 3 82 14.19 14.12 45 48
3 1 315 17.07 15.60 43 48
3 2 75 18.15 17.99 43 48
3 3 91 17.61 17.36 43 48
4 1 337 16.42 14.97 46 48
4 2 87 17.94 17.70 46 48
4 3 101 18.15 17.80 46 48
5 1 308 17.45 15.96 45 48
5 2 76 18.30 18.09 45 48
5 3 89 18.15 17.88 45 48
median ttft_ms 91 | median decode_tps 17.61 | median wall_tps 17.26 | n=15
tool calling (one tool), 10 runs, temperature 0.7 so runs differ
question: What is the temperature in Oslo in Celsius? Use the tools.
expected: currentTemperature(Oslo), answer contains 17
run 1 wrong or missing call calls=[celsiusToFahrenheit(12)] answer=The temperature in Oslo is 54 degrees Fahrenheit.
run 2 ok calls=[currentTemperature(Oslo)] answer=The temperature in Oslo is 17 degrees Celsius.
run 3 wrong or missing call calls=[celsiusToFahrenheit(12)] answer=The temperature in Oslo is 54 degrees Celsius.
run 4 wrong or missing call calls=[celsiusToFahrenheit(12)] answer=The temperature in Oslo is 54 degrees Fahrenheit.
run 5 wrong or missing call calls=[celsiusToFahrenheit(12)] answer=The temperature in Oslo is 54 degrees Fahrenheit.
run 6 wrong or missing call calls=[celsiusToFahrenheit(0)] answer=The temperature in Oslo is 32 degrees Celsius.
run 7 wrong or missing call calls=[celsiusToFahrenheit(10)] answer=The temperature in Oslo is 50 degrees Fahrenheit.
run 8 wrong or missing call calls=[celsiusToFahrenheit(0)] answer=The temperature in Oslo is 32 degrees Fahrenheit.
run 9 wrong or missing call calls=[celsiusToFahrenheit(14)] answer=The temperature in Oslo is 57 degrees Fahrenheit.
run 10 wrong or missing call calls=[celsiusToFahrenheit(12)] answer=The temperature in Oslo is 54 degrees Fahrenheit.
tool-call success (one tool): 1/10 (no call 0, wrong/missing call 9, wrong answer 0, errors 0)
tool calling (two tools in sequence), 10 runs, temperature 0.7 so runs differ
question: What is the temperature in Pune in Fahrenheit? Use the tools.
expected: currentTemperature(Pune), celsiusToFahrenheit(31), answer contains 88
run 1 wrong or missing call calls=[currentTemperature(Pune)] answer=The temperature in Pune is 31 degrees Fahrenheit.
run 2 wrong or missing call calls=[celsiusToFahrenheit(22)] answer=The temperature in Pune is 72 degrees Fahrenheit.
run 3 wrong or missing call calls=[currentTemperature(Pune)] answer=The temperature in Pune is 31 degrees Fahrenheit.
run 4 wrong or missing call calls=[celsiusToFahrenheit(25)] answer=The temperature in Pune is 77 degrees Fahrenheit.
run 5 wrong or missing call calls=[celsiusToFahrenheit(15)] answer=The temperature in Pune is 59 degrees Fahrenheit.
run 6 wrong or missing call calls=[celsiusToFahrenheit(0)] answer=The temperature in Pune is 32 degrees Fahrenheit.
run 7 wrong or missing call calls=[currentTemperature(Pune)] answer=The temperature in Pune is 31 degrees Fahrenheit.
run 8 wrong or missing call calls=[celsiusToFahrenheit(15)] answer=The temperature in Pune is 59 degrees Fahrenheit.
run 9 wrong or missing call calls=[celsiusToFahrenheit(32)] answer=The temperature in Pune is 90 degrees Fahrenheit.
run 10 wrong or missing call calls=[celsiusToFahrenheit(15)] answer=The temperature in Pune is 59 degrees Fahrenheit.
tool-call success (two tools in sequence): 0/10 (no call 0, wrong/missing call 10, wrong answer 0, errors 0)
memory (idle container memory before: 3.695MiB)
container memory during run: peak 546 MiB, median 511 MiB, 58 samples
docker-model-runner/output/ollama.txt
+56
View File
@@ -0,0 +1,56 @@
engine label: ollama
java: 25.0.5+8-LTS
first request (includes model load): ttft 4946 ms, 48 completion tokens
throughput, 5 prompts x 3 repeats, temperature 0, max_tokens 48
p rep ttft_ms decode_tps wall_tps prompt compl
1 1 247 17.59 16.40 41 48
1 2 66 18.43 18.32 41 48
1 3 63 18.02 17.93 41 48
2 1 289 18.99 17.35 45 48
2 2 92 18.93 18.58 45 48
2 3 65 18.56 18.43 45 48
3 1 322 18.36 16.63 43 48
3 2 58 20.94 20.79 43 48
3 3 84 17.65 17.46 43 48
4 1 287 19.89 18.07 46 48
4 2 68 19.30 19.15 46 48
4 3 70 18.57 18.44 46 48
5 1 282 17.87 16.46 45 48
5 2 90 18.76 18.44 45 48
5 3 68 18.74 18.55 45 48
median ttft_ms 84 | median decode_tps 18.57 | median wall_tps 18.32 | n=15
tool calling (one tool), 10 runs, temperature 0.7 so runs differ
question: What is the temperature in Oslo in Celsius? Use the tools.
expected: currentTemperature(Oslo), answer contains 17
run 1 ok calls=[currentTemperature(Oslo)] answer=The current temperature in Oslo is 17 degrees Celsius.
run 2 wrong or missing call calls=[celsiusToFahrenheit(0)] answer=The temperature in Oslo, Norway, is 32 degrees Celsius.
run 3 wrong or missing call calls=[celsiusToFahrenheit(20)] answer=The temperature in Oslo is 68 degrees Fahrenheit.
run 4 wrong or missing call calls=[celsiusToFahrenheit(0)] answer=The temperature in Oslo is 32 degrees Celsius.
run 5 wrong or missing call calls=[celsiusToFahrenheit(10)] answer=The temperature in Oslo is currently 50 degrees Fahrenheit.
run 6 wrong or missing call calls=[celsiusToFahrenheit(20)] answer=The temperature in Oslo, at 20 degrees Celsius, is 68 degree
run 7 wrong or missing call calls=[celsiusToFahrenheit(0)] answer=The temperature in Oslo in Celsius is 32 degrees.
run 8 wrong or missing call calls=[celsiusToFahrenheit(4)] answer=The temperature in Oslo (Oslo) is currently 39 degrees Celsi
run 9 wrong or missing call calls=[celsiusToFahrenheit(6)] answer=The temperature in Oslo is 43 degrees Fahrenheit.
run 10 wrong or missing call calls=[celsiusToFahrenheit(0)] answer=The temperature in Oslo is 32 degrees Celsius.
tool-call success (one tool): 1/10 (no call 0, wrong/missing call 9, wrong answer 0, errors 0)
tool calling (two tools in sequence), 10 runs, temperature 0.7 so runs differ
question: What is the temperature in Pune in Fahrenheit? Use the tools.
expected: currentTemperature(Pune), celsiusToFahrenheit(31), answer contains 88
run 1 wrong or missing call calls=[celsiusToFahrenheit(22)] answer=The temperature in Pune is 72 degrees Fahrenheit.
run 2 wrong or missing call calls=[celsiusToFahrenheit(32)] answer=The temperature in Pune is currently 90 degrees Fahrenheit.
run 3 wrong or missing call calls=[celsiusToFahrenheit(26)] answer=The temperature in Pune is 79 degrees Fahrenheit.
run 4 wrong or missing call calls=[celsiusToFahrenheit(25)] answer=The temperature in Pune is 77 degrees Fahrenheit.
run 5 wrong or missing call calls=[celsiusToFahrenheit(26)] answer=The temperature in Pune (Pune) is 79 degrees Fahrenheit. It'
run 6 wrong or missing call calls=[celsiusToFahrenheit(23)] answer=The temperature in Pune is 73 degrees Fahrenheit.
run 7 wrong or missing call calls=[celsiusToFahrenheit(23)] answer=The temperature in Pune is 73 degrees Fahrenheit.
run 8 wrong or missing call calls=[celsiusToFahrenheit(15)] answer=The temperature in Pune is 59 degrees Fahrenheit.
run 9 wrong or missing call calls=[celsiusToFahrenheit(0)] answer=The temperature in Pune is 32 degrees Fahrenheit.
run 10 wrong or missing call calls=[celsiusToFahrenheit(22)] answer=The temperature in Pune is 72 degrees Fahrenheit.
tool-call success (two tools in sequence): 0/10 (no call 0, wrong/missing call 10, wrong answer 0, errors 0)
memory (idle container memory before: 12.15MiB)
container memory during run: peak 510 MiB, median 509 MiB, 55 samples
docker-model-runner/pom.xml
+64
View File
@@ -0,0 +1,64 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>4.1.2</version>
<relativePath/>
</parent>
<groupId>com.ankurm</groupId>
<artifactId>docker-model-runner</artifactId>
<version>1.0.0</version>
<name>docker-model-runner</name>
<description>Spring AI 2.0 against Docker Model Runner and Ollama through the same OpenAI-compatible client, with a benchmark harness for tokens per second, memory and tool-calling reliability.</description>
<properties>
<java.version>25</java.version>
<spring-ai.version>2.0.2</spring-ai.version>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-bom</artifactId>
<version>${spring-ai.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-starter-model-openai</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<argLine>-Duser.timezone=UTC -Dstdout.encoding=UTF-8 -Dfile.encoding=UTF-8</argLine>
</configuration>
</plugin>
</plugins>
</build>
</project>
docker-model-runner/scripts/bench.sh
Executable
+34
View File
@@ -0,0 +1,34 @@
#!/usr/bin/env bash
# bench.sh <label> <base-url> <model> <container> [extra bench args]
# Runs the Spring AI benchmark against one OpenAI-compatible server and samples the container's
# memory with `docker stats` once a second while it runs. Writes output/<label>.txt.
# Docker Model Runner : bench.sh dmr http://localhost:12434/engines/v1 hf.co/qwen/qwen2.5-0.5b-instruct-gguf:Q4_K_M dmr
# Ollama : bench.sh ollama http://localhost:11434/v1 qwen2.5:0.5b ollama
set -euo pipefail
cd "$(dirname "$0")/.."
LABEL=$1; URL=$2; MODEL=$3; CTR=$4; shift 4
[ -f target/docker-model-runner-1.0.0.jar ] || mvn -B -q -T1 package -DskipTests
mkdir -p output
MEM=$(mktemp)
( while true; do docker stats --no-stream --format '{{.MemUsage}}' "$CTR" | cut -d/ -f1 >> "$MEM"; sleep 1; done ) &
SAMPLER=$!
trap 'kill $SAMPLER 2>/dev/null || true' EXIT
echo "idle container memory before: $(docker stats --no-stream --format '{{.MemUsage}}' "$CTR" | cut -d/ -f1)" > "output/$LABEL.mem.tmp"
java -jar target/docker-model-runner-1.0.0.jar --bench.run=true --bench.label="$LABEL" \
--bench.out-file="output/$LABEL.txt" --spring.ai.openai.base-url="$URL" \
--spring.ai.openai.chat.options.model="$MODEL" "$@" > "output/$LABEL.log" 2>&1 || { tail -20 "output/$LABEL.log"; exit 1; }
kill $SAMPLER 2>/dev/null || true
python3 - "$MEM" "output/$LABEL.mem.tmp" "output/$LABEL.txt" <<'P'
import sys,re
def mb(s):
s=s.strip(); m=re.match(r'([\d.]+)(\w+)',s)
if not m: return None
v=float(m[1]); u=m[2]
return v*{'B':1/1048576,'KiB':1/1024,'MiB':1,'GiB':1024}[u]
vals=[x for x in (mb(l) for l in open(sys.argv[1])) if x is not None]
idle=open(sys.argv[2]).read().strip()
with open(sys.argv[3],'a') as f:
f.write(f"\nmemory ({idle})\ncontainer memory during run: peak {max(vals):.0f} MiB, median {sorted(vals)[len(vals)//2]:.0f} MiB, {len(vals)} samples\n")
P
rm -f "output/$LABEL.mem.tmp" "output/$LABEL.log"
cat "output/$LABEL.txt"
docker-model-runner/scripts/run-all.sh
Executable
+12
View File
@@ -0,0 +1,12 @@
#!/usr/bin/env bash
# Regenerates output/dmr.txt and output/ollama.txt. Needs both servers up (scripts/start-dmr.sh,
# scripts/start-ollama.sh, or your own Docker Desktop Model Runner on :12434). Each engine is
# restarted first so its memory figure is a fresh container with only this one model loaded.
# Run it on an otherwise idle machine; the figures are only comparable within one run.
set -euo pipefail
cd "$(dirname "$0")/.."
ARGS="${BENCH_ARGS:---bench.repeats=3 --bench.max-tokens=48 --bench.tool-runs=10}"
docker restart ollama >/dev/null; sleep 5
scripts/bench.sh ollama http://localhost:11434/v1 qwen2.5:0.5b ollama $ARGS
docker restart dmr >/dev/null; sleep 5
scripts/bench.sh dmr http://localhost:12434/engines/v1 huggingface.co/qwen/qwen2.5-0.5b-instruct-gguf:Q4_K_M dmr $ARGS
docker-model-runner/scripts/start-dmr.sh
Executable
+26
View File
@@ -0,0 +1,26 @@
#!/usr/bin/env bash
# Starts the real Docker Model Runner server image (docker/model-runner, llama.cpp backend, CPU
# build here) on host port 12434 and pulls the model from Hugging Face into it.
#
# On a machine with Docker Desktop (4.40+) or Docker Engine plus the docker-model-plugin you do
# NOT need this script: `docker model pull hf.co/Qwen/Qwen2.5-1.5B-Instruct-GGUF:Q4_K_M` and
# enable host-side TCP support (Docker Desktop: Settings > Features in development > Enable
# host-side TCP support, port 12434). This script is the same server in a container, which is
# what lets the benchmark run on a Linux box with no Desktop.
#
# EXTRA_CA=/path/to/ca.crt (optional) bakes a corporate/proxy CA into the image, needed when
# outbound HTTPS is intercepted: the server downloads the model itself, from inside the container.
set -euo pipefail
NAME="${NAME:-dmr}"
IMAGE=docker/model-runner:latest
if [ -n "${EXTRA_CA:-}" ]; then
D=$(mktemp -d); cp "$EXTRA_CA" "$D/extra-ca.crt"
printf 'FROM %s\nUSER root\nCOPY extra-ca.crt /usr/local/share/ca-certificates/extra-ca.crt\nRUN update-ca-certificates\nUSER modelrunner\n' "$IMAGE" > "$D/Dockerfile"
docker build -q -t dmr-ca "$D" >/dev/null; IMAGE=dmr-ca
fi
docker rm -f "$NAME" >/dev/null 2>&1 || true
docker run -d --name "$NAME" -p 12434:12434 "$IMAGE" >/dev/null
until curl -sf localhost:12434/engines/v1/models >/dev/null; do sleep 1; done
curl -s -X POST localhost:12434/models/create -H 'Content-Type: application/json' \
-d '{"from":"hf.co/Qwen/Qwen2.5-1.5B-Instruct-GGUF:Q4_K_M"}' | tail -c 120; echo
curl -s localhost:12434/engines/v1/models; echo
docker-model-runner/scripts/start-ollama.sh
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env bash
# Starts ollama/ollama on host port 11434 and pulls qwen2.5:1.5b (Ollama's default tag is Q4_K_M,
# the same quantisation as the GGUF the Docker Model Runner script pulls).
# EXTRA_CA=/path/to/ca.crt (optional) as in start-dmr.sh.
set -euo pipefail
NAME="${NAME:-ollama}"
IMAGE=ollama/ollama:latest
if [ -n "${EXTRA_CA:-}" ]; then
D=$(mktemp -d); cp "$EXTRA_CA" "$D/extra-ca.crt"
printf 'FROM %s\nCOPY extra-ca.crt /usr/local/share/ca-certificates/extra-ca.crt\nRUN update-ca-certificates\n' "$IMAGE" > "$D/Dockerfile"
docker build -q -t ollama-ca "$D" >/dev/null; IMAGE=ollama-ca
fi
docker rm -f "$NAME" >/dev/null 2>&1 || true
docker run -d --name "$NAME" -p 11434:11434 "$IMAGE" >/dev/null
until docker exec "$NAME" ollama list >/dev/null 2>&1; do sleep 1; done
docker exec "$NAME" ollama pull qwen2.5:1.5b
docker exec "$NAME" ollama show qwen2.5:1.5b | head -12
docker-model-runner/src/main/java/com/ankurm/dmr/BenchProperties.java
+14
View File
@@ -0,0 +1,14 @@
package com.ankurm.dmr;
import org.springframework.boot.context.properties.ConfigurationProperties;
/** Settings for the benchmark itself (the engine settings are plain spring.ai.openai.* ones). */
@ConfigurationProperties("bench")
public record BenchProperties(String label, Integer repeats, Integer toolRuns, Integer maxTokens, String outFile) {
public BenchProperties {
if (label == null) label = "engine";
if (repeats == null) repeats = 3;
if (toolRuns == null) toolRuns = 20;
if (maxTokens == null) maxTokens = 128;
}
}
docker-model-runner/src/main/java/com/ankurm/dmr/Benchmark.java
+176
View File
@@ -0,0 +1,176 @@
package com.ankurm.dmr;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Duration;
import java.time.Instant;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.model.ChatModel;
import org.springframework.ai.chat.model.ChatResponse;
import org.springframework.ai.chat.prompt.Prompt;
import org.springframework.ai.openai.OpenAiChatOptions;
import org.springframework.boot.ApplicationArguments;
import org.springframework.boot.ApplicationRunner;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.stereotype.Component;
import reactor.core.publisher.Flux;
/**
* Runs the same prompts against whichever OpenAI-compatible server spring.ai.openai.base-url
* points at and prints one report. Only runs when {@code --bench.run=true} is passed, so the
* tests can start the context without triggering it.
*
* <p>Tokens per second is computed the same way for every engine, from what the client sees:
* the time to the first streamed chunk, the time to the last, and the completion token count the
* server reports in the final usage chunk. Server-side timers are not used because Ollama's
* /v1 endpoint does not return them and Docker Model Runner's llama.cpp does.
*/
@Component
@EnableConfigurationProperties(BenchProperties.class)
public class Benchmark implements ApplicationRunner {
static final List<String> PROMPTS = List.of(
"Explain in three sentences what a Java virtual thread is.",
"Write a Java method that reverses a string, with a one-line comment.",
"List five differences between a HashMap and a TreeMap, one per line.",
"Summarise the CAP theorem for a new developer in under 80 words.",
"What does the final keyword do on a class, a method and a variable?");
private final ChatModel model;
private final BenchProperties props;
/**
* Per-call options REPLACE the configured defaults rather than merging with them: an
* OpenAiChatOptions built here without .model(..) goes out as the library default
* (gpt-5-mini) and the local server answers 404. See output/01-options-replace-defaults.txt.
*/
private final String modelName;
public Benchmark(ChatModel model, BenchProperties props,
@org.springframework.beans.factory.annotation.Value("${spring.ai.openai.chat.options.model}") String modelName) {
this.model = model;
this.props = props;
this.modelName = modelName;
}
record Sample(double ttftMs, double decodeTps, double wallTps, int promptTokens, int completionTokens) {}
@Override
public void run(ApplicationArguments args) throws IOException {
if (!args.containsOption("bench.run")) return;
StringBuilder out = new StringBuilder();
out.append("engine label: ").append(props.label()).append('\n');
out.append("java: ").append(Runtime.version()).append('\n');
// One throw-away call: the first request makes the engine load the model into memory.
Sample cold = stream(PROMPTS.get(0));
out.append(String.format(Locale.ROOT, "first request (includes model load): ttft %.0f ms, %d completion tokens%n",
cold.ttftMs, cold.completionTokens));
List<Sample> all = new ArrayList<>();
out.append(String.format("%nthroughput, %d prompts x %d repeats, temperature 0, max_tokens %d%n",
PROMPTS.size(), props.repeats(), props.maxTokens()));
out.append(String.format("%-4s %-4s %9s %11s %9s %7s %7s%n", "p", "rep", "ttft_ms", "decode_tps", "wall_tps", "prompt", "compl"));
for (int p = 0; p < PROMPTS.size(); p++) {
for (int r = 1; r <= props.repeats(); r++) {
Sample s = stream(PROMPTS.get(p));
all.add(s);
out.append(String.format(Locale.ROOT, "%-4d %-4d %9.0f %11.2f %9.2f %7d %7d%n", p + 1, r, s.ttftMs,
s.decodeTps, s.wallTps, s.promptTokens, s.completionTokens));
}
}
out.append(String.format(Locale.ROOT, "%nmedian ttft_ms %.0f | median decode_tps %.2f | median wall_tps %.2f | n=%d%n",
median(all.stream().mapToDouble(Sample::ttftMs).toArray()),
median(all.stream().mapToDouble(Sample::decodeTps).toArray()),
median(all.stream().mapToDouble(Sample::wallTps).toArray()), all.size()));
out.append(toolCalling("one tool", "What is the temperature in Oslo in Celsius? Use the tools.", "currentTemperature(Oslo)", null, "17"));
out.append(toolCalling("two tools in sequence", "What is the temperature in Pune in Fahrenheit? Use the tools.", "currentTemperature(Pune)", "celsiusToFahrenheit(31)", "88"));
String report = out.toString();
System.out.println(report);
if (props.outFile() != null) Files.writeString(Path.of(props.outFile()), report);
}
/** One streamed completion; timings are taken on the client. */
Sample stream(String text) {
OpenAiChatOptions opts = OpenAiChatOptions.builder().model(modelName).temperature(0.0).maxTokens(props.maxTokens())
.streamOptions(OpenAiChatOptions.StreamOptions.builder().includeUsage(true).build()).build();
Instant start = Instant.now();
Flux<ChatResponse> flux = model.stream(new Prompt(text, opts));
long[] first = {0};
long[] last = {0};
int[] usage = {0, 0};
flux.doOnNext(r -> {
long now = System.nanoTime();
String t = r.getResult() == null || r.getResult().getOutput() == null ? null : r.getResult().getOutput().getText();
if (t != null && !t.isEmpty()) {
if (first[0] == 0) first[0] = now;
last[0] = now;
}
if (r.getMetadata() != null && r.getMetadata().getUsage() != null
&& r.getMetadata().getUsage().getCompletionTokens() != null
&& r.getMetadata().getUsage().getCompletionTokens() > 0) {
usage[0] = r.getMetadata().getUsage().getPromptTokens();
usage[1] = r.getMetadata().getUsage().getCompletionTokens();
}
}).blockLast(Duration.ofMinutes(10));
long end = System.nanoTime();
long begin = end - Duration.between(start, Instant.now()).toNanos();
double ttft = (first[0] - begin) / 1e6;
double decodeSecs = (last[0] - first[0]) / 1e9;
double decode = decodeSecs > 0 ? (usage[1] - 1) / decodeSecs : 0;
double wall = usage[1] / ((end - begin) / 1e9);
return new Sample(ttft, decode, wall, usage[0], usage[1]);
}
/**
* N runs of a question that needs one or two tools. A run succeeds only if the expected
* call(s) were made with the right arguments and the final answer contains the expected number.
*/
String toolCalling(String name, String question, String call1, String call2, String expected) {
int n = props.toolRuns();
int ok = 0, noCall = 0, wrongArgs = 0, noAnswer = 0, errors = 0;
StringBuilder sb = new StringBuilder();
sb.append(String.format("%ntool calling (%s), %d runs, temperature 0.7 so runs differ%n", name, n));
sb.append("question: ").append(question).append('\n');
sb.append("expected: ").append(call1).append(call2 == null ? "" : ", " + call2).append(", answer contains ").append(expected).append('\n');
for (int i = 1; i <= n; i++) {
WeatherTools tools = new WeatherTools();
String answer;
try {
answer = ChatClient.create(model).prompt().user(question)
.options(OpenAiChatOptions.builder().model(modelName).temperature(0.7).maxTokens(200))
.tools(tools).call().content();
} catch (RuntimeException e) {
errors++;
sb.append(String.format("run %2d ERROR %s%n", i, e.getClass().getSimpleName()));
continue;
}
boolean right = tools.calls.stream().anyMatch(c -> c.equalsIgnoreCase(call1))
&& (call2 == null || tools.calls.contains(call2));
boolean hasAnswer = answer != null && answer.contains(expected);
String verdict;
if (tools.calls.isEmpty()) { noCall++; verdict = "no tool call"; }
else if (!right) { wrongArgs++; verdict = "wrong or missing call"; }
else if (!hasAnswer) { noAnswer++; verdict = "answer lacks " + expected; }
else { ok++; verdict = "ok"; }
String flat = answer == null ? "null" : answer.replaceAll("\\s+", " ");
sb.append(String.format("run %2d %-22s calls=%s answer=%s%n", i, verdict, tools.calls, flat.substring(0, Math.min(60, flat.length()))));
}
sb.append(String.format("tool-call success (%s): %d/%d (no call %d, wrong/missing call %d, wrong answer %d, errors %d)%n",
name, ok, n, noCall, wrongArgs, noAnswer, errors));
return sb.toString();
}
static double median(double[] v) {
double[] c = v.clone();
java.util.Arrays.sort(c);
return c.length == 0 ? 0 : c.length % 2 == 1 ? c[c.length / 2] : (c[c.length / 2 - 1] + c[c.length / 2]) / 2;
}
}
docker-model-runner/src/main/java/com/ankurm/dmr/DmrApplication.java
+16
View File
@@ -0,0 +1,16 @@
package com.ankurm.dmr;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
/**
* The whole application is the Spring AI OpenAI starter plus the benchmark in {@link Benchmark}.
* There is no Docker Model Runner (or Ollama) specific class anywhere: which engine answers is
* decided by {@code spring.ai.openai.base-url} and {@code spring.ai.openai.chat.options.model}.
*/
@SpringBootApplication
public class DmrApplication {
public static void main(String[] args) {
SpringApplication.run(DmrApplication.class, args);
}
}
docker-model-runner/src/main/java/com/ankurm/dmr/WeatherTools.java
+34
View File
@@ -0,0 +1,34 @@
package com.ankurm.dmr;
import java.util.List;
import java.util.concurrent.CopyOnWriteArrayList;
import org.springframework.ai.tool.annotation.Tool;
import org.springframework.ai.tool.annotation.ToolParam;
/**
* Two deterministic tools. The temperatures are made up on purpose (31 and 17 are not what any
* model would guess), so a final answer containing the number proves the model read the tool
* result instead of answering from memory.
*/
public class WeatherTools {
/** Every call the model actually made, in order, as "tool(argument)". */
public final List<String> calls = new CopyOnWriteArrayList<>();
@Tool(description = "Get the current temperature in degrees Celsius for a city")
public int currentTemperature(@ToolParam(description = "City name, e.g. Pune") String city) {
calls.add("currentTemperature(" + city + ")");
return switch (city.trim().toLowerCase()) {
case "pune" -> 31;
case "oslo" -> 17;
default -> 22;
};
}
@Tool(description = "Convert a temperature from degrees Celsius to degrees Fahrenheit")
public int celsiusToFahrenheit(@ToolParam(description = "Temperature in Celsius") int celsius) {
calls.add("celsiusToFahrenheit(" + celsius + ")");
return Math.round(celsius * 9f / 5f + 32f);
}
}
docker-model-runner/src/main/resources/application.yml
+19
View File
@@ -0,0 +1,19 @@
# The OpenAI starter used as a generic OpenAI-compatible client. Defaults point at Docker Model
# Runner's TCP port on the host. Override on the command line to measure anything else:
# --spring.ai.openai.base-url=http://localhost:11434/v1 --spring.ai.openai.chat.options.model=qwen2.5:1.5b
# base-url must END IN /v1: the starter does not append it (OpenAiSetup.OPENAI_URL is
# "https://api.openai.com/v1") and the SDK adds only chat/completions to it.
spring:
main:
web-application-type: none
ai:
openai:
base-url: http://localhost:12434/engines/v1
# Local engines ignore the key but the client refuses to build without one.
api-key: not-needed
# The SDK default read timeout is 60 s; a CPU-only box can need longer for one streamed answer.
timeout: 10m
chat:
options:
model: huggingface.co/qwen/qwen2.5-0.5b-instruct-gguf:Q4_K_M
temperature: 0.0
docker-model-runner/src/test/java/com/ankurm/dmr/WireTest.java
+84
View File
@@ -0,0 +1,84 @@
package com.ankurm.dmr;
import static org.assertj.core.api.Assertions.assertThat;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
import com.sun.net.httpserver.HttpServer;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.ai.chat.model.ChatModel;
import org.springframework.ai.chat.prompt.Prompt;
import org.springframework.ai.openai.OpenAiChatOptions;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.test.context.DynamicPropertyRegistry;
import org.springframework.test.context.DynamicPropertySource;
/**
* What the OpenAI starter really puts on the wire when it is pointed at a local server. A fake
* server records the request line, the Authorization header and the model field, so the three
* questions people ask (does it add /v1? does it need a key? what model name goes out?) are
* answered by a request instead of by memory. Writes output/01-wire.txt.
*/
@SpringBootTest(properties = { "spring.ai.openai.api-key=not-needed",
"spring.ai.openai.chat.options.model=configured-model" })
class WireTest {
static HttpServer server;
static final List<String> seen = new ArrayList<>();
static String prefix = "/engines/v1";
@DynamicPropertySource
static void props(DynamicPropertyRegistry r) throws IOException {
server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.createContext("/", ex -> {
String body = new String(ex.getRequestBody().readAllBytes(), StandardCharsets.UTF_8);
String model = body.replaceAll("(?s).*\"model\"\\s*:\\s*\"([^\"]*)\".*", "$1");
seen.add(ex.getRequestMethod() + " " + ex.getRequestURI().getPath() + " | Authorization: "
+ ex.getRequestHeaders().getFirst("Authorization") + " | model: " + model);
byte[] ok = "{\"id\":\"x\",\"object\":\"chat.completion\",\"created\":1,\"model\":\"m\",\"choices\":[{\"index\":0,\"finish_reason\":\"stop\",\"message\":{\"role\":\"assistant\",\"content\":\"pong\"}}],\"usage\":{\"prompt_tokens\":3,\"completion_tokens\":1,\"total_tokens\":4}}"
.getBytes(StandardCharsets.UTF_8);
ex.getResponseHeaders().add("Content-Type", "application/json");
ex.sendResponseHeaders(200, ok.length);
ex.getResponseBody().write(ok);
ex.close();
});
server.start();
r.add("spring.ai.openai.base-url", () -> "http://127.0.0.1:" + server.getAddress().getPort() + prefix);
}
@Autowired
ChatModel model;
@AfterEach
void stop() {
// the server is shared by the class; closed by the JVM
}
@Test
void whatGoesOnTheWire() throws IOException {
seen.clear();
model.call(new Prompt("ping"));
model.call(new Prompt("ping", OpenAiChatOptions.builder().temperature(0.0).build()));
assertThat(seen).hasSize(2);
// base-url is used as given: the SDK appends only chat/completions, never /v1
assertThat(seen.get(0)).startsWith("POST /engines/v1/chat/completions");
assertThat(seen.get(0)).contains("Authorization: Bearer not-needed").contains("model: configured-model");
// per-call options replace the configured defaults: no .model(..) means the library default
assertThat(seen.get(1)).doesNotContain("configured-model");
Files.createDirectories(Path.of("output"));
Files.writeString(Path.of("output/01-wire.txt"), "base-url = http://127.0.0.1:<port>/engines/v1\n"
+ "spring.ai.openai.chat.options.model = configured-model\n\n"
+ "call 1, model.call(new Prompt(\"ping\"))\n " + seen.get(0) + "\n\n"
+ "call 2, same prompt with OpenAiChatOptions.builder().temperature(0.0).build()\n " + seen.get(1) + "\n");
}
}