asmhatre/spring-ai
1
0
Fork 0
Code Activity

Add docker-model-runner module: Spring AI OpenAI starter against Docker Model Runner and Ollama with a benchmark runner (time to first token, decode tokens/second, container memory, tool-call success); real docker/model-runner v1.2.9 CPU image and Ollama 0.40.13 on a shared 2-vCPU sandbox, no GPU

Browse Source 
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_017ic4b4VQVEJWetTT1dYzyJ
This commit is contained in:
asmhatre
2026-10-29 06:24:00 +00:00
co-authored by Claude Sonnet 5.5
parent 6d792f00e0
commit c97bcf9adf
18 changed files with 668 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docker-model-runner/README.md
+35
View File
@@ -0,0 +1,35 @@
# docker-model-runner
Companion code for [Spring AI with Docker Model Runner vs Ollama](https://ankurm.com/spring-ai-docker-model-runner-vs-ollama/), part of the [Spring AI series](../README.md) on ankurm.com.
One Spring AI `OpenAiChatModel` (the `spring-ai-starter-model-openai` starter) pointed at two local servers by changing only `spring.ai.openai.base-url` and the model name. A benchmark runner sends the same prompts to each and reports time to first token, decode tokens/second, container memory (`docker stats`) and tool-call success over N runs.
Measured here: the real `docker/model-runner:latest` image (v1.2.9, llama.cpp CPU backend) and `ollama/ollama:latest` (0.40.13), both serving Qwen2.5-0.5B-Instruct at Q4_K_M, on a shared 2-vCPU sandbox with no GPU. Not measured: Docker Desktop's `docker model` CLI path and any GPU path.
| Component | Version |
|---|---|
| Spring Boot | 4.1.2 |
| Spring AI | 2.0.2 |
| Java | 25 (LTS) |
## Run
```bash
scripts/start-dmr.sh && scripts/start-ollama.sh # EXTRA_CA=/path/ca.crt if HTTPS is intercepted
# (the model sizes in run-all.sh are the 0.5B ones; pull the same models in both engines)
scripts/run-all.sh # writes output/dmr.txt and output/ollama.txt
mvn test # WireTest writes output/01-wire.txt
```
On Docker Desktop with `docker model` instead: `docker model pull hf.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M`, enable host-side TCP on port 12434, then `scripts/bench.sh dmr http://localhost:12434/engines/v1 <model id from /engines/v1/models> <container name>` (memory sampling needs a container name; skip it if there is none).
## Files
| File | What |
|---|---|
| `src/main/resources/application.yml` | the whole engine configuration |
| `Benchmark.java`, `WeatherTools.java` | throughput and tool-call harness |
| `WireTest.java` | records what the starter sends: path, key header, model field |
| `output/01-wire.txt` | request line, header and model name seen by a fake server |
| `output/02-engines.txt`, `03-dmr-extra-fields.txt` | `/models` listings, image ids, usage and timing fields |
| `output/dmr.txt`, `output/ollama.txt` | benchmark transcripts |