diff --git a/README.md b/README.md index 1c3f5f7..f845897 100644 --- a/README.md +++ b/README.md @@ -32,5 +32,6 @@ Runnable companion code for the Spring AI articles on [ankurm.com](https://ankur | [`graphrag-neo4j/`](graphrag-neo4j) | Entities and relations extracted with `ChatClient.entity(...)` into a typed Neo4j graph with a vector index on entity names (Testcontainers, `neo4j:2026.10.1-community`; also run on 5.26), then multi-hop retrieval by span linking, variable-length Cypher paths and path ranking, against plain vector RAG over the same 35 chunks and 9 questions: complete evidence 7/9 for the graph, 5/9 for vector k=8, 2/9 for k=4, with the traps reproduced (duplicate entity under an alias, an embedding merge that joins the wrong incidents, a reply with a sentence before the JSON fence, a hostile relation type, hop explosion). **The extraction step is a rule-based stand-in, not a language model**; Neo4j, embeddings and Cypher are real. Spring Boot 4.1.2, Spring AI 2.0.2, Java 25. | [GraphRAG in Java with Neo4j and Spring AI](https://ankurm.com/graphrag-java-neo4j-spring-ai-knowledge-graph/) | | [`image-gen/`](image-gen) | `ImageModel` and `ImagePrompt` with `OpenAiImageOptions` (what the real `OpenAiImageModel` puts on the wire, a per-call options object that silently resets the model, a `List` prompt that throws), a prompt gate with `ModerationModel` that fails closed, and safe storage in MinIO: sniffed content type, byte and pixel caps, server-chosen keys, a fetcher for provider URLs, presigned reads. A per-image price table from the vendors' pages dated 2026-10-28. No image provider called: the `ImageModel` and `ModerationModel` are hand-written fakes and a mock server; the MinIO tests use a real server through Testcontainers. Spring Boot 4.1.2, Spring AI 2.0.2, Java 25. | [Image Generation and Vision in Spring AI: ImageModel, Prompt Options and Safe Storage](https://ankurm.com/spring-ai-image-generation-imagemodel-s3-minio-safe-storage/) | | [`batch-llm/`](batch-llm) | 100,000 synthetic support tickets classified by a `ChatClient` inside a Spring Batch 6 chunk step: a Resilience4j rate limiter and retry around the model call (and where Spring Batch's own retry and skip belong instead), replies that are not valid JSON skipped and recorded, a restart after `kill -9` in a second JVM that ends with a byte-identical result, and a cost ledger row committed with every chunk. **The model is a deterministic simulated `ChatModel`, not a language model; token counts are estimates and prices are configurable placeholders.** Spring Boot 4.1.2, Spring Batch 6.0.6, Spring AI 2.0.2, Resilience4j 2.4.0, Java 25. | [Spring AI + Spring Batch: Classifying 100,000 Support Tickets with an LLM](https://ankurm.com/spring-ai-spring-batch-llm-ticket-classification/) | +| [`docker-model-runner/`](docker-model-runner) | The Spring AI OpenAI starter pointed at Docker Model Runner and at Ollama by changing only `base-url` and the model name (no `/v1` is appended: the path is used exactly as given), plus a benchmark runner: time to first token, decode tokens/second, container memory from `docker stats` and tool-call success over 10 runs. Measured with the real `docker/model-runner` v1.2.9 CPU image and Ollama 0.40.13 on one shared 2-vCPU sandbox, Qwen2.5-0.5B Q4_K_M; no GPU, and Docker Desktop's `docker model` path was not used. Spring Boot 4.1.2, Spring AI 2.0.2, Java 25. | [Spring AI with Docker Model Runner vs Ollama: Local LLMs Benchmarked on the Same Laptop](https://ankurm.com/spring-ai-docker-model-runner-vs-ollama/) | Upgrading from Spring AI 1.x: [migration guide](https://ankurm.com/spring-ai-1-to-2-migration-guide/). diff --git a/docker-model-runner/README.md b/docker-model-runner/README.md new file mode 100644 index 0000000..276796b --- /dev/null +++ b/docker-model-runner/README.md @@ -0,0 +1,35 @@ +# docker-model-runner + +Companion code for [Spring AI with Docker Model Runner vs Ollama](https://ankurm.com/spring-ai-docker-model-runner-vs-ollama/), part of the [Spring AI series](../README.md) on ankurm.com. + +One Spring AI `OpenAiChatModel` (the `spring-ai-starter-model-openai` starter) pointed at two local servers by changing only `spring.ai.openai.base-url` and the model name. A benchmark runner sends the same prompts to each and reports time to first token, decode tokens/second, container memory (`docker stats`) and tool-call success over N runs. + +Measured here: the real `docker/model-runner:latest` image (v1.2.9, llama.cpp CPU backend) and `ollama/ollama:latest` (0.40.13), both serving Qwen2.5-0.5B-Instruct at Q4_K_M, on a shared 2-vCPU sandbox with no GPU. Not measured: Docker Desktop's `docker model` CLI path and any GPU path. + +| Component | Version | +|---|---| +| Spring Boot | 4.1.2 | +| Spring AI | 2.0.2 | +| Java | 25 (LTS) | + +## Run + +```bash +scripts/start-dmr.sh && scripts/start-ollama.sh # EXTRA_CA=/path/ca.crt if HTTPS is intercepted +# (the model sizes in run-all.sh are the 0.5B ones; pull the same models in both engines) +scripts/run-all.sh # writes output/dmr.txt and output/ollama.txt +mvn test # WireTest writes output/01-wire.txt +``` + +On Docker Desktop with `docker model` instead: `docker model pull hf.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M`, enable host-side TCP on port 12434, then `scripts/bench.sh dmr http://localhost:12434/engines/v1 ` (memory sampling needs a container name; skip it if there is none). + +## Files + +| File | What | +|---|---| +| `src/main/resources/application.yml` | the whole engine configuration | +| `Benchmark.java`, `WeatherTools.java` | throughput and tool-call harness | +| `WireTest.java` | records what the starter sends: path, key header, model field | +| `output/01-wire.txt` | request line, header and model name seen by a fake server | +| `output/02-engines.txt`, `03-dmr-extra-fields.txt` | `/models` listings, image ids, usage and timing fields | +| `output/dmr.txt`, `output/ollama.txt` | benchmark transcripts | diff --git a/docker-model-runner/output/01-wire.txt b/docker-model-runner/output/01-wire.txt new file mode 100644 index 0000000..de0db6d --- /dev/null +++ b/docker-model-runner/output/01-wire.txt @@ -0,0 +1,8 @@ +base-url = http://127.0.0.1:/engines/v1 +spring.ai.openai.chat.options.model = configured-model + +call 1, model.call(new Prompt("ping")) + POST /engines/v1/chat/completions | Authorization: Bearer not-needed | model: configured-model + +call 2, same prompt with OpenAiChatOptions.builder().temperature(0.0).build() + POST /engines/v1/chat/completions | Authorization: Bearer not-needed | model: gpt-5-mini diff --git a/docker-model-runner/output/02-engines.txt b/docker-model-runner/output/02-engines.txt new file mode 100644 index 0000000..de1c2b3 --- /dev/null +++ b/docker-model-runner/output/02-engines.txt @@ -0,0 +1,12 @@ +# GET http://localhost:12434/engines/v1/models (Docker Model Runner, docker/model-runner v1.2.9 image) +{"object":"list","data":[{"id":"huggingface.co/qwen/qwen2.5-0.5b-instruct-gguf:Q4_K_M","object":"model","created":1726813224,"owned_by":"docker","dmr":{"context_window":32768,"architecture":"qwen2","parameters":"630.17M","quantization":"MOSTLY_Q4_K_M","size":"462.96MiB"}}]} + + +# GET http://localhost:11434/v1/models (Ollama) +{"object":"list","data":[{"id":"qwen2.5:0.5b","object":"model","created":1793253801,"owned_by":"library"},{"id":"qwen2.5:1.5b","object":"model","created":1793253450,"owned_by":"library"}]} + + +# docker image digests +docker/model-runner:latest sha256:60e6a14d8cc331fab3e51d7b78bba98274b4d28573f4e0b345e475d52d167869 +ollama/ollama:latest sha256:b2f0122c476bc9fe7ce679d8bb4bd555351e25e2554eefb156f61cb0a89cf429 +ollama version is 0.40.13 diff --git a/docker-model-runner/output/03-dmr-extra-fields.txt b/docker-model-runner/output/03-dmr-extra-fields.txt new file mode 100644 index 0000000..eb9c652 --- /dev/null +++ b/docker-model-runner/output/03-dmr-extra-fields.txt @@ -0,0 +1,4 @@ +usage {'completion_tokens': 16, 'prompt_tokens': 31, 'total_tokens': 47, 'prompt_tokens_details': {'cached_tokens': 24}} +timings keys ['cache_n', 'predicted_ms', 'predicted_n', 'predicted_per_second', 'predicted_per_token_ms', 'prompt_ms', 'prompt_n', 'prompt_per_second', 'prompt_per_token_ms'] +ollama usage {'prompt_tokens': 31, 'prompt_tokens_details': {'cached_tokens': 24}, 'completion_tokens': 10, 'total_tokens': 41} +ollama top-level keys ['choices', 'created', 'id', 'model', 'object', 'system_fingerprint', 'usage'] diff --git a/docker-model-runner/output/dmr.txt b/docker-model-runner/output/dmr.txt new file mode 100644 index 0000000..3e4d5cb --- /dev/null +++ b/docker-model-runner/output/dmr.txt @@ -0,0 +1,56 @@ +engine label: dmr +java: 25.0.5+8-LTS +first request (includes model load): ttft 5018 ms, 48 completion tokens + +throughput, 5 prompts x 3 repeats, temperature 0, max_tokens 48 +p rep ttft_ms decode_tps wall_tps prompt compl +1 1 514 19.97 16.70 41 48 +1 2 74 18.03 17.88 41 48 +1 3 91 17.50 17.26 41 48 +2 1 350 16.35 14.85 45 48 +2 2 488 13.79 12.29 45 48 +2 3 82 14.19 14.12 45 48 +3 1 315 17.07 15.60 43 48 +3 2 75 18.15 17.99 43 48 +3 3 91 17.61 17.36 43 48 +4 1 337 16.42 14.97 46 48 +4 2 87 17.94 17.70 46 48 +4 3 101 18.15 17.80 46 48 +5 1 308 17.45 15.96 45 48 +5 2 76 18.30 18.09 45 48 +5 3 89 18.15 17.88 45 48 + +median ttft_ms 91 | median decode_tps 17.61 | median wall_tps 17.26 | n=15 + +tool calling (one tool), 10 runs, temperature 0.7 so runs differ +question: What is the temperature in Oslo in Celsius? Use the tools. +expected: currentTemperature(Oslo), answer contains 17 +run 1 wrong or missing call calls=[celsiusToFahrenheit(12)] answer=The temperature in Oslo is 54 degrees Fahrenheit. +run 2 ok calls=[currentTemperature(Oslo)] answer=The temperature in Oslo is 17 degrees Celsius. +run 3 wrong or missing call calls=[celsiusToFahrenheit(12)] answer=The temperature in Oslo is 54 degrees Celsius. +run 4 wrong or missing call calls=[celsiusToFahrenheit(12)] answer=The temperature in Oslo is 54 degrees Fahrenheit. +run 5 wrong or missing call calls=[celsiusToFahrenheit(12)] answer=The temperature in Oslo is 54 degrees Fahrenheit. +run 6 wrong or missing call calls=[celsiusToFahrenheit(0)] answer=The temperature in Oslo is 32 degrees Celsius. +run 7 wrong or missing call calls=[celsiusToFahrenheit(10)] answer=The temperature in Oslo is 50 degrees Fahrenheit. +run 8 wrong or missing call calls=[celsiusToFahrenheit(0)] answer=The temperature in Oslo is 32 degrees Fahrenheit. +run 9 wrong or missing call calls=[celsiusToFahrenheit(14)] answer=The temperature in Oslo is 57 degrees Fahrenheit. +run 10 wrong or missing call calls=[celsiusToFahrenheit(12)] answer=The temperature in Oslo is 54 degrees Fahrenheit. +tool-call success (one tool): 1/10 (no call 0, wrong/missing call 9, wrong answer 0, errors 0) + +tool calling (two tools in sequence), 10 runs, temperature 0.7 so runs differ +question: What is the temperature in Pune in Fahrenheit? Use the tools. +expected: currentTemperature(Pune), celsiusToFahrenheit(31), answer contains 88 +run 1 wrong or missing call calls=[currentTemperature(Pune)] answer=The temperature in Pune is 31 degrees Fahrenheit. +run 2 wrong or missing call calls=[celsiusToFahrenheit(22)] answer=The temperature in Pune is 72 degrees Fahrenheit. +run 3 wrong or missing call calls=[currentTemperature(Pune)] answer=The temperature in Pune is 31 degrees Fahrenheit. +run 4 wrong or missing call calls=[celsiusToFahrenheit(25)] answer=The temperature in Pune is 77 degrees Fahrenheit. +run 5 wrong or missing call calls=[celsiusToFahrenheit(15)] answer=The temperature in Pune is 59 degrees Fahrenheit. +run 6 wrong or missing call calls=[celsiusToFahrenheit(0)] answer=The temperature in Pune is 32 degrees Fahrenheit. +run 7 wrong or missing call calls=[currentTemperature(Pune)] answer=The temperature in Pune is 31 degrees Fahrenheit. +run 8 wrong or missing call calls=[celsiusToFahrenheit(15)] answer=The temperature in Pune is 59 degrees Fahrenheit. +run 9 wrong or missing call calls=[celsiusToFahrenheit(32)] answer=The temperature in Pune is 90 degrees Fahrenheit. +run 10 wrong or missing call calls=[celsiusToFahrenheit(15)] answer=The temperature in Pune is 59 degrees Fahrenheit. +tool-call success (two tools in sequence): 0/10 (no call 0, wrong/missing call 10, wrong answer 0, errors 0) + +memory (idle container memory before: 3.695MiB) +container memory during run: peak 546 MiB, median 511 MiB, 58 samples diff --git a/docker-model-runner/output/ollama.txt b/docker-model-runner/output/ollama.txt new file mode 100644 index 0000000..dfd9b53 --- /dev/null +++ b/docker-model-runner/output/ollama.txt @@ -0,0 +1,56 @@ +engine label: ollama +java: 25.0.5+8-LTS +first request (includes model load): ttft 4946 ms, 48 completion tokens + +throughput, 5 prompts x 3 repeats, temperature 0, max_tokens 48 +p rep ttft_ms decode_tps wall_tps prompt compl +1 1 247 17.59 16.40 41 48 +1 2 66 18.43 18.32 41 48 +1 3 63 18.02 17.93 41 48 +2 1 289 18.99 17.35 45 48 +2 2 92 18.93 18.58 45 48 +2 3 65 18.56 18.43 45 48 +3 1 322 18.36 16.63 43 48 +3 2 58 20.94 20.79 43 48 +3 3 84 17.65 17.46 43 48 +4 1 287 19.89 18.07 46 48 +4 2 68 19.30 19.15 46 48 +4 3 70 18.57 18.44 46 48 +5 1 282 17.87 16.46 45 48 +5 2 90 18.76 18.44 45 48 +5 3 68 18.74 18.55 45 48 + +median ttft_ms 84 | median decode_tps 18.57 | median wall_tps 18.32 | n=15 + +tool calling (one tool), 10 runs, temperature 0.7 so runs differ +question: What is the temperature in Oslo in Celsius? Use the tools. +expected: currentTemperature(Oslo), answer contains 17 +run 1 ok calls=[currentTemperature(Oslo)] answer=The current temperature in Oslo is 17 degrees Celsius. +run 2 wrong or missing call calls=[celsiusToFahrenheit(0)] answer=The temperature in Oslo, Norway, is 32 degrees Celsius. +run 3 wrong or missing call calls=[celsiusToFahrenheit(20)] answer=The temperature in Oslo is 68 degrees Fahrenheit. +run 4 wrong or missing call calls=[celsiusToFahrenheit(0)] answer=The temperature in Oslo is 32 degrees Celsius. +run 5 wrong or missing call calls=[celsiusToFahrenheit(10)] answer=The temperature in Oslo is currently 50 degrees Fahrenheit. +run 6 wrong or missing call calls=[celsiusToFahrenheit(20)] answer=The temperature in Oslo, at 20 degrees Celsius, is 68 degree +run 7 wrong or missing call calls=[celsiusToFahrenheit(0)] answer=The temperature in Oslo in Celsius is 32 degrees. +run 8 wrong or missing call calls=[celsiusToFahrenheit(4)] answer=The temperature in Oslo (Oslo) is currently 39 degrees Celsi +run 9 wrong or missing call calls=[celsiusToFahrenheit(6)] answer=The temperature in Oslo is 43 degrees Fahrenheit. +run 10 wrong or missing call calls=[celsiusToFahrenheit(0)] answer=The temperature in Oslo is 32 degrees Celsius. +tool-call success (one tool): 1/10 (no call 0, wrong/missing call 9, wrong answer 0, errors 0) + +tool calling (two tools in sequence), 10 runs, temperature 0.7 so runs differ +question: What is the temperature in Pune in Fahrenheit? Use the tools. +expected: currentTemperature(Pune), celsiusToFahrenheit(31), answer contains 88 +run 1 wrong or missing call calls=[celsiusToFahrenheit(22)] answer=The temperature in Pune is 72 degrees Fahrenheit. +run 2 wrong or missing call calls=[celsiusToFahrenheit(32)] answer=The temperature in Pune is currently 90 degrees Fahrenheit. +run 3 wrong or missing call calls=[celsiusToFahrenheit(26)] answer=The temperature in Pune is 79 degrees Fahrenheit. +run 4 wrong or missing call calls=[celsiusToFahrenheit(25)] answer=The temperature in Pune is 77 degrees Fahrenheit. +run 5 wrong or missing call calls=[celsiusToFahrenheit(26)] answer=The temperature in Pune (Pune) is 79 degrees Fahrenheit. It' +run 6 wrong or missing call calls=[celsiusToFahrenheit(23)] answer=The temperature in Pune is 73 degrees Fahrenheit. +run 7 wrong or missing call calls=[celsiusToFahrenheit(23)] answer=The temperature in Pune is 73 degrees Fahrenheit. +run 8 wrong or missing call calls=[celsiusToFahrenheit(15)] answer=The temperature in Pune is 59 degrees Fahrenheit. +run 9 wrong or missing call calls=[celsiusToFahrenheit(0)] answer=The temperature in Pune is 32 degrees Fahrenheit. +run 10 wrong or missing call calls=[celsiusToFahrenheit(22)] answer=The temperature in Pune is 72 degrees Fahrenheit. +tool-call success (two tools in sequence): 0/10 (no call 0, wrong/missing call 10, wrong answer 0, errors 0) + +memory (idle container memory before: 12.15MiB) +container memory during run: peak 510 MiB, median 509 MiB, 55 samples diff --git a/docker-model-runner/pom.xml b/docker-model-runner/pom.xml new file mode 100644 index 0000000..5179ac6 --- /dev/null +++ b/docker-model-runner/pom.xml @@ -0,0 +1,64 @@ + + + 4.0.0 + + + org.springframework.boot + spring-boot-starter-parent + 4.1.2 + + + + com.ankurm + docker-model-runner + 1.0.0 + docker-model-runner + Spring AI 2.0 against Docker Model Runner and Ollama through the same OpenAI-compatible client, with a benchmark harness for tokens per second, memory and tool-calling reliability. + + + 25 + 2.0.2 + + + + + + org.springframework.ai + spring-ai-bom + ${spring-ai.version} + pom + import + + + + + + + org.springframework.ai + spring-ai-starter-model-openai + + + org.springframework.boot + spring-boot-starter-test + test + + + + + + + org.springframework.boot + spring-boot-maven-plugin + + + org.apache.maven.plugins + maven-surefire-plugin + + -Duser.timezone=UTC -Dstdout.encoding=UTF-8 -Dfile.encoding=UTF-8 + + + + + diff --git a/docker-model-runner/scripts/bench.sh b/docker-model-runner/scripts/bench.sh new file mode 100755 index 0000000..9f91017 --- /dev/null +++ b/docker-model-runner/scripts/bench.sh @@ -0,0 +1,34 @@ +#!/usr/bin/env bash +# bench.sh