diff --git a/README.md b/README.md index 9a4cc0c..37dec4d 100644 --- a/README.md +++ b/README.md @@ -11,5 +11,6 @@ Runnable companion code for the Spring AI articles on [ankurm.com](https://ankur | [`mcp-secure/`](mcp-secure) | The mcp-server article's order-lookup tools behind a real OAuth2 resource server: JWT validation, one scope per tool via `@PreAuthorize`, unauthenticated tool discovery rejected outright, and every call audit-logged through MDC -- denials included. Spring Boot 4.1.1, Spring AI 2.0.1, Spring Security 7.1.1, Java 25. | [Securing an MCP Server with Spring Security 7](https://ankurm.com/spring-ai-2-0-mcp-server-security/) | | [`tool-calling/`](tool-calling) | `@Tool` methods, `ToolCallingAdvisor` (the advisor-layer replacement for Spring AI 1.x's per-model tool loop), `returnDirect`, `ToolContext`, and `ToolSearchToolCallingAdvisor` for progressive disclosure across a 230-tool synthetic library -- every test driven by a hand-written `ScriptedChatModel`, no live model anywhere. Spring Boot 4.1.1, Spring AI 2.0.1, Java 25. | [Tool Calling in Spring AI 2.0](https://ankurm.com/spring-ai-2-0-tool-calling/) | | [`structured-output/`](structured-output) | `ChatClient.entity()` mapping LLM responses to Java records, lists and maps; `StructuredOutputValidationAdvisor` retrying non-conforming JSON with a real enum-constrained schema, including a captured run that exhausts every retry without throwing. Spring Boot 4.1.1, Spring AI 2.0.1, Java 25. | [Structured Output in Spring AI 2.0](https://ankurm.com/spring-ai-2-0-structured-output/) | +| [`ollama-local/`](ollama-local) | Chat and embeddings against a real local `qwen2.5:0.5b`/`all-minilm`, no API key, driven by a Testcontainers-managed Ollama container started from a baked image; a confirmed model unload via `keep_alive: 0` and `/api/ps`, not a scripted model anywhere. Spring Boot 4.1.1, Spring AI 2.0.1, Testcontainers 2.0.5, Java 25. | [Run LLMs Locally with Spring AI and Ollama](https://ankurm.com/spring-ai-2-0-ollama-local/) | Upgrading from Spring AI 1.x: [migration guide](https://ankurm.com/spring-ai-1-to-2-migration-guide/). diff --git a/ollama-local/.gitignore b/ollama-local/.gitignore new file mode 100644 index 0000000..2f7896d --- /dev/null +++ b/ollama-local/.gitignore @@ -0,0 +1 @@ +target/ diff --git a/ollama-local/README.md b/ollama-local/README.md new file mode 100644 index 0000000..f455266 --- /dev/null +++ b/ollama-local/README.md @@ -0,0 +1,46 @@ +# ollama-local + +Companion code for [Run LLMs Locally with Spring AI and Ollama](https://ankurm.com/spring-ai-2-0-ollama-local/), part of the [Spring AI series](../README.md) on ankurm.com. + +`spring-ai-starter-model-ollama` autoconfigures a `ChatModel` and an `EmbeddingModel` from `spring.ai.ollama.*` properties alone -- no API key anywhere in this module. Every test in [`LocalChatAndEmbeddingTest.java`](src/test/java/com/ankurm/ollamalocal/LocalChatAndEmbeddingTest.java) drives a real, local Ollama server started by Testcontainers, answering with a real small model (`qwen2.5:0.5b` for chat, `all-minilm` for embeddings) -- unlike every other module in this series, nothing here is scripted. + +## Versions + +| Component | Version | +|---|---| +| Spring Boot | 4.1.1 | +| Spring AI | 2.0.1 | +| Testcontainers | 2.0.5 | +| Java | 25 (LTS) | + +`org.testcontainers:ollama` was renamed to **`org.testcontainers:testcontainers-ollama`** in the Testcontainers 2.x line, and `org.testcontainers:junit-jupiter` to **`org.testcontainers:testcontainers-junit-jupiter`** -- both confirmed by reading the real `testcontainers-bom-2.0.5.pom` that Spring Boot 4.1.1 imports. If you're copying an older Testcontainers-Ollama tutorial, the old artifact IDs still exist on Maven Central but are stuck on the 1.x line and are not what this BOM resolves. + +## Quickstart + +```bash +./scripts/bake-image.sh # once: pulls qwen2.5:0.5b + all-minilm into a local image +./scripts/run-all.sh # every time: runs the suite against that baked image +``` + +`bake-image.sh` is the real Testcontainers-recommended pattern for CI: pull the models into a container once, `docker commit` the result, and every subsequent test run starts a container that already has them on disk -- no registry pull, no network dependency, no per-run latency for the pull itself. + +## What's here + +| File | What it shows | +|---|---| +| [`OllamaLocalApplication.java`](src/main/java/com/ankurm/ollamalocal/OllamaLocalApplication.java) | The whole application: no manual `OllamaApi`/`OllamaChatModel` wiring, just the starter's autoconfiguration | +| [`config/ChatClientConfig.java`](src/main/java/com/ankurm/ollamalocal/config/ChatClientConfig.java) | Wraps the autoconfigured `ChatModel` in a `ChatClient`, same as every other module in this series | +| [`application.yml`](src/main/resources/application.yml) | The five `spring.ai.ollama.*` properties this module uses, with `pull-model-strategy: never` so the app fails fast instead of silently pulling gigabytes at startup | +| [`LocalChatAndEmbeddingTest.java`](src/test/java/com/ankurm/ollamalocal/LocalChatAndEmbeddingTest.java) | A Testcontainers-managed `OllamaContainer`, a confirmed model unload via `keep_alive: 0` + `/api/ps`, real chat calls, and a deterministic embedding-dimension assertion | + +## Output files + +| File | Captured from | +|---|---| +| `output/01-chat-reload-after-unload.txt` | `reloadAfterAConfirmedUnload` -- unloads the model, confirms via `/api/ps`, then reloads it | +| `output/02-chat-back-to-back-call.txt` | `backToBackCallReusesTheAlreadyLoadedModel` | +| `output/03-embedding-dimensions.txt` | `embeddingsAreDeterministicallySized` | + +## Requirements + +JDK 25, Maven, Docker. Run `scripts/bake-image.sh` before the test suite -- without a baked image present, `scripts/run-all.sh` refuses to start rather than silently pulling ~450MB of models over the network on every test run. diff --git a/ollama-local/output/01-chat-reload-after-unload.txt b/ollama-local/output/01-chat-reload-after-unload.txt new file mode 100644 index 0000000..17eadc2 --- /dev/null +++ b/ollama-local/output/01-chat-reload-after-unload.txt @@ -0,0 +1,10 @@ +/api/ps immediately after the unload call: {"models":[]} + +prompt: "Reply with a single short sentence: why do developers like small local models?" +response: Developers often prefer small local models because they are more efficient, faster, and easier to deploy and train. + +total-duration: 901ms +load-duration: 1ms +prompt-eval-count: 44, prompt-eval-duration: 39ms +eval-count: 23, eval-duration: 856ms +26.87 tokens/sec (eval-count / eval-duration) \ No newline at end of file diff --git a/ollama-local/output/02-chat-back-to-back-call.txt b/ollama-local/output/02-chat-back-to-back-call.txt new file mode 100644 index 0000000..8970e73 --- /dev/null +++ b/ollama-local/output/02-chat-back-to-back-call.txt @@ -0,0 +1,8 @@ +prompt: "Reply with a single short sentence: what is Testcontainers for?" +response: Testcontainers is a popular containerization platform that makes it easy to create and manage application containers for testing, development, and production environments. + +total-duration: 722ms +load-duration: 2ms +prompt-eval-count: 42, prompt-eval-duration: 61ms +eval-count: 17, eval-duration: 653ms +26.02 tokens/sec (eval-count / eval-duration) \ No newline at end of file diff --git a/ollama-local/output/03-embedding-dimensions.txt b/ollama-local/output/03-embedding-dimensions.txt new file mode 100644 index 0000000..bb809e4 --- /dev/null +++ b/ollama-local/output/03-embedding-dimensions.txt @@ -0,0 +1,4 @@ +model: all-minilm +dimensions(): 384 +vector.length: 384 +first 8 values: [-0.02818, -0.00275, -0.02190, -0.03044, 0.02368, -0.08213, -0.10182, -0.05197, ...] \ No newline at end of file diff --git a/ollama-local/pom.xml b/ollama-local/pom.xml new file mode 100644 index 0000000..eed9df6 --- /dev/null +++ b/ollama-local/pom.xml @@ -0,0 +1,85 @@ + + + 4.0.0 + + + org.springframework.boot + spring-boot-starter-parent + 4.1.1 + + + + com.ankurm + ollama-local + 1.0.0 + ollama-local + Running LLMs locally with Spring AI 2.0 and Ollama: chat and embeddings against a real local model, no API key, with a Testcontainers Ollama module baked for fast CI. + + + 25 + 2.0.1 + + + + + + org.springframework.ai + spring-ai-bom + ${spring-ai.version} + pom + import + + + + + + + org.springframework.ai + spring-ai-starter-model-ollama + + + + org.springframework.boot + spring-boot-starter-test + test + + + org.springframework.boot + spring-boot-testcontainers + test + + + + org.testcontainers + testcontainers-ollama + test + + + + org.testcontainers + testcontainers-junit-jupiter + test + + + + + + + org.springframework.boot + spring-boot-maven-plugin + + + org.apache.maven.plugins + maven-surefire-plugin + + -Duser.timezone=UTC -Dstdout.encoding=UTF-8 -Dfile.encoding=UTF-8 + + + + + diff --git a/ollama-local/scripts/bake-image.sh b/ollama-local/scripts/bake-image.sh new file mode 100755 index 0000000..6f7f42f --- /dev/null +++ b/ollama-local/scripts/bake-image.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# Pulls ollama/ollama:latest, starts it once, pulls qwen2.5:0.5b (chat, ~400MB) and all-minilm +# (embeddings, ~45MB) inside the running container, then `docker commit`s the result to a local +# image tag. This is the real Testcontainers-recommended pattern for CI: bake the models into +# an image once, and every subsequent test run starts a container that already has them on +# disk -- no registry pull, no flaky network dependency, no per-run latency for the pull itself. +# +# Run this once before scripts/run-all.sh, or whenever you want to refresh the baked models. +set -euo pipefail + +IMAGE_TAG="${1:-ollama-baked-qwen05b-minilm:local}" +CONTAINER_NAME="ollama-bake-$$" + +echo "Pulling ollama/ollama:latest..." +docker pull ollama/ollama:latest + +echo "Starting a container to pull models into..." +docker run -d --name "$CONTAINER_NAME" ollama/ollama:latest +trap 'docker rm -f "$CONTAINER_NAME" >/dev/null 2>&1 || true' EXIT + +echo "Waiting for the Ollama server to accept connections..." +until docker exec "$CONTAINER_NAME" ollama list >/dev/null 2>&1; do + sleep 1 +done + +echo "Pulling qwen2.5:0.5b (chat)..." +docker exec "$CONTAINER_NAME" ollama pull qwen2.5:0.5b + +echo "Pulling all-minilm (embeddings)..." +docker exec "$CONTAINER_NAME" ollama pull all-minilm + +echo "Committing to ${IMAGE_TAG}..." +docker commit "$CONTAINER_NAME" "$IMAGE_TAG" + +echo "Done. ${IMAGE_TAG} now has both models baked in -- scripts/run-all.sh will not touch the network." diff --git a/ollama-local/scripts/run-all.sh b/ollama-local/scripts/run-all.sh new file mode 100755 index 0000000..d1585d0 --- /dev/null +++ b/ollama-local/scripts/run-all.sh @@ -0,0 +1,15 @@ +#!/usr/bin/env bash +# Regenerates every file under output/ -- the test suite writes each one itself via the +# Transcript helper, overwriting it in place. Unlike every other module in this series, this +# one needs Docker and the baked image from scripts/bake-image.sh: these tests start a real +# container running a real small model, not a scripted one. +set -euo pipefail +cd "$(dirname "$0")/.." + +if ! docker image inspect ollama-baked-qwen05b-minilm:local >/dev/null 2>&1; then + echo "Image ollama-baked-qwen05b-minilm:local not found -- run scripts/bake-image.sh first." >&2 + exit 1 +fi + +rm -rf target +mvn -q -o test diff --git a/ollama-local/src/main/java/com/ankurm/ollamalocal/OllamaLocalApplication.java b/ollama-local/src/main/java/com/ankurm/ollamalocal/OllamaLocalApplication.java new file mode 100644 index 0000000..76911ac --- /dev/null +++ b/ollama-local/src/main/java/com/ankurm/ollamalocal/OllamaLocalApplication.java @@ -0,0 +1,19 @@ +package com.ankurm.ollamalocal; + +import org.springframework.boot.SpringApplication; +import org.springframework.boot.autoconfigure.SpringBootApplication; + +/** + * No {@code api-key} property anywhere in this module. {@code spring-ai-starter-model-ollama} + * autoconfigures a {@link org.springframework.ai.ollama.OllamaChatModel} and a + * {@link org.springframework.ai.ollama.OllamaEmbeddingModel} from {@code spring.ai.ollama.*} + * properties alone, talking to a local Ollama server over plain HTTP. + */ +@SpringBootApplication +public class OllamaLocalApplication { + + public static void main(String[] args) { + SpringApplication.run(OllamaLocalApplication.class, args); + } + +} diff --git a/ollama-local/src/main/java/com/ankurm/ollamalocal/config/ChatClientConfig.java b/ollama-local/src/main/java/com/ankurm/ollamalocal/config/ChatClientConfig.java new file mode 100644 index 0000000..e10b58b --- /dev/null +++ b/ollama-local/src/main/java/com/ankurm/ollamalocal/config/ChatClientConfig.java @@ -0,0 +1,22 @@ +package com.ankurm.ollamalocal.config; + +import org.springframework.ai.chat.client.ChatClient; +import org.springframework.ai.chat.model.ChatModel; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; + +/** + * The {@link ChatModel} bean itself comes from Spring AI's Ollama autoconfiguration -- + * nothing here constructs an {@code OllamaApi} or an {@code OllamaChatModel} by hand. + * This class only wraps whatever {@code ChatModel} autoconfiguration produced in a + * {@link ChatClient}, exactly like every other module in this series. + */ +@Configuration +public class ChatClientConfig { + + @Bean + ChatClient chatClient(ChatModel chatModel) { + return ChatClient.builder(chatModel).build(); + } + +} diff --git a/ollama-local/src/main/resources/application.yml b/ollama-local/src/main/resources/application.yml new file mode 100644 index 0000000..7b24345 --- /dev/null +++ b/ollama-local/src/main/resources/application.yml @@ -0,0 +1,18 @@ +# No API key anywhere in this file. spring.ai.ollama.base-url points at a local Ollama +# server (default: http://localhost:11434); tests override it with the real mapped +# endpoint of a Testcontainers-managed container via @DynamicPropertySource. +spring: + ai: + ollama: + base-url: http://localhost:11434 + chat: + options: + model: qwen2.5:0.5b + embedding: + options: + model: all-minilm + init: + # never: fail fast if the model isn't already present, instead of silently pulling + # gigabytes over the network at context-startup time. when_missing/always exist for + # environments that want auto-pull; see the module README. + pull-model-strategy: never diff --git a/ollama-local/src/test/java/com/ankurm/ollamalocal/LocalChatAndEmbeddingTest.java b/ollama-local/src/test/java/com/ankurm/ollamalocal/LocalChatAndEmbeddingTest.java new file mode 100644 index 0000000..c88c439 --- /dev/null +++ b/ollama-local/src/test/java/com/ankurm/ollamalocal/LocalChatAndEmbeddingTest.java @@ -0,0 +1,174 @@ +package com.ankurm.ollamalocal; + +import java.net.URI; +import java.net.http.HttpClient; +import java.net.http.HttpRequest; +import java.net.http.HttpResponse; +import java.time.Duration; + +import com.ankurm.ollamalocal.support.Transcript; +import org.junit.jupiter.api.MethodOrderer; +import org.junit.jupiter.api.Order; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.TestMethodOrder; +import org.testcontainers.junit.jupiter.Container; +import org.testcontainers.junit.jupiter.Testcontainers; +import org.testcontainers.ollama.OllamaContainer; +import org.testcontainers.utility.DockerImageName; + +import org.springframework.ai.chat.client.ChatClient; +import org.springframework.ai.chat.metadata.ChatResponseMetadata; +import org.springframework.ai.chat.model.ChatModel; +import org.springframework.ai.chat.prompt.Prompt; +import org.springframework.ai.embedding.EmbeddingModel; +import org.springframework.ai.ollama.api.OllamaChatOptions; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.boot.test.context.SpringBootTest; +import org.springframework.test.context.DynamicPropertyRegistry; +import org.springframework.test.context.DynamicPropertySource; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * One real, local Ollama server for the whole class, started from a baked image that already + * has {@code qwen2.5:0.5b} (chat) and {@code all-minilm} (embeddings) pulled -- see + * {@code scripts/bake-image.sh}. No network call happens during these tests: the container + * starts from a local image and both models are already on disk inside it. + * + *

Every other module in this series drives {@code ChatClient} against a hand-written + * {@code ScriptedChatModel} so the tests are deterministic and need no live model. This module + * is the deliberate exception: the whole point of "run it locally" is a real model answering a + * real prompt, so these tests assert only what is genuinely deterministic about a live small + * model (a non-blank response, a confirmed-empty model registry right after an explicit unload) + * and leave exact wording, and exact timings, alone. + */ +@Testcontainers +@SpringBootTest +@TestMethodOrder(MethodOrderer.OrderAnnotation.class) +class LocalChatAndEmbeddingTest { + + @Container + static final OllamaContainer OLLAMA = new OllamaContainer( + DockerImageName.parse("ollama-baked-qwen05b-minilm:local").asCompatibleSubstituteFor("ollama/ollama")); + + @DynamicPropertySource + static void ollamaProperties(DynamicPropertyRegistry registry) { + registry.add("spring.ai.ollama.base-url", OLLAMA::getEndpoint); + } + + private static final HttpClient HTTP = HttpClient.newHttpClient(); + + @Autowired + private ChatClient chatClient; + + @Autowired + private ChatModel chatModel; + + @Autowired + private EmbeddingModel embeddingModel; + + @Test + @Order(1) + void reloadAfterAConfirmedUnload() throws Exception { + // keepAlive("0") tells Ollama to unload the model from memory as soon as this call + // finishes -- the same knob you'd reach for in production to free RAM/VRAM between + // bursts of traffic. It's set through ChatModel.call() with an explicit Prompt rather + // than through ChatClient.prompt().options(...): with this module's versions, an option + // set that way never reaches the request Ollama receives, which the going-deeper note + // below covers. + this.chatModel + .call(new Prompt("OK", OllamaChatOptions.builder().model("qwen2.5:0.5b").keepAlive("0").build())); + + // Don't take the unload on faith -- ask Ollama's own /api/ps, which lists every + // currently loaded model, and confirm the registry is really empty before measuring + // what a reload costs. + String modelsAfterUnload = getModelRegistry(); + assertThat(modelsAfterUnload).contains("\"models\":[]"); + + ChatClient.CallResponseSpec response = this.chatClient.prompt() + .user("Reply with a single short sentence: why do developers like small local models?") + .call(); + + String content = response.content(); + ChatResponseMetadata metadata = response.chatResponse().getMetadata(); + + assertThat(content).isNotBlank(); + // Loading is real work -- reading weights off disk and initializing the runtime -- so + // even a fast reload can't be negative or missing. + Duration loadDuration = metadata.get("load-duration"); + assertThat(loadDuration).isNotNull(); + assertThat(loadDuration.isNegative()).isFalse(); + + Transcript.write("01-chat-reload-after-unload", + "/api/ps immediately after the unload call: " + modelsAfterUnload + "\n\n" + + "prompt: \"Reply with a single short sentence: why do developers like small local models?\"\n" + + "response: " + content + "\n\n" + formatOllamaMetadata(metadata)); + } + + @Test + @Order(2) + void backToBackCallReusesTheAlreadyLoadedModel() { + ChatClient.CallResponseSpec response = this.chatClient.prompt() + .user("Reply with a single short sentence: what is Testcontainers for?") + .call(); + + String content = response.content(); + ChatResponseMetadata metadata = response.chatResponse().getMetadata(); + + assertThat(content).isNotBlank(); + + Transcript.write("02-chat-back-to-back-call", + "prompt: \"Reply with a single short sentence: what is Testcontainers for?\"\n" + + "response: " + content + "\n\n" + formatOllamaMetadata(metadata)); + } + + @Test + @Order(3) + void embeddingsAreDeterministicallySized() { + float[] vector = this.embeddingModel.embed("Spring AI can run entirely against a local Ollama server."); + + assertThat(vector).hasSize(384); + assertThat(this.embeddingModel.dimensions()).isEqualTo(384); + + StringBuilder prefix = new StringBuilder(); + for (int i = 0; i < 8; i++) { + if (i > 0) { + prefix.append(", "); + } + prefix.append(String.format("%.5f", vector[i])); + } + + Transcript.write("03-embedding-dimensions", + "model: all-minilm\n" + "dimensions(): " + this.embeddingModel.dimensions() + "\n" + + "vector.length: " + vector.length + "\n" + "first 8 values: [" + prefix + ", ...]"); + } + + private static String getModelRegistry() throws Exception { + HttpRequest request = HttpRequest.newBuilder(URI.create(OLLAMA.getEndpoint() + "/api/ps")).GET().build(); + return HTTP.send(request, HttpResponse.BodyHandlers.ofString()).body(); + } + + private static String formatOllamaMetadata(ChatResponseMetadata metadata) { + Duration total = metadata.get("total-duration"); + Duration load = metadata.get("load-duration"); + Duration promptEval = metadata.get("prompt-eval-duration"); + Duration eval = metadata.get("eval-duration"); + Integer promptEvalCount = metadata.get("prompt-eval-count"); + Integer evalCount = metadata.get("eval-count"); + + StringBuilder sb = new StringBuilder(); + sb.append("total-duration: ").append(total.toMillis()).append("ms\n"); + sb.append("load-duration: ").append(load.toMillis()).append("ms\n"); + sb.append("prompt-eval-count: ").append(promptEvalCount); + sb.append(", prompt-eval-duration: ").append(promptEval.toMillis()).append("ms\n"); + sb.append("eval-count: ").append(evalCount); + sb.append(", eval-duration: ").append(eval.toMillis()).append("ms"); + + if (evalCount != null && eval != null && eval.toNanos() > 0) { + double tokensPerSecond = evalCount / (eval.toNanos() / 1_000_000_000.0); + sb.append(String.format("%n%.2f tokens/sec (eval-count / eval-duration)", tokensPerSecond)); + } + return sb.toString(); + } + +} diff --git a/ollama-local/src/test/java/com/ankurm/ollamalocal/support/Transcript.java b/ollama-local/src/test/java/com/ankurm/ollamalocal/support/Transcript.java new file mode 100644 index 0000000..8b42d62 --- /dev/null +++ b/ollama-local/src/test/java/com/ankurm/ollamalocal/support/Transcript.java @@ -0,0 +1,36 @@ +package com.ankurm.ollamalocal.support; + +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; + +/** + * Writes a test's captured, real output to {@code output/.txt} and echoes it to stdout. + * Every console block quoted in the companion article comes from a file this class wrote -- + * nothing in the article is retyped or tidied up by hand. Unlike every other module in this + * series, the numbers this class captures here come from a real local model, not a scripted + * one: cold/warm load times and token rates will vary run to run and machine to machine, which + * is exactly the point -- the article says so and gives readers the script to get their own. + */ +public final class Transcript { + + private Transcript() { + } + + public static void write(String name, String content) { + try { + Path dir = Path.of("output"); + Files.createDirectories(dir); + Path file = dir.resolve(name + ".txt"); + Files.writeString(file, content, StandardCharsets.UTF_8); + System.out.println("--- " + name + " ---"); + System.out.println(content); + } + catch (IOException ex) { + throw new UncheckedIOException(ex); + } + } + +}