Add ollama-local module: chat and embeddings against a real local Ollama server
- spring-ai-starter-model-ollama autoconfigures ChatModel/EmbeddingModel from spring.ai.ollama.* properties alone; no API key anywhere in this module. - org.testcontainers:ollama and org.testcontainers:junit-jupiter were both renamed in the Testcontainers 2.x line -- to org.testcontainers:testcontainers-ollama and org.testcontainers:testcontainers-junit-jupiter respectively -- confirmed by reading the real testcontainers-bom-2.0.5.pom that Spring Boot 4.1.1 imports (spring-boot-dependencies -> testcontainers.version=2.0.5). The pre-rename artifact IDs still exist on Maven Central but are stuck on the 1.x line. - Unlike every other module in this series, tests drive a real local model (qwen2.5:0.5b chat, all-minilm embeddings) via a Testcontainers-managed OllamaContainer started from a baked image (scripts/bake-image.sh), not a ScriptedChatModel -- the whole point of this post is a real model answering a real prompt. - LocalChatAndEmbeddingTest forces a genuine cold state with Ollama's keep_alive: 0 option (set via ChatModel.call(Prompt) -- ChatClient.options() does not carry a keepAlive override through to the request in this version) and confirms the unload actually happened via /api/ps before measuring a reload, rather than trusting whichever call happens to run first. - On this quiet sandbox host, even a confirmed-cold reload of the 500MB model came back in single-digit milliseconds once the underlying image layers were cached -- eval (generation) time dominates total latency here, not loading. Captured, not asserted as universal: readers get scripts/bake-image.sh to get their own numbers. - Embedding dimension (384, all-minilm) asserted deterministically. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01FtpJvZfg4nvLvtzgJTDWpB
This commit is contained in:
Claude
@@ -0,0 +1 @@
|
||||
target/
|
||||
@@ -0,0 +1,46 @@
|
||||
# ollama-local
|
||||
|
||||
Companion code for [Run LLMs Locally with Spring AI and Ollama](https://ankurm.com/spring-ai-2-0-ollama-local/), part of the [Spring AI series](../README.md) on ankurm.com.
|
||||
|
||||
`spring-ai-starter-model-ollama` autoconfigures a `ChatModel` and an `EmbeddingModel` from `spring.ai.ollama.*` properties alone -- no API key anywhere in this module. Every test in [`LocalChatAndEmbeddingTest.java`](src/test/java/com/ankurm/ollamalocal/LocalChatAndEmbeddingTest.java) drives a real, local Ollama server started by Testcontainers, answering with a real small model (`qwen2.5:0.5b` for chat, `all-minilm` for embeddings) -- unlike every other module in this series, nothing here is scripted.
|
||||
|
||||
## Versions
|
||||
|
||||
| Component | Version |
|
||||
|---|---|
|
||||
| Spring Boot | 4.1.1 |
|
||||
| Spring AI | 2.0.1 |
|
||||
| Testcontainers | 2.0.5 |
|
||||
| Java | 25 (LTS) |
|
||||
|
||||
`org.testcontainers:ollama` was renamed to **`org.testcontainers:testcontainers-ollama`** in the Testcontainers 2.x line, and `org.testcontainers:junit-jupiter` to **`org.testcontainers:testcontainers-junit-jupiter`** -- both confirmed by reading the real `testcontainers-bom-2.0.5.pom` that Spring Boot 4.1.1 imports. If you're copying an older Testcontainers-Ollama tutorial, the old artifact IDs still exist on Maven Central but are stuck on the 1.x line and are not what this BOM resolves.
|
||||
|
||||
## Quickstart
|
||||
|
||||
```bash
|
||||
./scripts/bake-image.sh # once: pulls qwen2.5:0.5b + all-minilm into a local image
|
||||
./scripts/run-all.sh # every time: runs the suite against that baked image
|
||||
```
|
||||
|
||||
`bake-image.sh` is the real Testcontainers-recommended pattern for CI: pull the models into a container once, `docker commit` the result, and every subsequent test run starts a container that already has them on disk -- no registry pull, no network dependency, no per-run latency for the pull itself.
|
||||
|
||||
## What's here
|
||||
|
||||
| File | What it shows |
|
||||
|---|---|
|
||||
| [`OllamaLocalApplication.java`](src/main/java/com/ankurm/ollamalocal/OllamaLocalApplication.java) | The whole application: no manual `OllamaApi`/`OllamaChatModel` wiring, just the starter's autoconfiguration |
|
||||
| [`config/ChatClientConfig.java`](src/main/java/com/ankurm/ollamalocal/config/ChatClientConfig.java) | Wraps the autoconfigured `ChatModel` in a `ChatClient`, same as every other module in this series |
|
||||
| [`application.yml`](src/main/resources/application.yml) | The five `spring.ai.ollama.*` properties this module uses, with `pull-model-strategy: never` so the app fails fast instead of silently pulling gigabytes at startup |
|
||||
| [`LocalChatAndEmbeddingTest.java`](src/test/java/com/ankurm/ollamalocal/LocalChatAndEmbeddingTest.java) | A Testcontainers-managed `OllamaContainer`, a confirmed model unload via `keep_alive: 0` + `/api/ps`, real chat calls, and a deterministic embedding-dimension assertion |
|
||||
|
||||
## Output files
|
||||
|
||||
| File | Captured from |
|
||||
|---|---|
|
||||
| `output/01-chat-reload-after-unload.txt` | `reloadAfterAConfirmedUnload` -- unloads the model, confirms via `/api/ps`, then reloads it |
|
||||
| `output/02-chat-back-to-back-call.txt` | `backToBackCallReusesTheAlreadyLoadedModel` |
|
||||
| `output/03-embedding-dimensions.txt` | `embeddingsAreDeterministicallySized` |
|
||||
|
||||
## Requirements
|
||||
|
||||
JDK 25, Maven, Docker. Run `scripts/bake-image.sh` before the test suite -- without a baked image present, `scripts/run-all.sh` refuses to start rather than silently pulling ~450MB of models over the network on every test run.
|
||||
@@ -0,0 +1,10 @@
|
||||
/api/ps immediately after the unload call: {"models":[]}
|
||||
|
||||
prompt: "Reply with a single short sentence: why do developers like small local models?"
|
||||
response: Developers often prefer small local models because they are more efficient, faster, and easier to deploy and train.
|
||||
|
||||
total-duration: 901ms
|
||||
load-duration: 1ms
|
||||
prompt-eval-count: 44, prompt-eval-duration: 39ms
|
||||
eval-count: 23, eval-duration: 856ms
|
||||
26.87 tokens/sec (eval-count / eval-duration)
|
||||
@@ -0,0 +1,8 @@
|
||||
prompt: "Reply with a single short sentence: what is Testcontainers for?"
|
||||
response: Testcontainers is a popular containerization platform that makes it easy to create and manage application containers for testing, development, and production environments.
|
||||
|
||||
total-duration: 722ms
|
||||
load-duration: 2ms
|
||||
prompt-eval-count: 42, prompt-eval-duration: 61ms
|
||||
eval-count: 17, eval-duration: 653ms
|
||||
26.02 tokens/sec (eval-count / eval-duration)
|
||||
@@ -0,0 +1,4 @@
|
||||
model: all-minilm
|
||||
dimensions(): 384
|
||||
vector.length: 384
|
||||
first 8 values: [-0.02818, -0.00275, -0.02190, -0.03044, 0.02368, -0.08213, -0.10182, -0.05197, ...]
|
||||
@@ -0,0 +1,85 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
|
||||
<parent>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-parent</artifactId>
|
||||
<version>4.1.1</version>
|
||||
<relativePath/>
|
||||
</parent>
|
||||
|
||||
<groupId>com.ankurm</groupId>
|
||||
<artifactId>ollama-local</artifactId>
|
||||
<version>1.0.0</version>
|
||||
<name>ollama-local</name>
|
||||
<description>Running LLMs locally with Spring AI 2.0 and Ollama: chat and embeddings against a real local model, no API key, with a Testcontainers Ollama module baked for fast CI.</description>
|
||||
|
||||
<properties>
|
||||
<java.version>25</java.version>
|
||||
<spring-ai.version>2.0.1</spring-ai.version>
|
||||
</properties>
|
||||
|
||||
<dependencyManagement>
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-bom</artifactId>
|
||||
<version>${spring-ai.version}</version>
|
||||
<type>pom</type>
|
||||
<scope>import</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
</dependencyManagement>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-starter-model-ollama</artifactId>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-test</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-testcontainers</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<!-- Renamed from org.testcontainers:ollama in Testcontainers 1.x to org.testcontainers:testcontainers-ollama
|
||||
in the 2.x line Spring Boot 4.1.1 manages (testcontainers-bom 2.0.5, imported transitively via
|
||||
spring-boot-dependencies); the old artifactId still exists on Maven Central but is stuck at 1.21.4
|
||||
and is NOT what this BOM resolves. -->
|
||||
<dependency>
|
||||
<groupId>org.testcontainers</groupId>
|
||||
<artifactId>testcontainers-ollama</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<!-- Also renamed in the 2.x line: org.testcontainers:junit-jupiter -> org.testcontainers:testcontainers-junit-jupiter. -->
|
||||
<dependency>
|
||||
<groupId>org.testcontainers</groupId>
|
||||
<artifactId>testcontainers-junit-jupiter</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-maven-plugin</artifactId>
|
||||
</plugin>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-surefire-plugin</artifactId>
|
||||
<configuration>
|
||||
<argLine>-Duser.timezone=UTC -Dstdout.encoding=UTF-8 -Dfile.encoding=UTF-8</argLine>
|
||||
</configuration>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
</project>
|
||||
Executable
+35
@@ -0,0 +1,35 @@
|
||||
#!/usr/bin/env bash
|
||||
# Pulls ollama/ollama:latest, starts it once, pulls qwen2.5:0.5b (chat, ~400MB) and all-minilm
|
||||
# (embeddings, ~45MB) inside the running container, then `docker commit`s the result to a local
|
||||
# image tag. This is the real Testcontainers-recommended pattern for CI: bake the models into
|
||||
# an image once, and every subsequent test run starts a container that already has them on
|
||||
# disk -- no registry pull, no flaky network dependency, no per-run latency for the pull itself.
|
||||
#
|
||||
# Run this once before scripts/run-all.sh, or whenever you want to refresh the baked models.
|
||||
set -euo pipefail
|
||||
|
||||
IMAGE_TAG="${1:-ollama-baked-qwen05b-minilm:local}"
|
||||
CONTAINER_NAME="ollama-bake-$$"
|
||||
|
||||
echo "Pulling ollama/ollama:latest..."
|
||||
docker pull ollama/ollama:latest
|
||||
|
||||
echo "Starting a container to pull models into..."
|
||||
docker run -d --name "$CONTAINER_NAME" ollama/ollama:latest
|
||||
trap 'docker rm -f "$CONTAINER_NAME" >/dev/null 2>&1 || true' EXIT
|
||||
|
||||
echo "Waiting for the Ollama server to accept connections..."
|
||||
until docker exec "$CONTAINER_NAME" ollama list >/dev/null 2>&1; do
|
||||
sleep 1
|
||||
done
|
||||
|
||||
echo "Pulling qwen2.5:0.5b (chat)..."
|
||||
docker exec "$CONTAINER_NAME" ollama pull qwen2.5:0.5b
|
||||
|
||||
echo "Pulling all-minilm (embeddings)..."
|
||||
docker exec "$CONTAINER_NAME" ollama pull all-minilm
|
||||
|
||||
echo "Committing to ${IMAGE_TAG}..."
|
||||
docker commit "$CONTAINER_NAME" "$IMAGE_TAG"
|
||||
|
||||
echo "Done. ${IMAGE_TAG} now has both models baked in -- scripts/run-all.sh will not touch the network."
|
||||
Executable
+15
@@ -0,0 +1,15 @@
|
||||
#!/usr/bin/env bash
|
||||
# Regenerates every file under output/ -- the test suite writes each one itself via the
|
||||
# Transcript helper, overwriting it in place. Unlike every other module in this series, this
|
||||
# one needs Docker and the baked image from scripts/bake-image.sh: these tests start a real
|
||||
# container running a real small model, not a scripted one.
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
if ! docker image inspect ollama-baked-qwen05b-minilm:local >/dev/null 2>&1; then
|
||||
echo "Image ollama-baked-qwen05b-minilm:local not found -- run scripts/bake-image.sh first." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
rm -rf target
|
||||
mvn -q -o test
|
||||
@@ -0,0 +1,19 @@
|
||||
package com.ankurm.ollamalocal;
|
||||
|
||||
import org.springframework.boot.SpringApplication;
|
||||
import org.springframework.boot.autoconfigure.SpringBootApplication;
|
||||
|
||||
/**
|
||||
* No {@code api-key} property anywhere in this module. {@code spring-ai-starter-model-ollama}
|
||||
* autoconfigures a {@link org.springframework.ai.ollama.OllamaChatModel} and a
|
||||
* {@link org.springframework.ai.ollama.OllamaEmbeddingModel} from {@code spring.ai.ollama.*}
|
||||
* properties alone, talking to a local Ollama server over plain HTTP.
|
||||
*/
|
||||
@SpringBootApplication
|
||||
public class OllamaLocalApplication {
|
||||
|
||||
public static void main(String[] args) {
|
||||
SpringApplication.run(OllamaLocalApplication.class, args);
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
package com.ankurm.ollamalocal.config;
|
||||
|
||||
import org.springframework.ai.chat.client.ChatClient;
|
||||
import org.springframework.ai.chat.model.ChatModel;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
|
||||
/**
|
||||
* The {@link ChatModel} bean itself comes from Spring AI's Ollama autoconfiguration --
|
||||
* nothing here constructs an {@code OllamaApi} or an {@code OllamaChatModel} by hand.
|
||||
* This class only wraps whatever {@code ChatModel} autoconfiguration produced in a
|
||||
* {@link ChatClient}, exactly like every other module in this series.
|
||||
*/
|
||||
@Configuration
|
||||
public class ChatClientConfig {
|
||||
|
||||
@Bean
|
||||
ChatClient chatClient(ChatModel chatModel) {
|
||||
return ChatClient.builder(chatModel).build();
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
# No API key anywhere in this file. spring.ai.ollama.base-url points at a local Ollama
|
||||
# server (default: http://localhost:11434); tests override it with the real mapped
|
||||
# endpoint of a Testcontainers-managed container via @DynamicPropertySource.
|
||||
spring:
|
||||
ai:
|
||||
ollama:
|
||||
base-url: http://localhost:11434
|
||||
chat:
|
||||
options:
|
||||
model: qwen2.5:0.5b
|
||||
embedding:
|
||||
options:
|
||||
model: all-minilm
|
||||
init:
|
||||
# never: fail fast if the model isn't already present, instead of silently pulling
|
||||
# gigabytes over the network at context-startup time. when_missing/always exist for
|
||||
# environments that want auto-pull; see the module README.
|
||||
pull-model-strategy: never
|
||||
@@ -0,0 +1,174 @@
|
||||
package com.ankurm.ollamalocal;
|
||||
|
||||
import java.net.URI;
|
||||
import java.net.http.HttpClient;
|
||||
import java.net.http.HttpRequest;
|
||||
import java.net.http.HttpResponse;
|
||||
import java.time.Duration;
|
||||
|
||||
import com.ankurm.ollamalocal.support.Transcript;
|
||||
import org.junit.jupiter.api.MethodOrderer;
|
||||
import org.junit.jupiter.api.Order;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.TestMethodOrder;
|
||||
import org.testcontainers.junit.jupiter.Container;
|
||||
import org.testcontainers.junit.jupiter.Testcontainers;
|
||||
import org.testcontainers.ollama.OllamaContainer;
|
||||
import org.testcontainers.utility.DockerImageName;
|
||||
|
||||
import org.springframework.ai.chat.client.ChatClient;
|
||||
import org.springframework.ai.chat.metadata.ChatResponseMetadata;
|
||||
import org.springframework.ai.chat.model.ChatModel;
|
||||
import org.springframework.ai.chat.prompt.Prompt;
|
||||
import org.springframework.ai.embedding.EmbeddingModel;
|
||||
import org.springframework.ai.ollama.api.OllamaChatOptions;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.test.context.SpringBootTest;
|
||||
import org.springframework.test.context.DynamicPropertyRegistry;
|
||||
import org.springframework.test.context.DynamicPropertySource;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
/**
|
||||
* One real, local Ollama server for the whole class, started from a baked image that already
|
||||
* has {@code qwen2.5:0.5b} (chat) and {@code all-minilm} (embeddings) pulled -- see
|
||||
* {@code scripts/bake-image.sh}. No network call happens during these tests: the container
|
||||
* starts from a local image and both models are already on disk inside it.
|
||||
*
|
||||
* <p>Every other module in this series drives {@code ChatClient} against a hand-written
|
||||
* {@code ScriptedChatModel} so the tests are deterministic and need no live model. This module
|
||||
* is the deliberate exception: the whole point of "run it locally" is a real model answering a
|
||||
* real prompt, so these tests assert only what is genuinely deterministic about a live small
|
||||
* model (a non-blank response, a confirmed-empty model registry right after an explicit unload)
|
||||
* and leave exact wording, and exact timings, alone.
|
||||
*/
|
||||
@Testcontainers
|
||||
@SpringBootTest
|
||||
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
|
||||
class LocalChatAndEmbeddingTest {
|
||||
|
||||
@Container
|
||||
static final OllamaContainer OLLAMA = new OllamaContainer(
|
||||
DockerImageName.parse("ollama-baked-qwen05b-minilm:local").asCompatibleSubstituteFor("ollama/ollama"));
|
||||
|
||||
@DynamicPropertySource
|
||||
static void ollamaProperties(DynamicPropertyRegistry registry) {
|
||||
registry.add("spring.ai.ollama.base-url", OLLAMA::getEndpoint);
|
||||
}
|
||||
|
||||
private static final HttpClient HTTP = HttpClient.newHttpClient();
|
||||
|
||||
@Autowired
|
||||
private ChatClient chatClient;
|
||||
|
||||
@Autowired
|
||||
private ChatModel chatModel;
|
||||
|
||||
@Autowired
|
||||
private EmbeddingModel embeddingModel;
|
||||
|
||||
@Test
|
||||
@Order(1)
|
||||
void reloadAfterAConfirmedUnload() throws Exception {
|
||||
// keepAlive("0") tells Ollama to unload the model from memory as soon as this call
|
||||
// finishes -- the same knob you'd reach for in production to free RAM/VRAM between
|
||||
// bursts of traffic. It's set through ChatModel.call() with an explicit Prompt rather
|
||||
// than through ChatClient.prompt().options(...): with this module's versions, an option
|
||||
// set that way never reaches the request Ollama receives, which the going-deeper note
|
||||
// below covers.
|
||||
this.chatModel
|
||||
.call(new Prompt("OK", OllamaChatOptions.builder().model("qwen2.5:0.5b").keepAlive("0").build()));
|
||||
|
||||
// Don't take the unload on faith -- ask Ollama's own /api/ps, which lists every
|
||||
// currently loaded model, and confirm the registry is really empty before measuring
|
||||
// what a reload costs.
|
||||
String modelsAfterUnload = getModelRegistry();
|
||||
assertThat(modelsAfterUnload).contains("\"models\":[]");
|
||||
|
||||
ChatClient.CallResponseSpec response = this.chatClient.prompt()
|
||||
.user("Reply with a single short sentence: why do developers like small local models?")
|
||||
.call();
|
||||
|
||||
String content = response.content();
|
||||
ChatResponseMetadata metadata = response.chatResponse().getMetadata();
|
||||
|
||||
assertThat(content).isNotBlank();
|
||||
// Loading is real work -- reading weights off disk and initializing the runtime -- so
|
||||
// even a fast reload can't be negative or missing.
|
||||
Duration loadDuration = metadata.get("load-duration");
|
||||
assertThat(loadDuration).isNotNull();
|
||||
assertThat(loadDuration.isNegative()).isFalse();
|
||||
|
||||
Transcript.write("01-chat-reload-after-unload",
|
||||
"/api/ps immediately after the unload call: " + modelsAfterUnload + "\n\n"
|
||||
+ "prompt: \"Reply with a single short sentence: why do developers like small local models?\"\n"
|
||||
+ "response: " + content + "\n\n" + formatOllamaMetadata(metadata));
|
||||
}
|
||||
|
||||
@Test
|
||||
@Order(2)
|
||||
void backToBackCallReusesTheAlreadyLoadedModel() {
|
||||
ChatClient.CallResponseSpec response = this.chatClient.prompt()
|
||||
.user("Reply with a single short sentence: what is Testcontainers for?")
|
||||
.call();
|
||||
|
||||
String content = response.content();
|
||||
ChatResponseMetadata metadata = response.chatResponse().getMetadata();
|
||||
|
||||
assertThat(content).isNotBlank();
|
||||
|
||||
Transcript.write("02-chat-back-to-back-call",
|
||||
"prompt: \"Reply with a single short sentence: what is Testcontainers for?\"\n"
|
||||
+ "response: " + content + "\n\n" + formatOllamaMetadata(metadata));
|
||||
}
|
||||
|
||||
@Test
|
||||
@Order(3)
|
||||
void embeddingsAreDeterministicallySized() {
|
||||
float[] vector = this.embeddingModel.embed("Spring AI can run entirely against a local Ollama server.");
|
||||
|
||||
assertThat(vector).hasSize(384);
|
||||
assertThat(this.embeddingModel.dimensions()).isEqualTo(384);
|
||||
|
||||
StringBuilder prefix = new StringBuilder();
|
||||
for (int i = 0; i < 8; i++) {
|
||||
if (i > 0) {
|
||||
prefix.append(", ");
|
||||
}
|
||||
prefix.append(String.format("%.5f", vector[i]));
|
||||
}
|
||||
|
||||
Transcript.write("03-embedding-dimensions",
|
||||
"model: all-minilm\n" + "dimensions(): " + this.embeddingModel.dimensions() + "\n"
|
||||
+ "vector.length: " + vector.length + "\n" + "first 8 values: [" + prefix + ", ...]");
|
||||
}
|
||||
|
||||
private static String getModelRegistry() throws Exception {
|
||||
HttpRequest request = HttpRequest.newBuilder(URI.create(OLLAMA.getEndpoint() + "/api/ps")).GET().build();
|
||||
return HTTP.send(request, HttpResponse.BodyHandlers.ofString()).body();
|
||||
}
|
||||
|
||||
private static String formatOllamaMetadata(ChatResponseMetadata metadata) {
|
||||
Duration total = metadata.get("total-duration");
|
||||
Duration load = metadata.get("load-duration");
|
||||
Duration promptEval = metadata.get("prompt-eval-duration");
|
||||
Duration eval = metadata.get("eval-duration");
|
||||
Integer promptEvalCount = metadata.get("prompt-eval-count");
|
||||
Integer evalCount = metadata.get("eval-count");
|
||||
|
||||
StringBuilder sb = new StringBuilder();
|
||||
sb.append("total-duration: ").append(total.toMillis()).append("ms\n");
|
||||
sb.append("load-duration: ").append(load.toMillis()).append("ms\n");
|
||||
sb.append("prompt-eval-count: ").append(promptEvalCount);
|
||||
sb.append(", prompt-eval-duration: ").append(promptEval.toMillis()).append("ms\n");
|
||||
sb.append("eval-count: ").append(evalCount);
|
||||
sb.append(", eval-duration: ").append(eval.toMillis()).append("ms");
|
||||
|
||||
if (evalCount != null && eval != null && eval.toNanos() > 0) {
|
||||
double tokensPerSecond = evalCount / (eval.toNanos() / 1_000_000_000.0);
|
||||
sb.append(String.format("%n%.2f tokens/sec (eval-count / eval-duration)", tokensPerSecond));
|
||||
}
|
||||
return sb.toString();
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
package com.ankurm.ollamalocal.support;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.UncheckedIOException;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
|
||||
/**
|
||||
* Writes a test's captured, real output to {@code output/<name>.txt} and echoes it to stdout.
|
||||
* Every console block quoted in the companion article comes from a file this class wrote --
|
||||
* nothing in the article is retyped or tidied up by hand. Unlike every other module in this
|
||||
* series, the numbers this class captures here come from a real local model, not a scripted
|
||||
* one: cold/warm load times and token rates will vary run to run and machine to machine, which
|
||||
* is exactly the point -- the article says so and gives readers the script to get their own.
|
||||
*/
|
||||
public final class Transcript {
|
||||
|
||||
private Transcript() {
|
||||
}
|
||||
|
||||
public static void write(String name, String content) {
|
||||
try {
|
||||
Path dir = Path.of("output");
|
||||
Files.createDirectories(dir);
|
||||
Path file = dir.resolve(name + ".txt");
|
||||
Files.writeString(file, content, StandardCharsets.UTF_8);
|
||||
System.out.println("--- " + name + " ---");
|
||||
System.out.println(content);
|
||||
}
|
||||
catch (IOException ex) {
|
||||
throw new UncheckedIOException(ex);
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
Reference in New Issue
Block a user