asmhatre/spring-ai
1
0
Fork 0
Code Actions Activity

Fix ollama-local: correct a wrong claim about ChatClient.options() and keepAlive

Browse Source 
Self-correction pass caught this before publishing: the previous commit's test
comment and companion post draft claimed a keepAlive set through
ChatClient.prompt().options(...) never reaches the request Ollama receives,
and worked around it by calling ChatModel.call(Prompt) directly instead. That
claim was never actually verified against /api/ps for the ChatClient path --
only inferred from a failed timing assertion that, it turned out, would have
failed the same way even with a genuinely confirmed unload (see below).

Checked directly: unloading via ChatClient.prompt().options(OllamaChatOptions
.builder()...keepAlive("0")).call() and immediately querying /api/ps shows an
empty model registry, same as the ChatModel path. The options merge works
correctly. Simplified the test back to ChatClient throughout, consistent with
the rest of this series, and removed the incorrect comment.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01FtpJvZfg4nvLvtzgJTDWpB
This commit is contained in:
Claude
2026-09-23 17:54:43 +00:00
parent faacda7079
commit c1e36c6c09
3 changed files with 18 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ollama-local/output/01-chat-reload-after-unload.txt
+6 -6
View File
@@ -1,10 +1,10 @@
/api/ps immediately after the unload call: {"models":[]} /api/ps immediately after the unload call: {"models":[]}
prompt: "Reply with a single short sentence: why do developers like small local models?" prompt: "Reply with a single short sentence: why do developers like small local models?"
response: Developers often prefer small local models because they are more efficient, faster, and easier to deploy and train. response: Developers often appreciate small local models because they are more efficient and can provide quicker, more relevant results for their applications.
total-duration: 901ms total-duration: 1076ms
load-duration: 1ms load-duration: 2ms
prompt-eval-count: 44, prompt-eval-duration: 39ms prompt-eval-count: 44, prompt-eval-duration: 41ms
eval-count: 23, eval-duration: 856ms eval-count: 27, eval-duration: 1028ms
26.87 tokens/sec (eval-count / eval-duration) 26.25 tokens/sec (eval-count / eval-duration)
ollama-local/output/02-chat-back-to-back-call.txt
+5 -5
View File
@@ -1,8 +1,8 @@
prompt: "Reply with a single short sentence: what is Testcontainers for?" prompt: "Reply with a single short sentence: what is Testcontainers for?"
response: Testcontainers is a popular containerization platform that makes it easy to create and manage application containers for testing, development, and production environments. response: Testcontainers is a containerization platform that simplifies the process of building and deploying applications quickly and efficiently.
total-duration: 722ms total-duration: 662ms
load-duration: 2ms load-duration: 2ms
prompt-eval-count: 42, prompt-eval-duration: 61ms prompt-eval-count: 42, prompt-eval-duration: 47ms
eval-count: 17, eval-duration: 653ms eval-count: 16, eval-duration: 607ms
26.02 tokens/sec (eval-count / eval-duration) 26.33 tokens/sec (eval-count / eval-duration)
ollama-local/src/test/java/com/ankurm/ollamalocal/LocalChatAndEmbeddingTest.java
+7 -11
View File
@@ -18,8 +18,6 @@ import org.testcontainers.utility.DockerImageName;
import org.springframework.ai.chat.client.ChatClient; import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.metadata.ChatResponseMetadata; import org.springframework.ai.chat.metadata.ChatResponseMetadata;
import org.springframework.ai.chat.model.ChatModel;
import org.springframework.ai.chat.prompt.Prompt;
import org.springframework.ai.embedding.EmbeddingModel; import org.springframework.ai.embedding.EmbeddingModel;
import org.springframework.ai.ollama.api.OllamaChatOptions; import org.springframework.ai.ollama.api.OllamaChatOptions;
import org.springframework.beans.factory.annotation.Autowired; import org.springframework.beans.factory.annotation.Autowired;
@@ -61,9 +59,6 @@ class LocalChatAndEmbeddingTest {
@Autowired @Autowired
private ChatClient chatClient; private ChatClient chatClient;
@Autowired
private ChatModel chatModel;
@Autowired @Autowired
private EmbeddingModel embeddingModel; private EmbeddingModel embeddingModel;
@@ -72,12 +67,13 @@ class LocalChatAndEmbeddingTest {
void reloadAfterAConfirmedUnload() throws Exception { void reloadAfterAConfirmedUnload() throws Exception {
// keepAlive("0") tells Ollama to unload the model from memory as soon as this call // keepAlive("0") tells Ollama to unload the model from memory as soon as this call
// finishes -- the same knob you'd reach for in production to free RAM/VRAM between // finishes -- the same knob you'd reach for in production to free RAM/VRAM between
// bursts of traffic. It's set through ChatModel.call() with an explicit Prompt rather // bursts of traffic. Passed through ChatClient's per-call options() exactly like any
// than through ChatClient.prompt().options(...): with this module's versions, an option // other OllamaChatOptions field.
// set that way never reaches the request Ollama receives, which the going-deeper note this.chatClient.prompt()
// below covers. .user("OK")
this.chatModel .options(OllamaChatOptions.builder().model("qwen2.5:0.5b").keepAlive("0"))
.call(new Prompt("OK", OllamaChatOptions.builder().model("qwen2.5:0.5b").keepAlive("0").build())); .call()
.content();
// Don't take the unload on faith -- ask Ollama's own /api/ps, which lists every // Don't take the unload on faith -- ask Ollama's own /api/ps, which lists every
// currently loaded model, and confirm the registry is really empty before measuring // currently loaded model, and confirm the registry is really empty before measuring