diff --git a/README.md b/README.md index c6097bd..e8e8592 100644 --- a/README.md +++ b/README.md @@ -15,6 +15,7 @@ facts, so the build fails when a claim stops being true. | `code-review-bot` | Build an AI Code-Review Bot in Java | a signed-webhook receiver, a diff chunker, a Spring AI reviewer with a cache and a chunk cap, and a guard that drops comments on lines the model invented | | `agentic-patterns` | Agentic Patterns in Spring AI: Routing, Orchestrator-Workers, Evaluator-Optimizer and Parallelization | Anthropic's five workflows as plain `ChatClient` code, tested offline with a scripted `ChatModel` that records every prompt, with failure modes and a calls-and-stages cost table | | `mcp-sdk-plain` | Write an MCP Server with the Plain MCP Java SDK (No Spring): stdio and Streamable HTTP | tools, a resource and a resource template on the framework-free SDK, a fat jar over stdio and Streamable HTTP, wire-level transcripts, and a Spring AI client connecting to it | +| `koog` | Koog: JetBrains' Kotlin AI Agent Framework on the JVM, Your First Agent with Tools in Java | a tool-calling agent, a strategy graph with a plain-Java node before the model, and the Spring Boot starter, tested against a scripted OpenAI-compatible endpoint, with tool-error, bad-argument, iteration-cap and Kotlin-BOM failure modes | ## Versions (verified on Maven Central, 2026-10-11) @@ -34,6 +35,8 @@ facts, so the build fails when a claim stops being true. | MCP Java SDK `io.modelcontextprotocol.sdk:mcp` | 2.1.0 | verified on Maven Central 2026-11-08; `mcp-sdk-plain` only; pom-only aggregator over `mcp-core` and `mcp-json-jackson3` | | Jetty `jetty-ee11-servlet` | 12.1.14 | `mcp-sdk-plain` only; the SDK declares the servlet API as `provided`, so HTTP needs a container | | Spring Boot / Spring AI (client example) | 4.1.2 / 2.0.2 | `mcp-sdk-plain/examples/spring-ai-client` only, a standalone project | +| Koog `koog-agents-jvm` | 1.3.0 | verified on Maven Central 2026-11-11; `koog` module only. `koog-spring-boot-starter` is 1.3.0-beta and brings Spring Boot 3.5.14, which is why this module uses Spring Boot 3.5.16 and not the 4.x of the others | +| Kotlin `kotlin-stdlib` | 2.3.10 | the Spring Boot 3.5.16 BOM pins 1.9.25 and would win; the `koog` pom imports the Kotlin BOMs first | | JDK | 25 LTS | | | JUnit | 6.1.3 | | diff --git a/koog/README.md b/koog/README.md new file mode 100644 index 0000000..65863d8 --- /dev/null +++ b/koog/README.md @@ -0,0 +1,32 @@ +# koog + +Companion code for *Koog: JetBrains' Kotlin AI Agent Framework on the JVM, Your First Agent with Tools in Java* +on [ankurm.com](https://ankurm.com). One agent with two tools, the same agent as an explicit strategy graph with a +plain-Java node in front of the model, and the Spring Boot starter around it. + +There is no OpenAI key here. `ScriptedOpenAIServer` is a local HTTP server that speaks the chat-completions +format and answers from a script, and Koog's real `OpenAILLMClient` is pointed at it. The agent loop, the tool +definitions Koog derives, the argument parsing and the HTTP calls all run for real; the model is the one scripted +part. What a real model would decide is not tested here. + +```bash +mvn test -pl koog # rewrites output/01..06 +koog/scripts/kotlin-bom-trap.sh # rewrites output/07 (run from the repository root, after mvn test) +``` + +| File | Role | +|---|---| +| `ServiceTools` | Two tools on a `ToolSet`: `@Tool`, `@LLMDescription`, `customName` | +| `OpsAgents` | The first agent, the redacting graph strategy, the Mermaid diagram | +| `OpsService`, `OpsController`, `OpsApplication` | Spring Boot: the starter's `openAIExecutor` bean and an agent built per request | +| `ScriptedOpenAIServer`, `TestExecutors` | The stand-in model endpoint and an executor pointed at it | +| `FirstAgentTest` | `output/01`: what Koog sends, request by request | +| `StrategyGraphTest` | `output/02`: the graph, and a secret that never reaches the model | +| `FailureModesTest` | `output/03`, `output/04`: tool errors, bad arguments, unknown tools, the iteration cap, running an agent twice | +| `SpringBootStarterTest` | `output/05`, `output/06`: the beans the starter registers, and startup with no API key | +| `scripts/kotlin-bom-trap.sh` | `output/07`: the `NoClassDefFoundError` when Spring Boot's BOM keeps Kotlin on 1.9.25 | + +The `pom.xml` imports the Kotlin BOMs **before** the Spring Boot BOM on purpose; the comment there explains why. + +Verified against `ai.koog:koog-agents-jvm` **1.3.0** and `ai.koog:koog-spring-boot-starter` **1.3.0-beta** (beta), +Spring Boot **3.5.16**, Kotlin **2.3.10**, JDK 25, JUnit 6.1.3. diff --git a/koog/output/01-first-agent-on-the-wire.txt b/koog/output/01-first-agent-on-the-wire.txt new file mode 100644 index 0000000..852fd14 --- /dev/null +++ b/koog/output/01-first-agent-on-the-wire.txt @@ -0,0 +1,18 @@ +agent.run("Is checkout healthy?") -> checkout is DEGRADED + +HTTP calls the model endpoint received: 2 + POST /v1/chat/completions + POST /v1/chat/completions + +Request 1 - messages: + system: You are an on-call assistant. Use the tools to answer; never guess a status. + user: Is checkout healthy? +Request 1 - tools Koog derived from ServiceTools: + {"name":"getServiceStatus","description":"Returns the current status of one service: HEALTHY, DEGRADED or DOWN","parameters":{"type":"object","properties":{"service":{"description":"Service name, for example checkout","type":"string"}},"required":["service"]}} + {"name":"error_rate","description":"Returns the percentage of failed requests for one service over the last N minutes","parameters":{"type":"object","properties":{"service":{"description":"Service name, for example checkout","type":"string"},"minutes":{"description":"Window in minutes","type":"integer"}},"required":["service","minutes"]}} + +Request 2 - messages (the tool call and its result are now in the history): + {"role":"system","content":"You are an on-call assistant. Use the tools to answer; never guess a status."} + {"role":"user","content":"Is checkout healthy?"} + {"role":"assistant","tool_calls":[{"id":"call_1","function":{"name":"getServiceStatus","arguments":"{\"service\":\"checkout\"}"},"type":"function"}]} + {"role":"tool","content":"\"DEGRADED\"","tool_call_id":"call_1"} diff --git a/koog/output/02-strategy-graph-and-redaction.txt b/koog/output/02-strategy-graph-and-redaction.txt new file mode 100644 index 0000000..6fdecd2 --- /dev/null +++ b/koog/output/02-strategy-graph-and-redaction.txt @@ -0,0 +1,23 @@ +Graph, as MermaidDiagramGenerator prints it: + +--- +title: redacting_single_run +--- +stateDiagram + state "redact" as redact + state "callLLM" as callLLM + state "executeTool" as executeTool + state "sendToolResult" as sendToolResult + + [*] --> redact + redact --> callLLM + callLLM --> executeTool : transformed + callLLM --> [*] : transformed + executeTool --> sendToolResult + sendToolResult --> executeTool : transformed + sendToolResult --> [*] : transformed + +The user typed: My key is sk-ABCDEFGH12345678. Is checkout healthy? +The model got: My key is [REDACTED]. Is checkout healthy? +Answer: checkout is DEGRADED +Secret present in any request body: false diff --git a/koog/output/03-failure-modes.txt b/koog/output/03-failure-modes.txt new file mode 100644 index 0000000..dba9776 --- /dev/null +++ b/koog/output/03-failure-modes.txt @@ -0,0 +1,17 @@ +1. The tool throws IllegalArgumentException("Unknown service: nope") + agent.run returned: There is no such service. + LLM calls: 2 + what the model was told: tool: Tool with name 'getServiceStatus' failed to execute due to the error: Unknown service: nope! + +2. The model sends {"minutes":"soon"} for an int parameter + what the model was told: Tool with name 'error_rate' failed to parse arguments due to the error: Failed to parse literal '"soon"' as an int value at element: $.primitive | JSON input: "soon" + +3. The model calls restartService, which is not registered + the history sent back, last two messages: + {"role":"assistant","content":"{\"tool_call_id\":\"c1\",\"tool_name\":\"restartService\",\"tool_args\":{\"service\":\"checkout\"}}"} + {"role":"user","content":"{\"tool_call_id\":\"c1\",\"tool_name\":\"restartService\",\"tool_result\":\"Tool with name 'restartService' not found in the tool registry. Use one of the available tools.\"}"} + +4. The model answers every request with another tool call, maxIterations(6) + AIAgentMaxNumberOfIterationsReachedException + LLM calls made before it stopped: 3 + default maxIterations of an agent you do not configure: 50 diff --git a/koog/output/04-agent-reuse.txt b/koog/output/04-agent-reuse.txt new file mode 100644 index 0000000..a4ec97f --- /dev/null +++ b/koog/output/04-agent-reuse.txt @@ -0,0 +1,4 @@ +run 1: first +run 2: second +messages sent on run 2: 1 + {"role":"user","content":"hello again"} diff --git a/koog/output/05-spring-boot-starter.txt b/koog/output/05-spring-boot-starter.txt new file mode 100644 index 0000000..d6b5b18 --- /dev/null +++ b/koog/output/05-spring-boot-starter.txt @@ -0,0 +1,11 @@ +PromptExecutor beans the starter registered (only OpenAI is configured): + multiLLMPromptExecutor + openAIExecutor + +GET /ask?q=How bad are checkout errors? + -> checkout fails 4.2% of requests over 15 minutes +Model endpoint saw: [POST /v1/chat/completions, POST /v1/chat/completions] +Tool argument the model sent: minutes=15 (an int, parsed by Koog) + +context.getBean(PromptExecutor.class) with one provider configured: + NoUniqueBeanDefinitionException: expected single matching bean but found 2: multiLLMPromptExecutor,openAIExecutor diff --git a/koog/output/06-spring-boot-no-api-key.txt b/koog/output/06-spring-boot-no-api-key.txt new file mode 100644 index 0000000..e3a1717 --- /dev/null +++ b/koog/output/06-spring-boot-no-api-key.txt @@ -0,0 +1,4 @@ +ai.koog.openai.api-key is empty, ai.koog.openai.enabled=true is set: + startup failed: true + cause: NoSuchBeanDefinitionException + message: No qualifying bean of type 'ai.koog.prompt.executor.model.PromptExecutor' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {@org.springframework.beans.factory.annotation.Qualifier("openAIExecutor")} diff --git a/koog/output/07-kotlin-bom-trap.txt b/koog/output/07-kotlin-bom-trap.txt new file mode 100644 index 0000000..1aa46df --- /dev/null +++ b/koog/output/07-kotlin-bom-trap.txt @@ -0,0 +1,11 @@ +kotlin-stdlib on the classpath (correct, Kotlin BOM imported first): + kotlin-stdlib-2.3.10.jar + +Same classpath with kotlin-stdlib 1.9.25, which is what the Spring Boot 3.5.16 BOM picks when it comes first: + kotlin-stdlib-1.9.25.jar + +$ java -cp com.ankurm.koog.TrapMain + Exception in thread "main" java.lang.NoClassDefFoundError: kotlin/time/Instant + at ai.koog.utils.time.KoogClock.(KoogClock.kt:27) + at ai.koog.prompt.executor.model.PromptExecutorBuilder.openAI$default(PromptExecutorBuilder.kt:85) + at ai.koog.prompt.executor.model.PromptExecutorBuilder.openAI(PromptExecutorBuilder.kt) diff --git a/koog/pom.xml b/koog/pom.xml new file mode 100644 index 0000000..556a84b --- /dev/null +++ b/koog/pom.xml @@ -0,0 +1,89 @@ + + + 4.0.0 + + + com.ankurm.agents + java-ai-agents + 1.0.0 + + + koog + Koog: an agent with tools, a strategy graph and Spring Boot + + + + 1.3.0 + 1.3.0-beta + + 3.5.16 + 1.10.0 + 1.10.2 + + + + + + + org.jetbrains.kotlin + kotlin-bom + 2.3.10 + pom + import + + + org.jetbrains.kotlinx + kotlinx-coroutines-bom + ${kotlinx-coroutines.version} + pom + import + + + org.jetbrains.kotlinx + kotlinx-serialization-bom + ${kotlinx-serialization.version} + pom + import + + + org.springframework.boot + spring-boot-dependencies + ${koog.spring-boot.version} + pom + import + + + + + + + ai.koog + koog-agents-jvm + ${koog.version} + + + ai.koog + koog-spring-boot-starter + ${koog.beta.version} + + + org.springframework.boot + spring-boot-starter-web + + + org.springframework.boot + spring-boot-starter-test + test + + + org.junit.jupiter + junit-jupiter + test + + + diff --git a/koog/scripts/kotlin-bom-trap.sh b/koog/scripts/kotlin-bom-trap.sh new file mode 100755 index 0000000..3e1cad7 --- /dev/null +++ b/koog/scripts/kotlin-bom-trap.sh @@ -0,0 +1,25 @@ +#!/usr/bin/env bash +# Reproduces the failure you get when Spring Boot's BOM wins and Kotlin stays on 1.9.25. +# Run from the repository root after `mvn test -pl koog` (needs koog/target/test-classes). +set -eu +cd "$(dirname "$0")/.." +OUT=output/07-kotlin-bom-trap.txt +mvn -B -q dependency:build-classpath -Dmdep.outputFile=target/cp.txt >/dev/null 2>&1 +mvn -B -q dependency:get -Dartifact=org.jetbrains.kotlin:kotlin-stdlib:1.9.25 >/dev/null 2>&1 +OLD="$HOME/.m2/repository/org/jetbrains/kotlin/kotlin-stdlib/1.9.25/kotlin-stdlib-1.9.25.jar" + +good="$(cat target/cp.txt)" +bad="$(echo "$good" | tr ':' '\n' | sed "s#.*/kotlin-stdlib-2\.3\.10\.jar#$OLD#" | tr '\n' ':')" + +{ + echo "kotlin-stdlib on the classpath (correct, Kotlin BOM imported first):" + echo "$good" | tr ':' '\n' | grep -o 'kotlin-stdlib-[0-9.]*\.jar' | sed 's/^/ /' + echo + echo "Same classpath with kotlin-stdlib 1.9.25, which is what the Spring Boot 3.5.16 BOM picks when it comes first:" + echo "$bad" | tr ':' '\n' | grep -o 'kotlin-stdlib-[0-9.]*\.jar' | sed 's/^/ /' + echo + echo '$ java -cp com.ankurm.koog.TrapMain' + java -cp "target/test-classes:target/classes:$bad" com.ankurm.koog.TrapMain 2>&1 \ + | grep -vE 'SLF4J|Picked up|kotlin-logging' | head -4 | sed 's/^/ /' +} > "$OUT" +cat "$OUT" diff --git a/koog/scripts/run-all.sh b/koog/scripts/run-all.sh new file mode 100755 index 0000000..885d9f1 --- /dev/null +++ b/koog/scripts/run-all.sh @@ -0,0 +1,6 @@ +#!/usr/bin/env bash +# Regenerates every file in output/. Needs JDK 25 on JAVA_HOME and a network for the first Maven run. +set -eu +cd "$(dirname "$0")/.." +mvn -B -q test +scripts/kotlin-bom-trap.sh diff --git a/koog/src/main/java/com/ankurm/koog/OpsAgents.java b/koog/src/main/java/com/ankurm/koog/OpsAgents.java new file mode 100644 index 0000000..b76adc1 --- /dev/null +++ b/koog/src/main/java/com/ankurm/koog/OpsAgents.java @@ -0,0 +1,70 @@ +package com.ankurm.koog; + +import ai.koog.agents.core.agent.AIAgent; +import ai.koog.agents.core.agent.MermaidDiagramGenerator; +import ai.koog.agents.core.agent.entity.AIAgentEdge; +import ai.koog.agents.core.agent.entity.AIAgentGraphStrategy; +import ai.koog.agents.core.agent.entity.AIAgentNode; +import ai.koog.agents.core.tools.ToolRegistry; +import ai.koog.prompt.executor.clients.openai.OpenAIModels; +import ai.koog.prompt.executor.model.PromptExecutor; + +/** Builds the agents used by the tests and by the Spring Boot service. */ +public final class OpsAgents { + + public static final String SYSTEM_PROMPT = + "You are an on-call assistant. Use the tools to answer; never guess a status."; + + private OpsAgents() {} + + /** The smallest useful agent: an executor, a model, a system prompt and a tool registry. */ + public static AIAgent firstAgent(PromptExecutor executor) { + return AIAgent.builder() + .promptExecutor(executor) + .llmModel(OpenAIModels.Chat.GPT4o) + .systemPrompt(SYSTEM_PROMPT) + .toolRegistry(ToolRegistry.builder().tools(new ServiceTools()).build()) + .build(); + } + + /** The same agent with one extra, ordinary Java node in front of the model: a redactor. */ + public static AIAgentGraphStrategy redactingStrategy() { + var graph = AIAgentGraphStrategy.builder("redacting_single_run") + .withInput(String.class) + .withOutput(String.class); + + // A node is a function of (input, context). This one never talks to the model. + var redact = AIAgentNode.builder("redact") + .withInput(String.class) + .withOutput(String.class) + .withAction((input, ctx) -> input.replaceAll("sk-[A-Za-z0-9]{8,}", "[REDACTED]")) + .build(); + + var callLLM = AIAgentNode.llmRequest("callLLM"); + var executeTool = AIAgentNode.executeTools("executeTool"); + var sendToolResult = AIAgentNode.llmSendToolResults("sendToolResult"); + + graph.edge(graph.nodeStart, redact); + graph.edge(redact, callLLM); + graph.edge(AIAgentEdge.builder().from(callLLM).to(executeTool).onToolCalls().build()); + graph.edge(AIAgentEdge.builder().from(callLLM).to(graph.nodeFinish).onTextMessage().build()); + graph.edge(executeTool, sendToolResult); + graph.edge(AIAgentEdge.builder().from(sendToolResult).to(executeTool).onToolCalls().build()); + graph.edge(AIAgentEdge.builder().from(sendToolResult).to(graph.nodeFinish).onTextMessage().build()); + return graph.build(); + } + + public static AIAgent redactingAgent(PromptExecutor executor) { + return AIAgent.builder() + .graphStrategy(redactingStrategy()) + .promptExecutor(executor) + .llmModel(OpenAIModels.Chat.GPT4o) + .systemPrompt(SYSTEM_PROMPT) + .toolRegistry(ToolRegistry.builder().tools(new ServiceTools()).build()) + .build(); + } + + public static String diagram() { + return MermaidDiagramGenerator.INSTANCE.generate(redactingStrategy()); + } +} diff --git a/koog/src/main/java/com/ankurm/koog/OpsApplication.java b/koog/src/main/java/com/ankurm/koog/OpsApplication.java new file mode 100644 index 0000000..e33b50f --- /dev/null +++ b/koog/src/main/java/com/ankurm/koog/OpsApplication.java @@ -0,0 +1,11 @@ +package com.ankurm.koog; + +import org.springframework.boot.SpringApplication; +import org.springframework.boot.autoconfigure.SpringBootApplication; + +@SpringBootApplication +public class OpsApplication { + public static void main(String[] args) { + SpringApplication.run(OpsApplication.class, args); + } +} diff --git a/koog/src/main/java/com/ankurm/koog/OpsController.java b/koog/src/main/java/com/ankurm/koog/OpsController.java new file mode 100644 index 0000000..bdd6c0a --- /dev/null +++ b/koog/src/main/java/com/ankurm/koog/OpsController.java @@ -0,0 +1,18 @@ +package com.ankurm.koog; + +import org.springframework.web.bind.annotation.GetMapping; +import org.springframework.web.bind.annotation.RequestParam; +import org.springframework.web.bind.annotation.RestController; + +@RestController +public class OpsController { + + private final OpsService service; + + public OpsController(OpsService service) { this.service = service; } + + @GetMapping("/ask") + public String ask(@RequestParam("q") String question) { + return service.ask(question); + } +} diff --git a/koog/src/main/java/com/ankurm/koog/OpsService.java b/koog/src/main/java/com/ankurm/koog/OpsService.java new file mode 100644 index 0000000..6d4ddd5 --- /dev/null +++ b/koog/src/main/java/com/ankurm/koog/OpsService.java @@ -0,0 +1,25 @@ +package com.ankurm.koog; + +import ai.koog.agents.core.agent.AIAgent; +import ai.koog.prompt.executor.model.PromptExecutor; +import org.springframework.beans.factory.annotation.Qualifier; +import org.springframework.stereotype.Service; + +/** + * The starter gives us a PromptExecutor bean per configured provider. It does not give us an agent: + * we build one per request around the shared executor, so no run shares state with another caller. + */ +@Service +public class OpsService { + + private final PromptExecutor executor; + + public OpsService(@Qualifier("openAIExecutor") PromptExecutor executor) { + this.executor = executor; + } + + public String ask(String question) { + AIAgent agent = OpsAgents.firstAgent(executor); + return agent.run(question); + } +} diff --git a/koog/src/main/java/com/ankurm/koog/ServiceTools.java b/koog/src/main/java/com/ankurm/koog/ServiceTools.java new file mode 100644 index 0000000..a36cd7c --- /dev/null +++ b/koog/src/main/java/com/ankurm/koog/ServiceTools.java @@ -0,0 +1,48 @@ +package com.ankurm.koog; + +import ai.koog.agents.core.tools.annotations.LLMDescription; +import ai.koog.agents.core.tools.annotations.Tool; +import ai.koog.agents.core.tools.reflect.ToolSet; + +import java.util.Map; + +/** + * Two read-only tools over a hard-coded service inventory. The model never sees this class: + * Koog reads the annotations and the method signatures and turns them into tool definitions. + */ +@LLMDescription("Read-only questions about the health of our services") +public class ServiceTools implements ToolSet { + + private static final Map STATUS = Map.of( + "checkout", "DEGRADED", + "search", "HEALTHY", + "billing", "HEALTHY"); + + private static final Map ERROR_RATE = Map.of( + "checkout", 4.2, + "search", 0.1, + "billing", 0.3); + + @Tool + @LLMDescription("Returns the current status of one service: HEALTHY, DEGRADED or DOWN") + public String getServiceStatus( + @LLMDescription("Service name, for example checkout") String service) { + String status = STATUS.get(service); + if (status == null) { + throw new IllegalArgumentException("Unknown service: " + service); + } + return status; + } + + @Tool(customName = "error_rate") + @LLMDescription("Returns the percentage of failed requests for one service over the last N minutes") + public double getErrorRate( + @LLMDescription("Service name, for example checkout") String service, + @LLMDescription("Window in minutes") int minutes) { + Double rate = ERROR_RATE.get(service); + if (rate == null) { + throw new IllegalArgumentException("Unknown service: " + service); + } + return rate; + } +} diff --git a/koog/src/main/resources/application.properties b/koog/src/main/resources/application.properties new file mode 100644 index 0000000..b30fd0b --- /dev/null +++ b/koog/src/main/resources/application.properties @@ -0,0 +1,4 @@ +# The starter reads ai.koog..*; setting OPENAI_API_KEY in the environment also activates it. +ai.koog.openai.api-key=${OPENAI_API_KEY:} +# Point the OpenAI client somewhere else (a proxy, a gateway, or the scripted server in the tests). +# ai.koog.openai.base-url=https://api.openai.com diff --git a/koog/src/test/java/com/ankurm/koog/FailureModesTest.java b/koog/src/test/java/com/ankurm/koog/FailureModesTest.java new file mode 100644 index 0000000..1fb919c --- /dev/null +++ b/koog/src/test/java/com/ankurm/koog/FailureModesTest.java @@ -0,0 +1,110 @@ +package com.ankurm.koog; + +import ai.koog.agents.core.agent.AIAgent; +import ai.koog.agents.core.agent.exception.AIAgentMaxNumberOfIterationsReachedException; +import ai.koog.agents.core.tools.ToolRegistry; +import ai.koog.prompt.executor.clients.openai.OpenAIModels; +import com.fasterxml.jackson.databind.JsonNode; +import org.junit.jupiter.api.Test; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/** What the agent does when the model, or a tool, misbehaves. Every case is a scripted model reply. */ +class FailureModesTest { + + private static JsonNode lastMessages(ScriptedOpenAIServer model) throws Exception { + String last = model.requests.get(model.requests.size() - 1); + return FirstAgentTest.JSON.readTree(last).get("messages"); + } + + @Test + void failuresReachTheModelAsText() throws Exception { + var t = new Transcript("03-failure-modes.txt"); + + // 1. a tool throws + try (var model = new ScriptedOpenAIServer()) { + model.thenToolCall("c1", "getServiceStatus", "{\"service\":\"nope\"}").thenText("There is no such service."); + String answer = OpsAgents.firstAgent(TestExecutors.openAI(model)).run("status of nope?"); + JsonNode msgs = lastMessages(model); + JsonNode toolMsg = msgs.get(msgs.size() - 1); + t.line("1. The tool throws IllegalArgumentException(\"Unknown service: nope\")"); + t.line(" agent.run returned: " + answer); + t.line(" LLM calls: " + model.requests.size()); + t.line(" what the model was told: " + toolMsg.get("role").asText() + ": " + toolMsg.get("content").asText()); + assertThat(answer).isEqualTo("There is no such service."); + assertThat(toolMsg.get("role").asText()).isEqualTo("tool"); + assertThat(toolMsg.get("content").asText()).contains("Unknown service: nope"); + } + + // 2. the model sends arguments of the wrong type + try (var model = new ScriptedOpenAIServer()) { + model.thenToolCall("c1", "error_rate", "{\"service\":\"checkout\",\"minutes\":\"soon\"}").thenText("sorry"); + OpsAgents.firstAgent(TestExecutors.openAI(model)).run("error rate?"); + JsonNode msgs = lastMessages(model); + JsonNode toolMsg = msgs.get(msgs.size() - 1); + t.line(""); + t.line("2. The model sends {\"minutes\":\"soon\"} for an int parameter"); + t.line(" what the model was told: " + toolMsg.get("content").asText().replace("\n", " | ")); + assertThat(toolMsg.get("content").asText()).contains("failed to parse arguments"); + } + + // 3. the model calls a tool that was never registered + try (var model = new ScriptedOpenAIServer()) { + model.thenToolCall("c1", "restartService", "{\"service\":\"checkout\"}").thenText("ok"); + OpsAgents.firstAgent(TestExecutors.openAI(model)).run("restart checkout"); + JsonNode msgs = lastMessages(model); + t.line(""); + t.line("3. The model calls restartService, which is not registered"); + t.line(" the history sent back, last two messages:"); + for (int i = msgs.size() - 2; i < msgs.size(); i++) t.line(" " + msgs.get(i)); + assertThat(msgs.get(msgs.size() - 1).get("content").asText()).contains("not found in the tool registry"); + } + + // 4. the model never stops calling tools + try (var model = new ScriptedOpenAIServer()) { + for (int i = 0; i < 20; i++) model.thenToolCall("c" + i, "getServiceStatus", "{\"service\":\"search\"}"); + AIAgent agent = AIAgent.builder() + .promptExecutor(TestExecutors.openAI(model)) + .llmModel(OpenAIModels.Chat.GPT4o) + .toolRegistry(ToolRegistry.builder().tools(new ServiceTools()).build()) + .maxIterations(6) + .build(); + t.line(""); + t.line("4. The model answers every request with another tool call, maxIterations(6)"); + Throwable thrown = null; + try { agent.run("loop"); } catch (Throwable e) { thrown = e; } + t.line(" " + thrown.getClass().getSimpleName()); + t.line(" LLM calls made before it stopped: " + model.requests.size()); + t.line(" default maxIterations of an agent you do not configure: " + + OpsAgents.firstAgent(TestExecutors.openAI(model)).getAgentConfig().getMaxAgentIterations()); + assertThat(thrown).isInstanceOf(AIAgentMaxNumberOfIterationsReachedException.class); + assertThat(model.requests.size()).isEqualTo(3); + } + t.write(); + } + + @Test + void anAgentInstanceCanRunTwice_andStartsFresh() throws Exception { + var t = new Transcript("04-agent-reuse.txt"); + try (var model = new ScriptedOpenAIServer()) { + model.thenText("first").thenText("second"); + AIAgent agent = AIAgent.builder() + .promptExecutor(TestExecutors.openAI(model)) + .llmModel(OpenAIModels.Chat.GPT4o) + .build(); + String first = agent.run("hello"); + String second = agent.run("hello again"); + t.line("run 1: " + first); + t.line("run 2: " + second); + JsonNode run2 = FirstAgentTest.JSON.readTree(model.requests.get(1)).get("messages"); + t.line("messages sent on run 2: " + run2.size()); + run2.forEach(m -> t.line(" " + m)); + assertThat(first).isEqualTo("first"); + assertThat(second).isEqualTo("second"); + // a second run() starts from a fresh history; run 1 does not leak into it + assertThat(run2).hasSize(1); + } + t.write(); + } +} diff --git a/koog/src/test/java/com/ankurm/koog/FirstAgentTest.java b/koog/src/test/java/com/ankurm/koog/FirstAgentTest.java new file mode 100644 index 0000000..3ff8a92 --- /dev/null +++ b/koog/src/test/java/com/ankurm/koog/FirstAgentTest.java @@ -0,0 +1,59 @@ +package com.ankurm.koog; + +import ai.koog.agents.core.agent.AIAgent; +import ai.koog.prompt.executor.model.PromptExecutor; +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; +import org.junit.jupiter.api.Test; + +import static org.assertj.core.api.Assertions.assertThat; + +class FirstAgentTest { + + static final ObjectMapper JSON = new ObjectMapper(); + + @Test + void agentCallsATool_thenAnswers() throws Exception { + try (var model = new ScriptedOpenAIServer()) { + model.thenToolCall("call_1", "getServiceStatus", "{\"service\":\"checkout\"}") + .thenText("checkout is DEGRADED"); + PromptExecutor executor = TestExecutors.openAI(model); + AIAgent agent = OpsAgents.firstAgent(executor); + + String answer = agent.run("Is checkout healthy?"); + + var t = new Transcript("01-first-agent-on-the-wire.txt"); + t.line("agent.run(\"Is checkout healthy?\") -> " + answer); + t.line(""); + t.line("HTTP calls the model endpoint received: " + model.paths.size()); + model.paths.forEach(p -> t.line(" " + p)); + + JsonNode first = JSON.readTree(model.requests.get(0)); + t.line(""); + t.line("Request 1 - messages:"); + first.get("messages").forEach(m -> t.line(" " + m.get("role").asText() + ": " + m.get("content").asText())); + t.line("Request 1 - tools Koog derived from ServiceTools:"); + first.get("tools").forEach(tool -> t.line(" " + compact(tool.get("function")))); + + JsonNode second = JSON.readTree(model.requests.get(1)); + t.line(""); + t.line("Request 2 - messages (the tool call and its result are now in the history):"); + second.get("messages").forEach(m -> t.line(" " + compact(m))); + t.write(); + + assertThat(answer).isEqualTo("checkout is DEGRADED"); + assertThat(model.paths).containsExactly("POST /v1/chat/completions", "POST /v1/chat/completions"); + assertThat(first.get("tools")).hasSize(2); + assertThat(first.get("tools").get(0).get("function").get("name").asText()).isEqualTo("getServiceStatus"); + // customName wins over the method name + assertThat(first.get("tools").get(1).get("function").get("name").asText()).isEqualTo("error_rate"); + JsonNode toolMessage = second.get("messages").get(3); + assertThat(toolMessage.get("role").asText()).isEqualTo("tool"); + assertThat(toolMessage.get("content").asText()).isEqualTo("\"DEGRADED\""); + } + } + + static String compact(JsonNode n) { + return n.toString(); + } +} diff --git a/koog/src/test/java/com/ankurm/koog/ScriptedOpenAIServer.java b/koog/src/test/java/com/ankurm/koog/ScriptedOpenAIServer.java new file mode 100644 index 0000000..5f58dde --- /dev/null +++ b/koog/src/test/java/com/ankurm/koog/ScriptedOpenAIServer.java @@ -0,0 +1,82 @@ +package com.ankurm.koog; + +import com.sun.net.httpserver.HttpServer; + +import java.io.IOException; +import java.net.InetSocketAddress; +import java.nio.charset.StandardCharsets; +import java.util.ArrayDeque; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Queue; + +/** + * A stand-in for api.openai.com: a local HTTP server that speaks the chat-completions wire format + * and answers from a script. Koog's real OpenAI client, tool serialisation and agent loop all run; + * only the model is scripted. Every request body is recorded so tests can show what Koog sent. + */ +final class ScriptedOpenAIServer implements AutoCloseable { + + private final HttpServer server; + private final Queue script = new ArrayDeque<>(); + final List requests = Collections.synchronizedList(new ArrayList<>()); + final List paths = Collections.synchronizedList(new ArrayList<>()); + + ScriptedOpenAIServer() throws IOException { + server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0); + server.createContext("/", exchange -> { + String body = new String(exchange.getRequestBody().readAllBytes(), StandardCharsets.UTF_8); + paths.add(exchange.getRequestMethod() + " " + exchange.getRequestURI().getPath()); + requests.add(body); + String reply = script.poll(); + if (reply == null) { + reply = text("(script exhausted)"); + } + byte[] bytes = reply.getBytes(StandardCharsets.UTF_8); + exchange.getResponseHeaders().add("Content-Type", "application/json"); + exchange.sendResponseHeaders(200, bytes.length); + exchange.getResponseBody().write(bytes); + exchange.close(); + }); + server.start(); + } + + String baseUrl() { return "http://127.0.0.1:" + server.getAddress().getPort(); } + + ScriptedOpenAIServer thenToolCall(String id, String tool, String argumentsJson) { + script.add(toolCall(id, tool, argumentsJson)); + return this; + } + + ScriptedOpenAIServer thenText(String content) { + script.add(text(content)); + return this; + } + + static String text(String content) { + return """ + {"id":"chatcmpl-1","object":"chat.completion","created":1,"model":"gpt-4o", + "choices":[{"index":0,"finish_reason":"stop", + "message":{"role":"assistant","content":%s}}], + "usage":{"prompt_tokens":1,"completion_tokens":1,"total_tokens":2}} + """.formatted(quote(content)); + } + + static String toolCall(String id, String tool, String argumentsJson) { + return """ + {"id":"chatcmpl-1","object":"chat.completion","created":1,"model":"gpt-4o", + "choices":[{"index":0,"finish_reason":"tool_calls", + "message":{"role":"assistant","content":null, + "tool_calls":[{"id":%s,"type":"function", + "function":{"name":%s,"arguments":%s}}]}}], + "usage":{"prompt_tokens":1,"completion_tokens":1,"total_tokens":2}} + """.formatted(quote(id), quote(tool), quote(argumentsJson)); + } + + private static String quote(String s) { + return "\"" + s.replace("\\", "\\\\").replace("\"", "\\\"").replace("\n", "\\n") + "\""; + } + + @Override public void close() { server.stop(0); } +} diff --git a/koog/src/test/java/com/ankurm/koog/SpringBootStarterTest.java b/koog/src/test/java/com/ankurm/koog/SpringBootStarterTest.java new file mode 100644 index 0000000..56d7bb7 --- /dev/null +++ b/koog/src/test/java/com/ankurm/koog/SpringBootStarterTest.java @@ -0,0 +1,96 @@ +package com.ankurm.koog; + +import ai.koog.prompt.executor.model.PromptExecutor; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.Test; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.boot.test.context.SpringBootTest; +import org.springframework.boot.test.web.client.TestRestTemplate; +import org.springframework.context.ApplicationContext; +import org.springframework.test.context.DynamicPropertyRegistry; +import org.springframework.test.context.DynamicPropertySource; + +import org.springframework.beans.factory.NoUniqueBeanDefinitionException; +import org.springframework.boot.builder.SpringApplicationBuilder; + +import java.io.IOException; +import java.util.Arrays; + +import static org.assertj.core.api.Assertions.assertThat; + +@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT) +class SpringBootStarterTest { + + static ScriptedOpenAIServer model; + + static { + try { model = new ScriptedOpenAIServer(); } catch (IOException e) { throw new IllegalStateException(e); } + model.thenToolCall("call_1", "error_rate", "{\"service\":\"checkout\",\"minutes\":15}") + .thenText("checkout fails 4.2% of requests over 15 minutes"); + } + + @DynamicPropertySource + static void koog(DynamicPropertyRegistry registry) { + registry.add("ai.koog.openai.api-key", () -> "test-key"); + registry.add("ai.koog.openai.base-url", model::baseUrl); + } + + @AfterAll + static void stop() { model.close(); } + + @Autowired ApplicationContext context; + @Autowired TestRestTemplate http; + + @Test + void theStarterWiresAnExecutor_andTheServiceBuildsTheAgent() { + String body = http.getForObject("/ask?q={q}", String.class, "How bad are checkout errors?"); + + var t = new Transcript("05-spring-boot-starter.txt"); + t.line("PromptExecutor beans the starter registered (only OpenAI is configured):"); + Arrays.stream(context.getBeanNamesForType(PromptExecutor.class)).sorted().forEach(n -> t.line(" " + n)); + t.line(""); + t.line("GET /ask?q=How bad are checkout errors?"); + t.line(" -> " + body); + t.line("Model endpoint saw: " + model.paths); + t.line("Tool argument the model sent: minutes=15 (an int, parsed by Koog)"); + t.line(""); + t.line("context.getBean(PromptExecutor.class) with one provider configured:"); + try { + context.getBean(PromptExecutor.class); + } catch (NoUniqueBeanDefinitionException e) { + t.line(" " + e.getClass().getSimpleName() + ": expected single matching bean but found 2: " + + String.join(",", e.getBeanNamesFound().stream().sorted().toList())); + assertThat(e.getBeanNamesFound()).containsExactlyInAnyOrder("openAIExecutor", "multiLLMPromptExecutor"); + } + try { t.write(); } catch (IOException e) { throw new IllegalStateException(e); } + + assertThat(body).isEqualTo("checkout fails 4.2% of requests over 15 minutes"); + assertThat(context.getBeanNamesForType(PromptExecutor.class)).contains("openAIExecutor"); + assertThat(model.paths).containsExactly("POST /v1/chat/completions", "POST /v1/chat/completions"); + assertThat(model.requests.get(1)).contains("\"content\":\"4.2\""); + } + + @Test + void withoutAnApiKeyThereIsNoExecutor_andTheServiceCannotStart() { + Throwable failure = null; + try (var ignored = new SpringApplicationBuilder(OpsApplication.class) + .properties("server.port=0", "ai.koog.openai.api-key=", "ai.koog.openai.enabled=true") + .run()) { + // not reached + } catch (Throwable e) { + failure = e; + } + Throwable root = failure; + while (root != null && root.getCause() != null && !(root instanceof org.springframework.beans.factory.NoSuchBeanDefinitionException)) { + root = root.getCause(); + } + var t = new Transcript("06-spring-boot-no-api-key.txt"); + t.line("ai.koog.openai.api-key is empty, ai.koog.openai.enabled=true is set:"); + t.line(" startup failed: " + (failure != null)); + t.line(" cause: " + (root == null ? "none" : root.getClass().getSimpleName())); + t.line(" message: " + (root == null ? "none" : root.getMessage())); + try { t.write(); } catch (IOException e) { throw new IllegalStateException(e); } + assertThat(failure).isNotNull(); + assertThat(root).isInstanceOf(org.springframework.beans.factory.NoSuchBeanDefinitionException.class); + } +} diff --git a/koog/src/test/java/com/ankurm/koog/StrategyGraphTest.java b/koog/src/test/java/com/ankurm/koog/StrategyGraphTest.java new file mode 100644 index 0000000..adcbfc7 --- /dev/null +++ b/koog/src/test/java/com/ankurm/koog/StrategyGraphTest.java @@ -0,0 +1,40 @@ +package com.ankurm.koog; + +import com.fasterxml.jackson.databind.JsonNode; +import org.junit.jupiter.api.Test; + +import static org.assertj.core.api.Assertions.assertThat; + +class StrategyGraphTest { + + @Test + void aJavaNodeRunsBeforeTheModel_andTheGraphRendersAsMermaid() throws Exception { + try (var model = new ScriptedOpenAIServer()) { + model.thenToolCall("call_1", "getServiceStatus", "{\"service\":\"checkout\"}") + .thenText("checkout is DEGRADED"); + var agent = OpsAgents.redactingAgent(TestExecutors.openAI(model)); + + String answer = agent.run("My key is sk-ABCDEFGH12345678. Is checkout healthy?"); + + JsonNode first = FirstAgentTest.JSON.readTree(model.requests.get(0)); + String userMessage = first.get("messages").get(1).get("content").asText(); + + var t = new Transcript("02-strategy-graph-and-redaction.txt"); + t.line("Graph, as MermaidDiagramGenerator prints it:"); + t.line(""); + for (String l : OpsAgents.diagram().split("\n")) t.line(l); + t.line(""); + t.line("The user typed: My key is sk-ABCDEFGH12345678. Is checkout healthy?"); + t.line("The model got: " + userMessage); + t.line("Answer: " + answer); + t.line("Secret present in any request body: " + + model.requests.stream().anyMatch(r -> r.contains("sk-ABCDEFGH12345678"))); + t.write(); + + assertThat(userMessage).isEqualTo("My key is [REDACTED]. Is checkout healthy?"); + assertThat(model.requests).noneMatch(r -> r.contains("sk-ABCDEFGH12345678")); + assertThat(answer).isEqualTo("checkout is DEGRADED"); + assertThat(OpsAgents.diagram()).contains("[*] --> redact", "redact --> callLLM"); + } + } +} diff --git a/koog/src/test/java/com/ankurm/koog/TestExecutors.java b/koog/src/test/java/com/ankurm/koog/TestExecutors.java new file mode 100644 index 0000000..be2e930 --- /dev/null +++ b/koog/src/test/java/com/ankurm/koog/TestExecutors.java @@ -0,0 +1,17 @@ +package com.ankurm.koog; + +import ai.koog.prompt.executor.clients.ConnectionTimeoutConfig; +import ai.koog.prompt.executor.clients.openai.OpenAIClientSettings; +import ai.koog.prompt.executor.model.PromptExecutor; + +/** A PromptExecutor whose OpenAI client talks to the scripted server instead of api.openai.com. */ +final class TestExecutors { + private TestExecutors() {} + + static PromptExecutor openAI(ScriptedOpenAIServer model) { + var settings = new OpenAIClientSettings( + model.baseUrl(), new ConnectionTimeoutConfig(), + "v1/chat/completions", "v1/responses", "v1/embeddings", "v1/moderations", "v1/models"); + return PromptExecutor.builder().openAI("test-key", settings).build(); + } +} diff --git a/koog/src/test/java/com/ankurm/koog/Transcript.java b/koog/src/test/java/com/ankurm/koog/Transcript.java new file mode 100644 index 0000000..6b21c94 --- /dev/null +++ b/koog/src/test/java/com/ankurm/koog/Transcript.java @@ -0,0 +1,23 @@ +package com.ankurm.koog; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; + +/** Collects lines and writes them to output/NN-name.txt, so the test IS the transcript. */ +final class Transcript { + private final List lines = new ArrayList<>(); + private final String file; + + Transcript(String file) { this.file = file; } + + void line(String s) { lines.add(s); } + + void write() throws IOException { + Path dir = Path.of("output"); + Files.createDirectories(dir); + Files.write(dir.resolve(file), lines); + } +} diff --git a/koog/src/test/java/com/ankurm/koog/TrapMain.java b/koog/src/test/java/com/ankurm/koog/TrapMain.java new file mode 100644 index 0000000..cc8984e --- /dev/null +++ b/koog/src/test/java/com/ankurm/koog/TrapMain.java @@ -0,0 +1,11 @@ +package com.ankurm.koog; + +import ai.koog.prompt.executor.model.PromptExecutor; + +/** Only used by scripts/kotlin-bom-trap.sh: the first Koog call that touches kotlin.time. */ +public class TrapMain { + public static void main(String[] args) { + PromptExecutor.builder().openAI("k").build(); + System.out.println("no error"); + } +} diff --git a/koog/src/test/resources/logback-test.xml b/koog/src/test/resources/logback-test.xml new file mode 100644 index 0000000..ce46f3c --- /dev/null +++ b/koog/src/test/resources/logback-test.xml @@ -0,0 +1,9 @@ + + + + %d{HH:mm:ss.SSS} %-5level %logger{20} - %msg%n + + + + diff --git a/pom.xml b/pom.xml index dddd4c5..65c9933 100644 --- a/pom.xml +++ b/pom.xml @@ -19,6 +19,7 @@ code-review-bot agentic-patterns mcp-sdk-plain + koog