diff --git a/README.md b/README.md index 39e9fb3..e3d9b6d 100644 --- a/README.md +++ b/README.md @@ -24,5 +24,6 @@ Runnable companion code for the Spring AI articles on [ankurm.com](https://ankur | [`llm-gateway/`](llm-gateway) | A gateway service on Spring AI 2.0: model-hint routing, failover under the tool-calling advisor (a provider failing mid tool loop does not re-run the tool), one circuit breaker per provider, dollar caps that reserve before the call, a token-per-hour limiter, and a tenant-keyed semantic cache. Real OpenAI and Anthropic models against a local fake; no vendor API called. Spring Boot 4.1.1, Spring AI 2.0.1, Java 25. | [The LLM Gateway Pattern for Java Microservices](https://ankurm.com/llm-gateway-pattern-java-microservices/) | | [`semantic-cache/`](semantic-cache) | A `CallAdvisor` that serves a stored answer when a new question embeds close to an old one, over a Redis Stack vector index and a real in-process embedding model (all-MiniLM-L6-v2). A labelled set of 30 intents x 4 phrasings plus 20 off-topic questions swept across thresholds (hit rate against wrong answers), 600 requests replayed with an illustrative cost model, and the traps reproduced: Spring AI's score is `(1 + cosine) / 2`, reset-password and reset-2FA are close enough to answer for each other, a tenant filter, expiry, and an index built for another embedding model. The chat model is a script that counts calls; one test uses a real local model. Spring Boot 4.1.1, Spring AI 2.0.1, Java 25. | [Semantic Caching for LLM Calls in Spring Boot with Redis Vector Search](https://ankurm.com/semantic-caching-for-llm-calls-in-spring-boot-with-redis-vector-search/) | | [`langchain4j-interop/`](langchain4j-interop) | The same order assistant in Spring AI 2.0.1 and LangChain4j 1.22.0 in one Spring Boot 4.1.1 app, against a local fake OpenAI server: the bean collision when both OpenAI starters are present, the two requests compared on the wire, a throwing tool and a looping model (what each framework bounds), the meters each records, one MCP tool server used from both, and which lines of the 2025 version of the article still compile. Java 25. | [Spring AI vs LangChain4j in 2026: Is the Pairing Still Worth It?](https://ankurm.com/spring-ai-langchain4j-production-ai-microservices-java/) | +| [`agent-skills/`](agent-skills) | `SKILL.md` folders loaded on demand by the `Skill` tool from `spring-ai-agent-utils` 0.13.0: what the model sees before it loads anything (name and description only), the two-call load round trip, unknown, duplicate and description-less skills, and listing size against inlining every body (character counts). Scripted chat model, no live model. Spring Boot 4.1.1, Spring AI 2.0.1, Java 25. | [Agent Skills in Spring AI: Packaging Reusable Capabilities for Your Agents](https://ankurm.com/agent-skills-spring-ai-packaging-reusable-capabilities/) | Upgrading from Spring AI 1.x: [migration guide](https://ankurm.com/spring-ai-1-to-2-migration-guide/). diff --git a/agent-skills/.gitignore b/agent-skills/.gitignore new file mode 100644 index 0000000..2f7896d --- /dev/null +++ b/agent-skills/.gitignore @@ -0,0 +1 @@ +target/ diff --git a/agent-skills/README.md b/agent-skills/README.md new file mode 100644 index 0000000..a50cc5f --- /dev/null +++ b/agent-skills/README.md @@ -0,0 +1,48 @@ +# agent-skills + +Packaging reusable capabilities for a Spring AI agent as `SKILL.md` folders, loaded on demand through the `Skill` tool from `org.springaicommunity:spring-ai-agent-utils`. + +Article: [Agent Skills in Spring AI: Packaging Reusable Capabilities for Your Agents](https://ankurm.com/agent-skills-spring-ai-packaging-reusable-capabilities/). The whole walkthrough lives in the article; this directory is only the code and the captured output. + +## Versions + +| Component | Version | +|---|---| +| Spring Boot | 4.1.1 | +| Spring AI | 2.0.1 | +| spring-ai-agent-utils | 0.13.0 | +| Java | 25 | + +## Run it + +``` +./scripts/run-all.sh +``` + +Every test drives a hand-written scripted chat model, so no API key and no network are needed. + +## What is in here + +| Path | What it is | +|---|---| +| `src/main/java/com/ankurm/agentskills/SkillClients.java` | Builds the `SkillsTool` from classpath or directory skills and a `ChatClient` with `ToolCallingAdvisor` | +| `src/main/resources/skills/release-notes` | A skill with a `references/` file | +| `src/main/resources/skills/sql-review` | A skill with a `scripts/` file and an `allowed-tools` key | +| `src/main/resources/skills/incident-summary` | A minimal skill | +| `src/test/java/.../SkillListingTest` | What the model sees before it loads anything | +| `src/test/java/.../SkillLoadRoundTripTest` | The two model calls for one skill | +| `src/test/java/.../SkillLoadingEdgeCasesTest` | Unknown name, duplicate name, no description, `allowed-tools` | +| `src/test/java/.../ListingCostTest` | Tool-description size against inlining every body, 3 to 100 skills | + +## Captured output + +| File | Shows | +|---|---| +| `output/01-skill-listing.txt` | The `Skill` tool name, schema and `` block | +| `output/02-skill-load-round-trip.txt` | The tool result fed back to the model | +| `output/03-loading-edge-cases.txt` | Unknown, duplicate and description-less skills | +| `output/04-listing-cost.txt` | Character counts (not tokens) as skills are added | + +## Not measured + +No real model was called. Whether a given model chooses the right skill is not tested here. diff --git a/agent-skills/output/01-skill-listing.txt b/agent-skills/output/01-skill-listing.txt new file mode 100644 index 0000000..fb3e7ff --- /dev/null +++ b/agent-skills/output/01-skill-listing.txt @@ -0,0 +1,38 @@ +# What the model sees before it loads any skill + +tool name : Skill +input schema : { + "$schema" : "https://json-schema.org/draft/2020-12/schema", + "type" : "object", + "properties" : { + "command" : { + "type" : "string", + "description" : "The skill name (no arguments). E.g., \"pdf\" or \"xlsx\"" + } + }, + "required" : [ "command" ], + "additionalProperties" : false +} +description size : 1602 characters + +--- the block of the tool description --- + + + incident-summary + Summarise an incident timeline into impact, cause, and follow-ups for a status page or postmortem. + + + release-notes + Write release notes for a software version from a list of merged changes. Use when asked to draft, tidy or summarise release notes or a changelog entry. + + + sql-review + Review a SQL migration for locking, data-loss and rollback problems. Use when asked to check, review or approve a database migration script. + Bash + + + + +contains 'release-notes' name : true +contains a body line ('Breaking') : false +contains the reference file name : false diff --git a/agent-skills/output/02-skill-load-round-trip.txt b/agent-skills/output/02-skill-load-round-trip.txt new file mode 100644 index 0000000..5b538b5 --- /dev/null +++ b/agent-skills/output/02-skill-load-round-trip.txt @@ -0,0 +1,22 @@ +# Loading one skill: two model calls + +model calls: 2 +first call mentions the skill body ('Breaking' in any message) : false + +--- tool result fed back to the model (what the Skill tool returned) --- +Base directory for this skill: /target/classes/skills/release-notes + +# Release notes + +Turn a list of merged changes into release notes a user can act on. + +1. Group the changes under **Breaking**, **Added**, **Fixed** and **Internal**, in that order. +2. Lead every **Breaking** item with what the reader must change, not with what the team did. +3. Read `references/style-guide.md` (in this skill's directory) before writing the final text. +4. Never invent a ticket number. If a change has none, leave the reference out. + +Return the notes as Markdown with one bullet per change. + +--- final answer --- +## Breaking +- Rename `cache.ttl` to `cache.expiry`. diff --git a/agent-skills/output/03-loading-edge-cases.txt b/agent-skills/output/03-loading-edge-cases.txt new file mode 100644 index 0000000..aea1ac1 --- /dev/null +++ b/agent-skills/output/03-loading-edge-cases.txt @@ -0,0 +1,16 @@ +# Unknown, duplicate and description-less skills + +== unknown skill name +call Skill(command="does-not-exist") returns: "Skill not found: does-not-exist" + +== the same name in two directories (classpath skills first, then the copy) +body served for release-notes contains the first copy ('Breaking') : true +body served for release-notes contains the second copy : false + +== a SKILL.md with no description +loaded: name=no-description frontMatterKeys=[name] +loaded: name=release-notes frontMatterKeys=[name, description] +'no-description' appears in the tool description : true + +== the allowed-tools key in sql-review's frontmatter +echoed into the listing as : true diff --git a/agent-skills/output/04-listing-cost.txt b/agent-skills/output/04-listing-cost.txt new file mode 100644 index 0000000..cbfe4d4 --- /dev/null +++ b/agent-skills/output/04-listing-cost.txt @@ -0,0 +1,10 @@ +# Characters in the tool description as skills are added + +skills tool description chars bodies, if all inlined ratio +3 1399 24000 17.2x smaller +10 2456 80000 32.6x smaller +25 4766 200000 42.0x smaller +50 8616 400000 46.4x smaller +100 16316 800000 49.0x smaller + +each skill body used here is 8000 characters; each description about 90. diff --git a/agent-skills/pom.xml b/agent-skills/pom.xml new file mode 100644 index 0000000..89b92de --- /dev/null +++ b/agent-skills/pom.xml @@ -0,0 +1,68 @@ + + + 4.0.0 + + + org.springframework.boot + spring-boot-starter-parent + 4.1.1 + + + + com.ankurm + agent-skills + 1.0.0 + agent-skills + Agent Skills with Spring AI: SKILL.md folders loaded by SkillsTool from spring-ai-agent-utils, what the model sees up front, what it sees after loading one, and what goes wrong. + + + 25 + + 2.0.1 + + 0.13.0 + + + + + + org.springframework.ai + spring-ai-bom + ${spring-ai.version} + pom + import + + + + + + + org.springframework.ai + spring-ai-client-chat + + + org.springaicommunity + spring-ai-agent-utils + ${agent-utils.version} + + + org.springframework.boot + spring-boot-starter-test + test + + + + + + + org.apache.maven.plugins + maven-surefire-plugin + + -Duser.timezone=UTC -Dstdout.encoding=UTF-8 -Dfile.encoding=UTF-8 + + + + + diff --git a/agent-skills/scripts/run-all.sh b/agent-skills/scripts/run-all.sh new file mode 100755 index 0000000..8461cb3 --- /dev/null +++ b/agent-skills/scripts/run-all.sh @@ -0,0 +1,7 @@ +#!/usr/bin/env bash +# Regenerates every file under output/ -- the test suite writes each one itself via the +# Transcript helper, overwriting it in place. No file in this module's output/ is hand-captured. +set -euo pipefail +cd "$(dirname "$0")/.." +rm -rf target +mvn -q test diff --git a/agent-skills/src/main/java/com/ankurm/agentskills/SkillClients.java b/agent-skills/src/main/java/com/ankurm/agentskills/SkillClients.java new file mode 100644 index 0000000..00a2769 --- /dev/null +++ b/agent-skills/src/main/java/com/ankurm/agentskills/SkillClients.java @@ -0,0 +1,44 @@ +package com.ankurm.agentskills; + +import java.util.List; + +import org.springaicommunity.agent.tools.SkillsTool; + +import org.springframework.ai.chat.client.ChatClient; +import org.springframework.ai.chat.client.advisor.ToolCallingAdvisor; +import org.springframework.ai.chat.model.ChatModel; +import org.springframework.ai.model.tool.ToolCallingChatOptions; +import org.springframework.ai.tool.ToolCallback; +import org.springframework.core.io.ClassPathResource; +import org.springframework.core.io.Resource; + +/** + * The wiring from the article, in one place so the tests exercise exactly what an application would run. + * Skills ship inside the jar, so they are loaded from the classpath rather than from a directory on disk. + */ +public final class SkillClients { + + private SkillClients() { + } + + /** The one tool the model gets for skills: {@code Skill}, with every skill's name and description in its description. */ + public static ToolCallback skillsTool(Resource... skillRoots) { + SkillsTool.Builder builder = SkillsTool.builder(); + for (Resource root : skillRoots) { + builder.addSkillsResource(root); + } + return builder.build(); + } + + public static ToolCallback bundledSkills() { + return skillsTool(new ClassPathResource("skills")); + } + + public static ChatClient client(ChatModel model, List tools) { + return ChatClient.builder(model) + .defaultOptions(ToolCallingChatOptions.builder()) + .defaultToolCallbacks(tools) + .defaultAdvisors(ToolCallingAdvisor.builder().build()) + .build(); + } +} diff --git a/agent-skills/src/main/resources/skills/incident-summary/SKILL.md b/agent-skills/src/main/resources/skills/incident-summary/SKILL.md new file mode 100644 index 0000000..68b62fa --- /dev/null +++ b/agent-skills/src/main/resources/skills/incident-summary/SKILL.md @@ -0,0 +1,9 @@ +--- +name: incident-summary +description: Summarise an incident timeline into impact, cause, and follow-ups for a status page or postmortem. +--- + +# Incident summary + +State impact first (who, how long), then the proximate cause, then the follow-ups with owners. +Keep the timeline to the five events that changed what responders did. diff --git a/agent-skills/src/main/resources/skills/release-notes/SKILL.md b/agent-skills/src/main/resources/skills/release-notes/SKILL.md new file mode 100644 index 0000000..36ab5a2 --- /dev/null +++ b/agent-skills/src/main/resources/skills/release-notes/SKILL.md @@ -0,0 +1,15 @@ +--- +name: release-notes +description: Write release notes for a software version from a list of merged changes. Use when asked to draft, tidy or summarise release notes or a changelog entry. +--- + +# Release notes + +Turn a list of merged changes into release notes a user can act on. + +1. Group the changes under **Breaking**, **Added**, **Fixed** and **Internal**, in that order. +2. Lead every **Breaking** item with what the reader must change, not with what the team did. +3. Read `references/style-guide.md` (in this skill's directory) before writing the final text. +4. Never invent a ticket number. If a change has none, leave the reference out. + +Return the notes as Markdown with one bullet per change. diff --git a/agent-skills/src/main/resources/skills/release-notes/references/style-guide.md b/agent-skills/src/main/resources/skills/release-notes/references/style-guide.md new file mode 100644 index 0000000..f1349e5 --- /dev/null +++ b/agent-skills/src/main/resources/skills/release-notes/references/style-guide.md @@ -0,0 +1,5 @@ +# Release notes style guide + +- Present tense, active voice: "Fixes a crash when the cache is empty", not "A crash was fixed". +- One sentence per bullet. A second sentence belongs in the migration guide, not the notes. +- Name the setting, class or endpoint in backticks so it can be searched for. diff --git a/agent-skills/src/main/resources/skills/sql-review/SKILL.md b/agent-skills/src/main/resources/skills/sql-review/SKILL.md new file mode 100644 index 0000000..a95c95b --- /dev/null +++ b/agent-skills/src/main/resources/skills/sql-review/SKILL.md @@ -0,0 +1,12 @@ +--- +name: sql-review +description: Review a SQL migration for locking, data-loss and rollback problems. Use when asked to check, review or approve a database migration script. +allowed-tools: Bash +--- + +# SQL migration review + +1. Run `scripts/lint.sh ` from this skill's directory and quote its output. +2. Flag any `DROP`, `TRUNCATE` or `ALTER ... TYPE` as data-loss risk. +3. Flag any `CREATE INDEX` without `CONCURRENTLY` as a table lock on PostgreSQL. +4. State whether a rollback script exists. If it does not, say so first. diff --git a/agent-skills/src/main/resources/skills/sql-review/scripts/lint.sh b/agent-skills/src/main/resources/skills/sql-review/scripts/lint.sh new file mode 100755 index 0000000..2ec5594 --- /dev/null +++ b/agent-skills/src/main/resources/skills/sql-review/scripts/lint.sh @@ -0,0 +1,5 @@ +#!/usr/bin/env bash +# A deliberately tiny linter: it only counts the statements the review skill cares about. +file="${1:?usage: lint.sh }" +echo "drop/truncate : $(grep -ciE '^\s*(drop|truncate)\b' "$file")" +echo "index w/o CONCURRENTLY : $(grep -iE 'create (unique )?index' "$file" | grep -vciE 'concurrently')" diff --git a/agent-skills/src/test/java/com/ankurm/agentskills/ListingCostTest.java b/agent-skills/src/test/java/com/ankurm/agentskills/ListingCostTest.java new file mode 100644 index 0000000..a27520b --- /dev/null +++ b/agent-skills/src/test/java/com/ankurm/agentskills/ListingCostTest.java @@ -0,0 +1,40 @@ +package com.ankurm.agentskills; + +import java.util.stream.IntStream; + +import com.ankurm.agentskills.support.Transcript; +import org.junit.jupiter.api.Test; +import org.springaicommunity.agent.tools.SkillsTool; +import org.springframework.ai.tool.ToolCallback; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * The reason skills exist: the listing grows with the number of skills, not with their size. + * Sizes are characters; tokens are roughly characters / 4 for English prose, which is only an estimate. + */ +class ListingCostTest { + + private static final String BODY = "Step one. Step two. Step three. ".repeat(250); // ~8,000 characters of instructions + + @Test + void listingCostFollowsDescriptionsNotBodies() { + try (Transcript t = new Transcript("04-listing-cost.txt", "Characters in the tool description as skills are added")) { + t.line("%-8s %-22s %-24s %s", "skills", "tool description chars", "bodies, if all inlined", "ratio"); + for (int n : new int[] { 3, 10, 25, 50, 100 }) { + SkillsTool.Builder b = SkillsTool.builder(); + IntStream.range(0, n).forEach(i -> b.addSkill("skill-" + i, + "Handle the recurring task number " + i + " in the usual way. Use when asked about task " + i + ".", BODY)); + ToolCallback tool = b.build(); + int listing = tool.getToolDefinition().description().length(); + int bodies = n * BODY.length(); + t.line("%-8d %-22d %-24d %.1fx smaller", n, listing, bodies, (double) bodies / listing); + if (n == 100) { + assertThat(listing).isLessThan(bodies / 10); + } + } + t.blank(); + t.line("each skill body used here is %d characters; each description about 90.", BODY.length()); + } + } +} diff --git a/agent-skills/src/test/java/com/ankurm/agentskills/SkillListingTest.java b/agent-skills/src/test/java/com/ankurm/agentskills/SkillListingTest.java new file mode 100644 index 0000000..d6aeda5 --- /dev/null +++ b/agent-skills/src/test/java/com/ankurm/agentskills/SkillListingTest.java @@ -0,0 +1,34 @@ +package com.ankurm.agentskills; + +import com.ankurm.agentskills.support.Transcript; +import org.junit.jupiter.api.Test; +import org.springframework.ai.tool.ToolCallback; + +import static org.assertj.core.api.Assertions.assertThat; + +/** What the model is given before it has chosen any skill: one tool whose description lists every skill. */ +class SkillListingTest { + + @Test + void theModelSeesNamesAndDescriptionsButNoBodies() { + ToolCallback tool = SkillClients.bundledSkills(); + String description = tool.getToolDefinition().description(); + + try (Transcript t = new Transcript("01-skill-listing.txt", "What the model sees before it loads any skill")) { + t.line("tool name : %s", tool.getToolDefinition().name()); + t.line("input schema : %s", tool.getToolDefinition().inputSchema()); + t.line("description size : %d characters", description.length()); + t.blank(); + t.line("--- the block of the tool description ---"); + String block = description.substring(description.lastIndexOf("")); + t.line("%s", block); + t.blank(); + t.line("contains 'release-notes' name : %s", description.contains("release-notes")); + t.line("contains a body line ('Breaking') : %s", description.contains("Breaking")); + t.line("contains the reference file name : %s", description.contains("style-guide")); + } + + assertThat(description).contains("release-notes", "sql-review", "incident-summary"); + assertThat(description).doesNotContain("Breaking").doesNotContain("style-guide"); + } +} diff --git a/agent-skills/src/test/java/com/ankurm/agentskills/SkillLoadRoundTripTest.java b/agent-skills/src/test/java/com/ankurm/agentskills/SkillLoadRoundTripTest.java new file mode 100644 index 0000000..91c17cf --- /dev/null +++ b/agent-skills/src/test/java/com/ankurm/agentskills/SkillLoadRoundTripTest.java @@ -0,0 +1,63 @@ +package com.ankurm.agentskills; + +import java.util.List; + +import com.ankurm.agentskills.support.ScriptedChatModel; +import com.ankurm.agentskills.support.Transcript; +import org.junit.jupiter.api.Test; +import org.springframework.ai.chat.client.ChatClient; +import org.springframework.ai.chat.messages.AssistantMessage; +import org.springframework.ai.chat.messages.ToolResponseMessage; +import org.springframework.ai.chat.prompt.Prompt; + +import static org.assertj.core.api.Assertions.assertThat; + +/** The model asks for one skill by name; the full SKILL.md comes back as the tool result, in the second model call. */ +class SkillLoadRoundTripTest { + + @Test + void loadingASkillPutsItsBodyInTheSecondCall() { + ScriptedChatModel model = ScriptedChatModel.builder() + .thenCallTools(new AssistantMessage.ToolCall("call-1", "function", "Skill", "{\"command\":\"release-notes\"}")) + .thenRespond("## Breaking\n- Rename `cache.ttl` to `cache.expiry`.") + .build(); + ChatClient client = SkillClients.client(model, List.of(SkillClients.bundledSkills())); + + String answer = client.prompt().user("Draft release notes for 2.4.0 from these changes: ...").call().content(); + + Prompt first = model.capturedPrompts().get(0); + Prompt second = model.capturedPrompts().get(1); + String toolResult = second.getInstructions().stream() + .filter(ToolResponseMessage.class::isInstance) + .map(ToolResponseMessage.class::cast) + .flatMap(m -> m.getResponses().stream()) + .map(ToolResponseMessage.ToolResponse::responseData) + .findFirst() + .orElseThrow(); + + try (Transcript t = new Transcript("02-skill-load-round-trip.txt", "Loading one skill: two model calls")) { + t.line("model calls: %d", model.callCount()); + t.line("first call mentions the skill body ('Breaking' in any message) : %s", + first.getInstructions().stream().anyMatch(m -> m.getText() != null && m.getText().contains("Breaking"))); + t.blank(); + t.line("--- tool result fed back to the model (what the Skill tool returned) ---"); + t.line("%s", readable(toolResult)); + t.blank(); + t.line("--- final answer ---"); + t.line("%s", answer); + } + + assertThat(model.callCount()).isEqualTo(2); + assertThat(toolResult).contains("Base directory").contains("Breaking").contains("references/style-guide.md"); + } + + /** The tool result is a JSON string literal; show it as the model reads it, minus this machine's absolute path. */ + private static String readable(String jsonString) { + String s = jsonString.strip(); + if (s.startsWith("\"") && s.endsWith("\"")) { + s = s.substring(1, s.length() - 1); + } + s = s.replace("\\n", "\n").replace("\\\"", "\"").replace("\\\\", "\\"); + return s.replaceAll("(?m)^(Base directory for this skill: ).*?(/target/classes/)", "$1$2"); + } +} diff --git a/agent-skills/src/test/java/com/ankurm/agentskills/SkillLoadingEdgeCasesTest.java b/agent-skills/src/test/java/com/ankurm/agentskills/SkillLoadingEdgeCasesTest.java new file mode 100644 index 0000000..c104abf --- /dev/null +++ b/agent-skills/src/test/java/com/ankurm/agentskills/SkillLoadingEdgeCasesTest.java @@ -0,0 +1,64 @@ +package com.ankurm.agentskills; + +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; + +import com.ankurm.agentskills.support.Transcript; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import org.springaicommunity.agent.tools.SkillsTool; +import org.springaicommunity.agent.utils.Skills; +import org.springframework.ai.tool.ToolCallback; + +/** What the loader does with an unknown name, a duplicate name and a SKILL.md with no description. */ +class SkillLoadingEdgeCasesTest { + + @TempDir + Path dir; + + @Test + void recordWhatActuallyHappens() throws Exception { + Path dup = Files.createDirectories(dir.resolve("copy/release-notes")); + Files.writeString(dup.resolve("SKILL.md"), + "---\nname: release-notes\ndescription: A second skill that reuses an existing name.\n---\n\nSECOND COPY BODY\n"); + Path nodesc = Files.createDirectories(dir.resolve("copy/no-description")); + Files.writeString(nodesc.resolve("SKILL.md"), "---\nname: no-description\n---\n\nBODY WITHOUT A DESCRIPTION\n"); + + try (Transcript t = new Transcript("03-loading-edge-cases.txt", "Unknown, duplicate and description-less skills")) { + ToolCallback bundled = SkillClients.bundledSkills(); + + t.line("== unknown skill name"); + t.line("call Skill(command=\"does-not-exist\") returns: %s", oneLine(call(bundled, "does-not-exist"))); + + t.blank(); + t.line("== the same name in two directories (classpath skills first, then the copy)"); + ToolCallback both = SkillsTool.builder() + .addSkillsResource(new org.springframework.core.io.ClassPathResource("skills")) + .addSkillsDirectory(dir.resolve("copy").toString()) + .build(); + String loaded = call(both, "release-notes"); + t.line("body served for release-notes contains the first copy ('Breaking') : %s", loaded.contains("Breaking")); + t.line("body served for release-notes contains the second copy : %s", loaded.contains("SECOND COPY BODY")); + + t.blank(); + t.line("== a SKILL.md with no description"); + List skills = Skills.loadDirectory(dir.resolve("copy").toString()); + skills.forEach(s -> t.line("loaded: name=%s frontMatterKeys=%s", s.name(), s.frontMatter().keySet())); + String listing = both.getToolDefinition().description(); + t.line("'no-description' appears in the tool description : %s", listing.contains("no-description")); + + t.blank(); + t.line("== the allowed-tools key in sql-review's frontmatter"); + t.line("echoed into the listing as : %s", bundled.getToolDefinition().description().contains("Bash")); + } + } + + private static String call(ToolCallback tool, String command) { + return tool.call("{\"command\":\"" + command + "\"}"); + } + + private static String oneLine(String s) { + return s.replace("\n", "\\n"); + } +} diff --git a/agent-skills/src/test/java/com/ankurm/agentskills/support/ScriptedChatModel.java b/agent-skills/src/test/java/com/ankurm/agentskills/support/ScriptedChatModel.java new file mode 100644 index 0000000..5b52ae7 --- /dev/null +++ b/agent-skills/src/test/java/com/ankurm/agentskills/support/ScriptedChatModel.java @@ -0,0 +1,116 @@ +package com.ankurm.agentskills.support; + +import org.springframework.ai.chat.messages.AssistantMessage; +import org.springframework.ai.chat.model.ChatModel; +import org.springframework.ai.chat.model.ChatResponse; +import org.springframework.ai.chat.model.Generation; +import org.springframework.ai.chat.prompt.ChatOptions; +import org.springframework.ai.chat.prompt.Prompt; +import org.springframework.ai.model.tool.ToolCallingChatOptions; + +import java.util.ArrayDeque; +import java.util.ArrayList; +import java.util.Deque; +import java.util.List; +import java.util.Queue; +import java.util.concurrent.CopyOnWriteArrayList; + +/** + * A hand-written {@link ChatModel} that returns a pre-programmed queue of {@link ChatResponse}s, + * one per {@link #call(Prompt)} invocation, instead of calling a real LLM API. {@code ChatModel} + * has exactly one abstract method -- confirmed with {@code javap} against + * {@code spring-ai-model-2.0.1.jar}, everything else on the interface has a default implementation + * -- so this is the entire surface a deterministic test double needs to implement. + * + *

ToolCallingAdvisor drive the tool-calling loop by calling the + * underlying {@link ChatModel} once per round: once to get the model's first response (which may + * contain tool calls), then once more per round of tool results fed back in, until a response + * comes back with no tool calls. Queuing responses here lets a test assert the exact shape of that + * loop -- how many rounds it took, what tool calls appeared, what the final answer was -- without + * an API key, network access, or the nondeterminism of an actual model. + * + *

Every {@link Prompt} the advisor sends is recorded in {@link #capturedPrompts()} so a test can + * also assert on what the advisor sent back on the next round -- in particular, that a + * {@code ToolResponseMessage} was appended after tool execution. + */ +public class ScriptedChatModel implements ChatModel { + + private final Queue script; + private final List capturedPrompts = new CopyOnWriteArrayList<>(); + + private ScriptedChatModel(Deque script) { + this.script = script; + } + + public static Builder builder() { + return new Builder(); + } + + @Override + public ChatResponse call(Prompt prompt) { + this.capturedPrompts.add(prompt); + ChatResponse next = this.script.poll(); + if (next == null) { + throw new IllegalStateException( + "ScriptedChatModel ran out of queued responses after " + this.capturedPrompts.size() + + " calls. Prompts so far: " + this.capturedPrompts); + } + return next; + } + + public List capturedPrompts() { + return List.copyOf(this.capturedPrompts); + } + + /** + * {@code DefaultChatClientUtils} builds every outgoing {@link Prompt}'s options from + * {@code chatModel.getOptions().mutate()} -- not {@code getDefaultOptions()}, which is a + * separate default method nobody in the request-building path actually calls. Confirmed by + * disassembling both: {@link ChatModel#getOptions()}'s default body is a bare + * {@code ChatOptions.builder().build()}, a plain {@link ChatOptions} that is not a + * {@link ToolCallingChatOptions}. Since {@code ToolCallingAdvisor.adviseCall} starts with an + * {@code instanceof ToolCallingChatOptions} check on that exact object and falls straight + * through to the underlying model with no tool loop at all when it fails, leaving this method's + * default in place silently turns every tool call in this repository into a no-op -- confirmed + * the hard way, by a first version of this class that overrode {@code getDefaultOptions()} + * instead and watched every test below get back an empty answer after exactly one model call. + * Real providers (OpenAI, Anthropic) return their own {@code ToolCallingChatOptions} + * implementation from {@code getOptions()} for the same reason. + */ + @Override + public ChatOptions getOptions() { + return ToolCallingChatOptions.builder().build(); + } + + public int callCount() { + return this.capturedPrompts.size(); + } + + public static final class Builder { + + private final Deque script = new ArrayDeque<>(); + + private Builder() { + } + + /** Queues a plain-text final answer with no tool calls -- ends the tool-calling loop. */ + public Builder thenRespond(String text) { + this.script.add(new ChatResponse(List.of(new Generation(new AssistantMessage(text))))); + return this; + } + + /** Queues an assistant turn that calls one or more tools, continuing the loop. */ + public Builder thenCallTools(AssistantMessage.ToolCall... toolCalls) { + AssistantMessage message = AssistantMessage.builder() + .content("") + .toolCalls(List.of(toolCalls)) + .build(); + this.script.add(new ChatResponse(List.of(new Generation(message)))); + return this; + } + + public ScriptedChatModel build() { + return new ScriptedChatModel(new ArrayDeque<>(this.script)); + } + } +} diff --git a/agent-skills/src/test/java/com/ankurm/agentskills/support/Transcript.java b/agent-skills/src/test/java/com/ankurm/agentskills/support/Transcript.java new file mode 100644 index 0000000..49e471a --- /dev/null +++ b/agent-skills/src/test/java/com/ankurm/agentskills/support/Transcript.java @@ -0,0 +1,47 @@ +package com.ankurm.agentskills.support; + +import java.io.IOException; +import java.io.PrintWriter; +import java.io.StringWriter; +import java.nio.file.Files; +import java.nio.file.Path; + +/** + * Writes a numbered transcript under {@code output/} and echoes it to the console. Every console + * block quoted in the article comes out of one of these files verbatim. + */ +public final class Transcript implements AutoCloseable { + + private final Path path; + private final StringWriter buffer = new StringWriter(); + private final PrintWriter out = new PrintWriter(buffer); + + public Transcript(String fileName, String title) { + this.path = Path.of("output", fileName); + out.println("# " + title); + out.println(); + } + + public Transcript line(String format, Object... args) { + out.println(args.length == 0 ? format : String.format(format, args)); + return this; + } + + public Transcript blank() { + out.println(); + return this; + } + + @Override + public void close() { + out.flush(); + try { + Files.createDirectories(path.getParent()); + Files.writeString(path, buffer.toString()); + } + catch (IOException e) { + throw new IllegalStateException("could not write " + path, e); + } + System.out.print(buffer); + } +}