diff --git a/README.md b/README.md index 503bcde..c2d1dc4 100644 --- a/README.md +++ b/README.md @@ -9,5 +9,6 @@ Runnable companion code for the Spring AI articles on [ankurm.com](https://ankur | [`mcp-server/`](mcp-server) | An order-lookup service exposed as MCP tools, a resource, and a prompt with `@McpTool`/`@McpResource`/`@McpPrompt`, served over Streamable HTTP (Spring AI 2.0's default MCP server transport). Spring Boot 4.1.1, Spring AI 2.0.1, Java 25. | [Build an MCP Server with Spring AI 2.0](https://ankurm.com/spring-ai-2-0-mcp-server-streamable-http/) | | [`mcp-client/`](mcp-client) | `ChatClient` calling tools from two real external MCP servers (filesystem, git) over stdio via `defaultToolCallbacks(ToolCallbackProvider...)`, contrasted with a local `@Tool` method, with every call logged through one Micrometer `ObservationHandler`. Spring Boot 4.1.1, Spring AI 2.0.1, Java 25. | [Spring AI MCP Client: Calling External MCP Servers from ChatClient](https://ankurm.com/spring-ai-2-0-mcp-client/) | | [`mcp-secure/`](mcp-secure) | The mcp-server article's order-lookup tools behind a real OAuth2 resource server: JWT validation, one scope per tool via `@PreAuthorize`, unauthenticated tool discovery rejected outright, and every call audit-logged through MDC -- denials included. Spring Boot 4.1.1, Spring AI 2.0.1, Spring Security 7.1.1, Java 25. | [Securing an MCP Server with Spring Security 7](https://ankurm.com/spring-ai-2-0-mcp-server-security/) | +| [`tool-calling/`](tool-calling) | `@Tool` methods, `ToolCallingAdvisor` (the advisor-layer replacement for Spring AI 1.x's per-model tool loop), `returnDirect`, `ToolContext`, and `ToolSearchToolCallingAdvisor` for progressive disclosure across a 230-tool synthetic library -- every test driven by a hand-written `ScriptedChatModel`, no live model anywhere. Spring Boot 4.1.1, Spring AI 2.0.1, Java 25. | [Tool Calling in Spring AI 2.0](https://ankurm.com/spring-ai-2-0-tool-calling/) | Upgrading from Spring AI 1.x: [migration guide](https://ankurm.com/spring-ai-1-to-2-migration-guide/). diff --git a/tool-calling/.gitignore b/tool-calling/.gitignore new file mode 100644 index 0000000..2f7896d --- /dev/null +++ b/tool-calling/.gitignore @@ -0,0 +1 @@ +target/ diff --git a/tool-calling/README.md b/tool-calling/README.md new file mode 100644 index 0000000..b1769b4 --- /dev/null +++ b/tool-calling/README.md @@ -0,0 +1,59 @@ +# tool-calling + +`@Tool` methods, `ToolCallingAdvisor` (the advisor-layer replacement for Spring AI 1.x's +per-`ChatModel` tool-calling loop), `returnDirect`, `ToolContext`, and `ToolSearchToolCallingAdvisor` +for progressive disclosure across a 230-tool synthetic library -- with no live model anywhere in +the test suite. Every test drives the real advisor classes against a hand-written `ScriptedChatModel` +that queues real `ChatResponse` objects, so what gets asserted is the advisor's real behaviour, not +a description of it. + +Companion code for [Tool Calling in Spring AI 2.0](https://ankurm.com/spring-ai-2-0-tool-calling/) +on [ankurm.com](https://ankurm.com). + +## Versions + +| Component | Version | +|---|---| +| Spring Boot | 4.1.1 | +| Spring AI | 2.0.1 | +| `spring-ai-tool-search-advisor` / `spring-ai-tool-search-tool` | 2.0.1 (separately versioned artifacts, not covered by `spring-ai-bom` -- see the article's "going deeper" note) | +| Java | 25 (LTS) | + +## Quickstart + +```bash +./scripts/run-all.sh +``` + +Runs the full test suite. No API key, no network access, no Docker -- every test builds a +`ChatClient` around `ScriptedChatModel` (a queue of pre-programmed `ChatResponse`s) instead of a +real LLM. Output lands in `output/`. + +## What's here + +| File | What it does | +|---|---| +| [`tools/WeatherTools.java`](src/main/java/com/ankurm/toolcalling/tools/WeatherTools.java) | The baseline: an ordinary `@Tool` method, no special attributes. | +| [`tools/ServerStatusTools.java`](src/main/java/com/ankurm/toolcalling/tools/ServerStatusTools.java) | `@Tool(returnDirect = true)` -- the tool's own return value skips the second model round trip entirely. | +| [`tools/UserContextTools.java`](src/main/java/com/ankurm/toolcalling/tools/UserContextTools.java) | A `ToolContext`-typed parameter -- excluded from the model-facing JSON schema, still delivered at call time. | +| [`config/LargeToolLibrary.java`](src/main/java/com/ankurm/toolcalling/config/LargeToolLibrary.java) | Generates 230 synthetic `ToolCallback`s across six fake business domains via `FunctionToolCallback.builder(...)`, so the search-advisor section has something realistic to search. | +| [`config/ChatClientFactory.java`](src/main/java/com/ankurm/toolcalling/config/ChatClientFactory.java) | The exact `ChatClient` construction shared by production wiring (`ChatClientConfig`) and every test -- including the `ToolCallingChatOptions` base-options fix documented in its Javadoc, found by disassembling `ToolCallingAdvisor.adviseCall`. | +| [`config/ChatClientConfig.java`](src/main/java/com/ankurm/toolcalling/config/ChatClientConfig.java) | Production Spring `@Bean` wiring around a real (autoconfigured) `ChatModel`. | +| [`support/ScriptedChatModel.java`](src/test/java/com/ankurm/toolcalling/support/ScriptedChatModel.java) | A hand-written `ChatModel` (only `call(Prompt)` is abstract -- confirmed with `javap`) that returns a queued script of responses instead of calling a real API. | + +## Output files + +| File | What it captures | +|---|---| +| `output/01-plain-tool-call-round-trip.txt` | Two model calls for one tool question: the tool-call request, the tool result fed back, the final answer. | +| `output/02-return-direct-skips-second-round.txt` | `returnDirect = true`: exactly one model call, the tool's own JSON returned unparaphrased. | +| `output/03-tool-context-hidden-from-schema.txt` | `my_account`'s real JSON schema (no `userId` property anywhere in it) next to the real answer, built from a `userId` the caller supplied out-of-band. | +| `output/04-tool-search-progressive-disclosure.txt` | Tool counts actually offered to the model, read straight off `ToolCallingChatOptions.getToolCallbacks()`: 1 before searching, 6 after. | +| `output/05-tool-description-footprint.txt` | Measured character-count comparison between the full 230-tool definition text and `toolSearchTool`'s definition alone. | +| `output/06-tool-library-size.txt` | The exact size and per-domain breakdown of the synthetic library, asserted by a test rather than hand-counted. | + +## Requirements + +Nothing beyond the JDK and Maven. `spring-ai-starter-model-openai` is on the classpath for the +production `ChatClientConfig` bean to compile and (optionally) run against a real key via +`OPENAI_API_KEY`, but the test suite never constructs it. diff --git a/tool-calling/output/01-plain-tool-call-round-trip.txt b/tool-calling/output/01-plain-tool-call-round-trip.txt new file mode 100644 index 0000000..6a30db1 --- /dev/null +++ b/tool-calling/output/01-plain-tool-call-round-trip.txt @@ -0,0 +1,12 @@ +# Plain ToolCallingAdvisor: two model calls for one weather question + +model call count: 2 + +call 1 -- model requests a tool: + tool call: What's the weather in Boston? + +call 2 -- advisor sends the tool result back to the model: + tool response: ToolResponse[id=call-1, name=current_weather, responseData={"city":"Boston","temperatureCelsius":14.5,"conditions":"Overcast","humidityPercent":71}] + +final answer returned to the caller: + It's 14.5C and overcast in Boston right now, with 71% humidity. diff --git a/tool-calling/output/02-return-direct-skips-second-round.txt b/tool-calling/output/02-return-direct-skips-second-round.txt new file mode 100644 index 0000000..aae45a6 --- /dev/null +++ b/tool-calling/output/02-return-direct-skips-second-round.txt @@ -0,0 +1,6 @@ +# returnDirect = true: the model is called exactly once + +model call count: 1 + +raw content returned to the caller (the tool's own JSON, unparaphrased): + {"service":"orders-api","status":"UP","activeConnections":214,"checkedAt":"2026-09-23T18:00:00Z"} diff --git a/tool-calling/output/03-tool-context-hidden-from-schema.txt b/tool-calling/output/03-tool-context-hidden-from-schema.txt new file mode 100644 index 0000000..8f46efb --- /dev/null +++ b/tool-calling/output/03-tool-context-hidden-from-schema.txt @@ -0,0 +1,16 @@ +# ToolContext: excluded from the model-facing schema, still delivered to the tool + +my_account input schema sent to the model: + { + "$schema" : "https://json-schema.org/draft/2020-12/schema", + "type" : "object", + "properties" : { }, + "required" : [ ], + "additionalProperties" : false +} + +model's tool call arguments (empty -- it was never told a userId parameter exists): + {} + +final answer, using the real userId the CALLER supplied via ChatClient.toolContext(): + You're on the GOLD tier with a balance of $4820.50. diff --git a/tool-calling/output/04-tool-search-progressive-disclosure.txt b/tool-calling/output/04-tool-search-progressive-disclosure.txt new file mode 100644 index 0000000..c0ab757 --- /dev/null +++ b/tool-calling/output/04-tool-search-progressive-disclosure.txt @@ -0,0 +1,11 @@ +# ToolSearchToolCallingAdvisor: one tool offered up front instead of the whole library + +tools registered on the ChatClient: 230 + +tools OFFERED to the model on call 1 (before any search): 1 + [toolSearchTool] + +tools OFFERED to the model on call 2 (after it searched for "look up an invoice"): 6 + [crm_lookup_support_ticket, finance_lookup_invoice, hr_lookup_employee_record, hr_lookup_open_requisition, hr_lookup_pto_balance, toolSearchTool] + +final answer: Invoice INV-1001 was found in the finance system. diff --git a/tool-calling/output/05-tool-description-footprint.txt b/tool-calling/output/05-tool-description-footprint.txt new file mode 100644 index 0000000..18317f1 --- /dev/null +++ b/tool-calling/output/05-tool-description-footprint.txt @@ -0,0 +1,10 @@ +# Definition-text footprint: full library vs. one search-tool definition + +tools in the library: 230 +total characters of name + description + input schema, ALL 230 tools: 69958 +characters of name + description + input schema, toolSearchTool ONLY: 463 +reduction on the first call of a conversation: 99.3% + +caveat: characters are not tokens, and this is the FIRST call only -- once the model +has searched, the tools it found are added back in for the rest of that conversation. +See ToolSearchProgressiveDisclosureTest / output/04 for what gets added back. diff --git a/tool-calling/output/06-tool-library-size.txt b/tool-calling/output/06-tool-library-size.txt new file mode 100644 index 0000000..c13a85f --- /dev/null +++ b/tool-calling/output/06-tool-library-size.txt @@ -0,0 +1,16 @@ +# Synthetic tool library: size and per-domain breakdown + +total tools: 230 + +crm 40 tools +devops 38 tools +finance 40 tools +hr 36 tools +logistics 40 tools +security 36 tools + +sample tool definitions: + hr_lookup_employee_record Look up a single employee record by ID in the hr system. + hr_list_employee_record List employee record records in the hr system, optionally filtered. + hr_create_employee_record Create a new employee record in the hr system. + hr_update_employee_record Update an existing employee record in the hr system. diff --git a/tool-calling/pom.xml b/tool-calling/pom.xml new file mode 100644 index 0000000..6f15e55 --- /dev/null +++ b/tool-calling/pom.xml @@ -0,0 +1,79 @@ + + + 4.0.0 + + + org.springframework.boot + spring-boot-starter-parent + 4.1.1 + + + + com.ankurm + tool-calling + 1.0.0 + tool-calling + Tool calling in Spring AI 2.0: @Tool methods, ToolCallingAdvisor, returnDirect, tool context, and the Tool Search Advisor for progressive disclosure across a large tool library. + + + 25 + + 2.0.1 + + + + + + org.springframework.ai + spring-ai-bom + ${spring-ai.version} + pom + import + + + + + + + org.springframework.ai + spring-ai-starter-model-openai + + + + + org.springframework.ai + spring-ai-tool-search-advisor + ${spring-ai.version} + + + org.springframework.ai + spring-ai-tool-search-tool + ${spring-ai.version} + + + + org.springframework.boot + spring-boot-starter-test + test + + + + + + + org.springframework.boot + spring-boot-maven-plugin + + + org.apache.maven.plugins + maven-surefire-plugin + + -Duser.timezone=UTC -Dstdout.encoding=UTF-8 -Dfile.encoding=UTF-8 + + + + + diff --git a/tool-calling/scripts/run-all.sh b/tool-calling/scripts/run-all.sh new file mode 100644 index 0000000..8876081 --- /dev/null +++ b/tool-calling/scripts/run-all.sh @@ -0,0 +1,7 @@ +#!/usr/bin/env bash +# Regenerates every file under output/ -- the test suite writes each one itself via the +# Transcript helper, overwriting it in place. No file in this module's output/ is hand-captured. +set -euo pipefail +cd "$(dirname "$0")/.." +rm -rf target +mvn -q -o test diff --git a/tool-calling/src/main/java/com/ankurm/toolcalling/ToolCallingApplication.java b/tool-calling/src/main/java/com/ankurm/toolcalling/ToolCallingApplication.java new file mode 100644 index 0000000..d68f405 --- /dev/null +++ b/tool-calling/src/main/java/com/ankurm/toolcalling/ToolCallingApplication.java @@ -0,0 +1,12 @@ +package com.ankurm.toolcalling; + +import org.springframework.boot.SpringApplication; +import org.springframework.boot.autoconfigure.SpringBootApplication; + +@SpringBootApplication +public class ToolCallingApplication { + + public static void main(String[] args) { + SpringApplication.run(ToolCallingApplication.class, args); + } +} diff --git a/tool-calling/src/main/java/com/ankurm/toolcalling/config/ChatClientConfig.java b/tool-calling/src/main/java/com/ankurm/toolcalling/config/ChatClientConfig.java new file mode 100644 index 0000000..9e8cd7c --- /dev/null +++ b/tool-calling/src/main/java/com/ankurm/toolcalling/config/ChatClientConfig.java @@ -0,0 +1,39 @@ +package com.ankurm.toolcalling.config; + +import org.springframework.ai.chat.client.ChatClient; +import org.springframework.ai.chat.model.ChatModel; +import org.springframework.ai.tool.ToolCallback; +import org.springframework.ai.tool.toolsearch.index.regex.RegexToolIndex; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; + +import java.util.List; + +/** + * Production wiring for the two {@link ChatClient} beans this module demonstrates. Both share the + * same autoconfigured {@link ChatModel} (OpenAI, from {@code spring-ai-starter-model-openai}) and + * the same synthetic tool library from {@link LargeToolLibrary} -- the only thing that differs is + * the advisor, which is the whole point of the comparison. See {@link ChatClientFactory} for the + * shared construction logic, and the test sources for the same factory methods driven by + * {@code ScriptedChatModel} instead of a live model. + */ +@Configuration +public class ChatClientConfig { + + @Bean + public RegexToolIndex regexToolIndex() { + return new RegexToolIndex(); + } + + @Bean("plainToolCallingChatClient") + public ChatClient plainToolCallingChatClient(ChatModel chatModel) { + List library = LargeToolLibrary.buildAll(); + return ChatClientFactory.plainToolCallingClient(chatModel, library); + } + + @Bean("toolSearchChatClient") + public ChatClient toolSearchChatClient(ChatModel chatModel, RegexToolIndex regexToolIndex) { + List library = LargeToolLibrary.buildAll(); + return ChatClientFactory.toolSearchClient(chatModel, library, regexToolIndex, 5); + } +} diff --git a/tool-calling/src/main/java/com/ankurm/toolcalling/config/ChatClientFactory.java b/tool-calling/src/main/java/com/ankurm/toolcalling/config/ChatClientFactory.java new file mode 100644 index 0000000..04c56df --- /dev/null +++ b/tool-calling/src/main/java/com/ankurm/toolcalling/config/ChatClientFactory.java @@ -0,0 +1,86 @@ +package com.ankurm.toolcalling.config; + +import org.springframework.ai.chat.client.ChatClient; +import org.springframework.ai.chat.client.advisor.ToolCallingAdvisor; +import org.springframework.ai.chat.client.advisor.toolsearch.ToolSearchToolCallingAdvisor; +import org.springframework.ai.chat.model.ChatModel; +import org.springframework.ai.model.tool.ToolCallingChatOptions; +import org.springframework.ai.tool.ToolCallback; +import org.springframework.ai.tool.toolsearch.ToolIndex; +import org.springframework.ai.tool.toolsearch.ToolSearchTool; + +import java.util.List; + +/** + * Builds the two {@link ChatClient} configurations this module compares, from the same underlying + * {@link ChatModel} and the same tool library. Shared between {@link ChatClientConfig} (real Spring + * beans, wired with a real {@code OpenAiChatModel}) and the test sources (wired with + * {@code ScriptedChatModel}) so the exact construction under test is the exact construction that + * runs in production -- nothing about advisor wiring is reimplemented for the tests. + */ +public final class ChatClientFactory { + + private ChatClientFactory() { + } + + /** + * The Spring AI 1.x-shaped baseline: every tool callback registered directly on the + * {@link ChatClient}, and a plain {@link ToolCallingAdvisor} driving the call/execute/recall + * loop. Every tool's full name, description and JSON schema goes into the model's context on + * every single call -- fine for a handful of tools, expensive once the library reaches + * hundreds. + */ + public static ChatClient plainToolCallingClient(ChatModel chatModel, List toolLibrary) { + ToolCallingAdvisor advisor = ToolCallingAdvisor.builder().build(); + return ChatClient.builder(chatModel) + .defaultOptions(baseToolCallingOptions(chatModel)) + .defaultToolCallbacks(toolLibrary) + .defaultAdvisors(advisor) + .build(); + } + + /** + * The progressive-disclosure alternative: the tool library is still registered with the + * client (so it can actually be called once found), but {@link ToolSearchToolCallingAdvisor} + * indexes it into a {@link ToolIndex} and gives the model exactly one tool up front -- + * {@code toolSearchTool} -- plus a system-message suffix explaining how to use it. The model + * searches for the handful of tools relevant to the current request instead of reading + * descriptions for all of them on every call. + */ + public static ChatClient toolSearchClient(ChatModel chatModel, List toolLibrary, + ToolIndex toolIndex, Integer maxResults) { + ToolSearchToolCallingAdvisor advisor = ToolSearchToolCallingAdvisor.builder() + .toolIndex(toolIndex) + .maxResults(maxResults) + .sessionIdKeyName(ToolSearchTool.TOOL_SEARCH_TOOL_SESSION_ID_KEY) + .systemMessageSuffix("Use the toolSearchTool to find the specific tool you need before calling " + + "it. Do not guess a tool name that was not returned by a search.") + .build(); + return ChatClient.builder(chatModel) + .defaultOptions(baseToolCallingOptions(chatModel)) + .defaultToolCallbacks(toolLibrary) + .defaultAdvisors(advisor) + .build(); + } + + /** + * {@code ToolCallingAdvisor} only engages its call/execute/recall loop when the request's + * {@link org.springframework.ai.chat.prompt.Prompt Prompt} carries {@link ToolCallingChatOptions} + * -- confirmed by disassembling {@code ToolCallingAdvisor.adviseCall}, whose very first check is + * an {@code instanceof ToolCallingChatOptions} on {@code chatClientRequest.prompt().getOptions()} + * that falls straight through to {@code chain.nextCall(request)} (no tool execution at all) when + * it is anything else. {@code DefaultChatClientUtils} builds that options object from + * {@code chatModel.getOptions().mutate()} -- not {@code getDefaultOptions()}, a separate default + * method the request-building path never calls -- merged with whatever this builder sets here. + * Real providers (OpenAI, Anthropic) return their own {@code ToolCallingChatOptions} + * implementation from {@code getOptions()}; this mutates a copy of that so provider-specific + * settings survive, or falls back to an empty builder for a model (like + * {@code ScriptedChatModel}) whose own {@code getOptions()} already does the same thing. + */ + private static ToolCallingChatOptions.Builder baseToolCallingOptions(ChatModel chatModel) { + if (chatModel.getOptions() instanceof ToolCallingChatOptions existing) { + return existing.mutate(); + } + return ToolCallingChatOptions.builder(); + } +} diff --git a/tool-calling/src/main/java/com/ankurm/toolcalling/config/ClockConfig.java b/tool-calling/src/main/java/com/ankurm/toolcalling/config/ClockConfig.java new file mode 100644 index 0000000..0a69ee0 --- /dev/null +++ b/tool-calling/src/main/java/com/ankurm/toolcalling/config/ClockConfig.java @@ -0,0 +1,15 @@ +package com.ankurm.toolcalling.config; + +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; + +import java.time.Clock; + +@Configuration +public class ClockConfig { + + @Bean + public Clock clock() { + return Clock.systemUTC(); + } +} diff --git a/tool-calling/src/main/java/com/ankurm/toolcalling/config/LargeToolLibrary.java b/tool-calling/src/main/java/com/ankurm/toolcalling/config/LargeToolLibrary.java new file mode 100644 index 0000000..273e7b9 --- /dev/null +++ b/tool-calling/src/main/java/com/ankurm/toolcalling/config/LargeToolLibrary.java @@ -0,0 +1,110 @@ +package com.ankurm.toolcalling.config; + +import com.ankurm.toolcalling.domain.SyntheticToolRequest; +import org.springframework.ai.tool.ToolCallback; +import org.springframework.ai.tool.function.FunctionToolCallback; + +import java.util.ArrayList; +import java.util.List; +import java.util.function.Function; + +/** + * Generates a synthetic library of tool callbacks spread across several fake business domains, so + * the Tool Search Advisor section has something realistic to search over. A real system this size + * -- a platform team's internal tool catalog, say -- would have these as separate + * {@code @Service} classes; they are generated here so the repository does not need 130 near- + * identical Java files to make the point. + * + *

Every tool has the same {@link SyntheticToolRequest} input shape and the same trivial + * behaviour (echo a deterministic string). What varies, deliberately, is the tool name and + * description -- the only two things {@link org.springframework.ai.tool.toolsearch.index.regex + * RegexToolIndex} has to search against. See {@code ToolLibrarySizeTest} for the exact count this + * produces, captured to {@code output/06-tool-library-size.txt}. + */ +public final class LargeToolLibrary { + + /** domain, verb, noun -- combined into "__" tool names. */ + private static final String[] DOMAINS = { + "hr", "finance", "devops", "crm", "logistics", "security" + }; + + private static final String[] VERBS = { + "lookup", "list", "create", "update", "archive", "search", "export", "summarize" + }; + + private static final String[][] NOUNS_BY_DOMAIN = { + // hr + { "employee_record", "open_requisition", "pto_balance", "org_chart_node", "onboarding_task" }, + // finance + { "invoice", "purchase_order", "expense_report", "budget_line", "vendor_contract" }, + // devops + { "deployment", "incident", "build_pipeline", "feature_flag", "service_health_check" }, + // crm + { "lead", "opportunity", "support_ticket", "account_note", "renewal_forecast" }, + // logistics + { "shipment", "warehouse_slot", "carrier_rate", "inventory_count", "delivery_route" }, + }; + + private LargeToolLibrary() { + } + + /** + * Builds every synthetic tool in the library. Not every domain crosses every verb -- the + * generator skips a few combinations to land at a round, documented count rather than an + * arbitrary one; the exact number is asserted by {@code ToolLibrarySizeTest}, not hand-counted + * here. + */ + public static List buildAll() { + List callbacks = new ArrayList<>(); + for (int d = 0; d < DOMAINS.length; d++) { + String domain = DOMAINS[d]; + String[] nouns = NOUNS_BY_DOMAIN[d % NOUNS_BY_DOMAIN.length]; + for (String noun : nouns) { + for (String verb : VERBS) { + // Skip a couple of combinations per noun so not every noun exposes every verb -- + // "archive_pto_balance" and "create_service_health_check" do not make business + // sense, and a library where every tool crosses every verb would be unrealistically + // uniform for the search index to tell apart. + if (skip(verb, noun)) { + continue; + } + callbacks.add(buildOne(domain, verb, noun)); + } + } + } + return callbacks; + } + + private static boolean skip(String verb, String noun) { + boolean lifecycleNoun = noun.endsWith("_balance") || noun.endsWith("_check") || noun.endsWith("_node"); + return lifecycleNoun && (verb.equals("archive") || verb.equals("create")); + } + + private static ToolCallback buildOne(String domain, String verb, String noun) { + String name = domain + "_" + verb + "_" + noun; + String description = describe(domain, verb, noun); + Function function = request -> { + String id = (request != null && request.id() != null) ? request.id() : "unspecified"; + return verb + " " + noun.replace('_', ' ') + " [" + domain + "] for id=" + id + ": ok"; + }; + return FunctionToolCallback.builder(name, function) + .description(description) + .inputType(SyntheticToolRequest.class) + .build(); + } + + private static String describe(String domain, String verb, String noun) { + String nounPhrase = noun.replace('_', ' '); + return switch (verb) { + case "lookup" -> "Look up a single " + nounPhrase + " by ID in the " + domain + " system."; + case "list" -> "List " + nounPhrase + " records in the " + domain + " system, optionally filtered."; + case "create" -> "Create a new " + nounPhrase + " in the " + domain + " system."; + case "update" -> "Update an existing " + nounPhrase + " in the " + domain + " system."; + case "archive" -> "Archive a " + nounPhrase + " in the " + domain + " system so it no longer appears in active lists."; + case "search" -> "Full-text search across " + nounPhrase + " records in the " + domain + " system."; + case "export" -> "Export " + nounPhrase + " records from the " + domain + " system as a downloadable report."; + case "summarize" -> "Summarize the current state of " + nounPhrase + " records in the " + domain + " system."; + default -> "Operate on " + nounPhrase + " in the " + domain + " system."; + }; + } +} diff --git a/tool-calling/src/main/java/com/ankurm/toolcalling/domain/AccountSummary.java b/tool-calling/src/main/java/com/ankurm/toolcalling/domain/AccountSummary.java new file mode 100644 index 0000000..8ea3e26 --- /dev/null +++ b/tool-calling/src/main/java/com/ankurm/toolcalling/domain/AccountSummary.java @@ -0,0 +1,7 @@ +package com.ankurm.toolcalling.domain; + +/** + * See {@code com.ankurm.toolcalling.tools.UserContextTools#myAccount}. + */ +public record AccountSummary(String userId, String tier, double accountBalanceUsd) { +} diff --git a/tool-calling/src/main/java/com/ankurm/toolcalling/domain/ServerStatus.java b/tool-calling/src/main/java/com/ankurm/toolcalling/domain/ServerStatus.java new file mode 100644 index 0000000..40ff4da --- /dev/null +++ b/tool-calling/src/main/java/com/ankurm/toolcalling/domain/ServerStatus.java @@ -0,0 +1,11 @@ +package com.ankurm.toolcalling.domain; + +import java.time.Instant; + +/** + * Returned as-is to the caller by a {@code returnDirect = true} tool -- the model never sees this + * record. See {@code com.ankurm.toolcalling.tools.ServerStatusTools#serverStatus} and the + * "what returnDirect actually skips" section of the post. + */ +public record ServerStatus(String service, String status, int activeConnections, Instant checkedAt) { +} diff --git a/tool-calling/src/main/java/com/ankurm/toolcalling/domain/SyntheticToolRequest.java b/tool-calling/src/main/java/com/ankurm/toolcalling/domain/SyntheticToolRequest.java new file mode 100644 index 0000000..029d424 --- /dev/null +++ b/tool-calling/src/main/java/com/ankurm/toolcalling/domain/SyntheticToolRequest.java @@ -0,0 +1,11 @@ +package com.ankurm.toolcalling.domain; + +/** + * The single input type shared by every tool in {@code LargeToolLibrary}. Real tools in a system + * this size would each have their own request shape; this repository uses one shared shape on + * purpose, so the only thing that varies from tool to tool is the name and description text the + * search index has to tell apart -- which is exactly what {@code ToolSearchToolCallingAdvisor} is + * being asked to do well. + */ +public record SyntheticToolRequest(String id) { +} diff --git a/tool-calling/src/main/java/com/ankurm/toolcalling/domain/WeatherReport.java b/tool-calling/src/main/java/com/ankurm/toolcalling/domain/WeatherReport.java new file mode 100644 index 0000000..75c788e --- /dev/null +++ b/tool-calling/src/main/java/com/ankurm/toolcalling/domain/WeatherReport.java @@ -0,0 +1,10 @@ +package com.ankurm.toolcalling.domain; + +/** + * The shape a {@code @Tool} method returns when the framework should serialize the result back + * into the model's tool-response message. There is nothing MCP- or Spring-AI-specific about this + * record -- it is deserialized to JSON by the same Jackson machinery that serializes any other + * return type. See {@code com.ankurm.toolcalling.tools.WeatherTools#currentWeather}. + */ +public record WeatherReport(String city, double temperatureCelsius, String conditions, int humidityPercent) { +} diff --git a/tool-calling/src/main/java/com/ankurm/toolcalling/tools/ServerStatusTools.java b/tool-calling/src/main/java/com/ankurm/toolcalling/tools/ServerStatusTools.java new file mode 100644 index 0000000..cf345c2 --- /dev/null +++ b/tool-calling/src/main/java/com/ankurm/toolcalling/tools/ServerStatusTools.java @@ -0,0 +1,48 @@ +package com.ankurm.toolcalling.tools; + +import com.ankurm.toolcalling.domain.ServerStatus; +import org.springframework.ai.tool.annotation.Tool; +import org.springframework.ai.tool.annotation.ToolParam; +import org.springframework.stereotype.Service; + +import java.time.Clock; +import java.time.Instant; +import java.util.Map; + +/** + * {@code returnDirect = true} means the tool's raw return value goes straight back to whatever + * called the {@code ChatClient} -- it is never sent to the model as a tool-result message for a + * second inference pass. Useful for tools whose output is already the final answer (a status + * dashboard, a generated file, a confirmation payload) where a second model round-trip would only + * add latency and a chance for the model to paraphrase (or misstate) a number that was already + * correct. See {@code ScriptedChatModel} in the test sources for how the test proves the model + * really is skipped -- it queues only one response and the test still gets the tool's own object + * back. + */ +@Service +public class ServerStatusTools { + + private static final Map ACTIVE_CONNECTIONS = Map.of( + "orders-api", 214, + "inventory-api", 58, + "notifications-api", 3); + + private final Clock clock; + + public ServerStatusTools(Clock clock) { + this.clock = clock; + } + + @Tool(name = "server_status", description = "Get the live status of one of this company's internal services " + + "(orders-api, inventory-api, notifications-api). The result is returned directly to the " + + "caller without a further model round-trip.", returnDirect = true) + public ServerStatus serverStatus( + @ToolParam(description = "The service name, e.g. orders-api", required = true) String serviceName) { + Integer connections = ACTIVE_CONNECTIONS.get(serviceName); + if (connections == null) { + throw new IllegalArgumentException("Unknown service: " + serviceName + + ". Known services: " + ACTIVE_CONNECTIONS.keySet()); + } + return new ServerStatus(serviceName, "UP", connections, Instant.now(this.clock)); + } +} diff --git a/tool-calling/src/main/java/com/ankurm/toolcalling/tools/UserContextTools.java b/tool-calling/src/main/java/com/ankurm/toolcalling/tools/UserContextTools.java new file mode 100644 index 0000000..af657d1 --- /dev/null +++ b/tool-calling/src/main/java/com/ankurm/toolcalling/tools/UserContextTools.java @@ -0,0 +1,49 @@ +package com.ankurm.toolcalling.tools; + +import com.ankurm.toolcalling.domain.AccountSummary; +import org.springframework.ai.chat.model.ToolContext; +import org.springframework.ai.tool.annotation.Tool; +import org.springframework.stereotype.Service; + +import java.util.Map; + +/** + * {@link ToolContext} is how the calling code hands a tool data that the model must never see or + * choose -- the currently authenticated user, a tenant ID, a feature flag -- without inventing a + * fake tool parameter and hoping the model fills it in correctly (or, worse, lets the model supply + * its own "userId" string). A {@code @Tool} method that declares a parameter of type + * {@link ToolContext} receives whatever map the caller passed to + * {@code ChatClient#toolContext(Map)}; Spring AI recognizes the parameter by its type, not its + * name or position, and never exposes it to the model's function-calling schema. + * + *

This class deliberately has no repository or security layer behind it -- {@code myAccount} + * trusts the {@code userId} key in the {@link ToolContext} outright. In a real deployment that key + * would be populated from the authenticated principal on the server side (the same + * {@code SecurityContextHolder} pattern used in the {@code mcp-secure} module), never from + * anything a client sends. + */ +@Service +public class UserContextTools { + + static final String USER_ID_KEY = "userId"; + + private static final Map ACCOUNTS = Map.of( + "alice", new AccountSummary("alice", "GOLD", 4820.50), + "bob", new AccountSummary("bob", "STANDARD", 112.75)); + + @Tool(name = "my_account", description = "Get the current user's own account summary: tier and balance. " + + "Takes no arguments -- the user is identified from the caller's context, not from anything " + + "the model supplies.") + public AccountSummary myAccount(ToolContext toolContext) { + Object userId = (toolContext != null) ? toolContext.getContext().get(USER_ID_KEY) : null; + if (userId == null) { + throw new IllegalStateException("No " + USER_ID_KEY + " in ToolContext -- the caller must set one " + + "with ChatClient.toolContext(Map.of(\"" + USER_ID_KEY + "\", ...)) before this tool can run."); + } + AccountSummary account = ACCOUNTS.get(userId.toString()); + if (account == null) { + throw new IllegalArgumentException("No account for userId: " + userId); + } + return account; + } +} diff --git a/tool-calling/src/main/java/com/ankurm/toolcalling/tools/WeatherTools.java b/tool-calling/src/main/java/com/ankurm/toolcalling/tools/WeatherTools.java new file mode 100644 index 0000000..bb390b5 --- /dev/null +++ b/tool-calling/src/main/java/com/ankurm/toolcalling/tools/WeatherTools.java @@ -0,0 +1,35 @@ +package com.ankurm.toolcalling.tools; + +import com.ankurm.toolcalling.domain.WeatherReport; +import org.springframework.ai.tool.annotation.Tool; +import org.springframework.ai.tool.annotation.ToolParam; +import org.springframework.stereotype.Service; + +import java.util.Map; + +/** + * The plain, ordinary case: a {@code @Tool} method with no special attributes. The model calls + * it, gets JSON back, and decides what to say about it. This is the baseline every other tool in + * this repository is contrasted against -- {@code ServerStatusTools} for {@code returnDirect}, + * {@code UserContextTools} for {@link org.springframework.ai.chat.model.ToolContext}. + */ +@Service +public class WeatherTools { + + private static final Map FIXTURE_DATA = Map.of( + "Boston", new WeatherReport("Boston", 14.5, "Overcast", 71), + "Austin", new WeatherReport("Austin", 29.0, "Clear", 44), + "Seattle", new WeatherReport("Seattle", 12.0, "Light rain", 88)); + + @Tool(name = "current_weather", description = "Get the current weather conditions for a named city. " + + "Returns temperature in Celsius, a short conditions description, and relative humidity.") + public WeatherReport currentWeather( + @ToolParam(description = "The city name, e.g. Boston", required = true) String city) { + WeatherReport report = FIXTURE_DATA.get(city); + if (report == null) { + throw new IllegalArgumentException("No weather fixture for city: " + city + + ". Known cities: " + FIXTURE_DATA.keySet()); + } + return report; + } +} diff --git a/tool-calling/src/main/resources/application.yml b/tool-calling/src/main/resources/application.yml new file mode 100644 index 0000000..f3c1ab4 --- /dev/null +++ b/tool-calling/src/main/resources/application.yml @@ -0,0 +1,11 @@ +spring: + application: + name: tool-calling + ai: + openai: + # No real key is needed to run this module's test suite -- the tests never construct the + # autoconfigured OpenAiChatModel bean, they build a ChatClient directly around + # ScriptedChatModel. This placeholder only exists so the application context *could* start + # standalone (e.g. via `mvn spring-boot:run`) with a real key supplied as an environment + # variable. + api-key: ${OPENAI_API_KEY:demo-key-not-used-by-tests} diff --git a/tool-calling/src/test/java/com/ankurm/toolcalling/PlainToolCallingRoundTripTest.java b/tool-calling/src/test/java/com/ankurm/toolcalling/PlainToolCallingRoundTripTest.java new file mode 100644 index 0000000..02f8b6a --- /dev/null +++ b/tool-calling/src/test/java/com/ankurm/toolcalling/PlainToolCallingRoundTripTest.java @@ -0,0 +1,74 @@ +package com.ankurm.toolcalling; + +import com.ankurm.toolcalling.config.ChatClientFactory; +import com.ankurm.toolcalling.support.ScriptedChatModel; +import com.ankurm.toolcalling.support.Transcript; +import com.ankurm.toolcalling.tools.WeatherTools; +import org.junit.jupiter.api.Test; +import org.springframework.ai.chat.client.ChatClient; +import org.springframework.ai.chat.messages.AssistantMessage; +import org.springframework.ai.chat.messages.Message; +import org.springframework.ai.chat.messages.ToolResponseMessage; +import org.springframework.ai.chat.prompt.Prompt; +import org.springframework.ai.support.ToolCallbacks; + +import java.util.List; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * The baseline round trip: the model asks for a tool, {@code ToolCallingAdvisor} executes it and + * calls the model a second time with the result, and only then does the caller get a final answer. + * See {@code com.ankurm.toolcalling.tools.WeatherTools} for the tool itself. + */ +class PlainToolCallingRoundTripTest { + + @Test + void modelCallsToolThenGetsCalledAgainWithTheResult() { + ScriptedChatModel model = ScriptedChatModel.builder() + .thenCallTools(new AssistantMessage.ToolCall("call-1", "function", "current_weather", + "{\"city\":\"Boston\"}")) + .thenRespond("It's 14.5C and overcast in Boston right now, with 71% humidity.") + .build(); + + ChatClient client = ChatClientFactory.plainToolCallingClient(model, + List.of(ToolCallbacks.from(new WeatherTools()))); + + String answer = client.prompt().user("What's the weather in Boston?").call().content(); + + assertThat(model.callCount()).isEqualTo(2); + + Prompt secondPrompt = model.capturedPrompts().get(1); + ToolResponseMessage toolResponse = secondPrompt.getInstructions().stream() + .filter(ToolResponseMessage.class::isInstance) + .map(ToolResponseMessage.class::cast) + .findFirst() + .orElseThrow(() -> new AssertionError("second call did not include a ToolResponseMessage")); + + try (Transcript t = new Transcript("01-plain-tool-call-round-trip.txt", + "Plain ToolCallingAdvisor: two model calls for one weather question")) { + t.line("model call count: %d", model.callCount()); + t.blank(); + t.line("call 1 -- model requests a tool:"); + t.line(" tool call: %s", toolCallOf(model.capturedPrompts().get(0))); + t.blank(); + t.line("call 2 -- advisor sends the tool result back to the model:"); + t.line(" tool response: %s", toolResponse.getResponses().get(0)); + t.blank(); + t.line("final answer returned to the caller:"); + t.line(" %s", answer); + + assertThat(toolResponse.getResponses().get(0).name()).isEqualTo("current_weather"); + assertThat(toolResponse.getResponses().get(0).responseData()).contains("Boston").contains("Overcast"); + assertThat(answer).isEqualTo("It's 14.5C and overcast in Boston right now, with 71% humidity."); + } + } + + private static String toolCallOf(Prompt prompt) { + // First prompt has no assistant tool-call yet -- this reports the request itself, since the + // *response* to it (queued in the script) is what carries the tool call. Kept simple: log the + // user message that triggered it. + Message last = prompt.getInstructions().get(prompt.getInstructions().size() - 1); + return last.getText(); + } +} diff --git a/tool-calling/src/test/java/com/ankurm/toolcalling/ReturnDirectSkipsSecondRoundTest.java b/tool-calling/src/test/java/com/ankurm/toolcalling/ReturnDirectSkipsSecondRoundTest.java new file mode 100644 index 0000000..e420c58 --- /dev/null +++ b/tool-calling/src/test/java/com/ankurm/toolcalling/ReturnDirectSkipsSecondRoundTest.java @@ -0,0 +1,57 @@ +package com.ankurm.toolcalling; + +import com.ankurm.toolcalling.config.ChatClientFactory; +import com.ankurm.toolcalling.support.ScriptedChatModel; +import com.ankurm.toolcalling.support.Transcript; +import com.ankurm.toolcalling.tools.ServerStatusTools; +import org.junit.jupiter.api.Test; +import org.springframework.ai.chat.client.ChatClient; +import org.springframework.ai.chat.messages.AssistantMessage; +import org.springframework.ai.support.ToolCallbacks; + +import java.time.Clock; +import java.time.Instant; +import java.time.ZoneOffset; +import java.util.List; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * {@code @Tool(returnDirect = true)} means the advisor never makes the second model call at all -- + * the tool's own return value becomes the answer. This test only queues one scripted response on + * purpose: if the advisor tried to call the model again, {@link ScriptedChatModel} would throw on + * the second call with no response left, and the test would fail loudly rather than silently + * passing on a lucky assertion. See {@code com.ankurm.toolcalling.tools.ServerStatusTools}. + */ +class ReturnDirectSkipsSecondRoundTest { + + @Test + void toolResultIsReturnedWithoutAFollowUpModelCall() { + Clock fixedClock = Clock.fixed(Instant.parse("2026-09-23T18:00:00Z"), ZoneOffset.UTC); + + ScriptedChatModel model = ScriptedChatModel.builder() + .thenCallTools(new AssistantMessage.ToolCall("call-1", "function", "server_status", + "{\"serviceName\":\"orders-api\"}")) + // Deliberately no second response queued -- returnDirect must mean the advisor never + // asks. + .build(); + + ChatClient client = ChatClientFactory.plainToolCallingClient(model, + List.of(ToolCallbacks.from(new ServerStatusTools(fixedClock)))); + + String answer = client.prompt().user("Is orders-api up?").call().content(); + + try (Transcript t = new Transcript("02-return-direct-skips-second-round.txt", + "returnDirect = true: the model is called exactly once")) { + t.line("model call count: %d", model.callCount()); + t.blank(); + t.line("raw content returned to the caller (the tool's own JSON, unparaphrased):"); + t.line(" %s", answer); + + assertThat(model.callCount()) + .as("returnDirect should mean the advisor never calls the model a second time") + .isEqualTo(1); + assertThat(answer).contains("orders-api").contains("UP").contains("214"); + } + } +} diff --git a/tool-calling/src/test/java/com/ankurm/toolcalling/ToolContextNotExposedTest.java b/tool-calling/src/test/java/com/ankurm/toolcalling/ToolContextNotExposedTest.java new file mode 100644 index 0000000..0e32d9e --- /dev/null +++ b/tool-calling/src/test/java/com/ankurm/toolcalling/ToolContextNotExposedTest.java @@ -0,0 +1,62 @@ +package com.ankurm.toolcalling; + +import com.ankurm.toolcalling.config.ChatClientFactory; +import com.ankurm.toolcalling.support.ScriptedChatModel; +import com.ankurm.toolcalling.support.Transcript; +import com.ankurm.toolcalling.tools.UserContextTools; +import org.junit.jupiter.api.Test; +import org.springframework.ai.chat.client.ChatClient; +import org.springframework.ai.chat.messages.AssistantMessage; +import org.springframework.ai.support.ToolCallbacks; +import org.springframework.ai.tool.ToolCallback; + +import java.util.List; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * Two things worth proving separately about {@link org.springframework.ai.chat.model.ToolContext + * ToolContext}: first, that it never appears in the JSON schema the model sees (so the model + * cannot supply, guess, or overwrite it); second, that the tool still receives the real value at + * call time, sourced from {@code ChatClient.toolContext(Map)} on the request, not from the model's + * tool-call arguments. See {@code com.ankurm.toolcalling.tools.UserContextTools}. + */ +class ToolContextNotExposedTest { + + @Test + void toolContextParameterIsHiddenFromTheSchemaButStillDeliveredToTheTool() { + ToolCallback callback = ToolCallbacks.from(new UserContextTools())[0]; + String schema = callback.getToolDefinition().inputSchema(); + + ScriptedChatModel model = ScriptedChatModel.builder() + // The model calls my_account with an EMPTY arguments object -- it has no userId to + // supply, because the schema never told it one exists. + .thenCallTools(new AssistantMessage.ToolCall("call-1", "function", "my_account", "{}")) + .thenRespond("You're on the GOLD tier with a balance of $4820.50.") + .build(); + + ChatClient client = ChatClientFactory.plainToolCallingClient(model, List.of(callback)); + + String answer = client.prompt() + .user("What's my account tier?") + .toolContext(Map.of("userId", "alice")) + .call() + .content(); + + try (Transcript t = new Transcript("03-tool-context-hidden-from-schema.txt", + "ToolContext: excluded from the model-facing schema, still delivered to the tool")) { + t.line("my_account input schema sent to the model:"); + t.line(" %s", schema); + t.blank(); + t.line("model's tool call arguments (empty -- it was never told a userId parameter exists):"); + t.line(" {}"); + t.blank(); + t.line("final answer, using the real userId the CALLER supplied via ChatClient.toolContext():"); + t.line(" %s", answer); + + assertThat(schema).doesNotContain("userId").doesNotContain("ToolContext").doesNotContain("toolContext"); + assertThat(answer).contains("GOLD").contains("4820.50"); + } + } +} diff --git a/tool-calling/src/test/java/com/ankurm/toolcalling/ToolDescriptionFootprintTest.java b/tool-calling/src/test/java/com/ankurm/toolcalling/ToolDescriptionFootprintTest.java new file mode 100644 index 0000000..632a64d --- /dev/null +++ b/tool-calling/src/test/java/com/ankurm/toolcalling/ToolDescriptionFootprintTest.java @@ -0,0 +1,70 @@ +package com.ankurm.toolcalling; + +import com.ankurm.toolcalling.config.LargeToolLibrary; +import com.ankurm.toolcalling.support.Transcript; +import org.junit.jupiter.api.Test; +import org.springframework.ai.tool.ToolCallback; +import org.springframework.ai.tool.definition.ToolDefinition; + +import java.util.List; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * Not a token count from a real model call -- no live model is involved anywhere in this module -- + * but a real, measured character count of the tool name/description/JSON-schema text that would go + * into the model's context on every single call under the plain {@code ToolCallingAdvisor} + * (all 230 tools), versus what goes in under {@code ToolSearchToolCallingAdvisor} on the first call + * of a conversation (just {@code toolSearchTool}'s own definition). Characters are not tokens, but + * they are a real, reproducible, unit-testable proxy for the same claim the Spring blog post this + * feature originated from makes in tokens -- see the "going deeper" note in the article for why + * this test measures characters rather than pretending to know the model's tokenizer. + */ +class ToolDescriptionFootprintTest { + + @Test + void fullLibraryDefinitionTextIsMuchLargerThanOneSearchToolDefinition() { + List library = LargeToolLibrary.buildAll(); + + int fullLibraryChars = library.stream().mapToInt(ToolDescriptionFootprintTest::definitionSize).sum(); + + // The tool-search client's first call offers exactly one definition: toolSearchTool itself. + // Its schema is small and fixed regardless of library size -- captured directly from the + // resolved artifact rather than hand-typed, so this number tracks the real dependency. + int searchToolOnlyChars = definitionSizeOf("toolSearchTool", + "Search for tools in the tool registry to discover capabilities for completing the " + + "current task.\nUse this when you need functionality not provided by your currently " + + "available tools.\nThe search queries against tool names, descriptions, and parameter " + + "information to find the most relevant tools.\nReturns references to matching tools " + + "which will be expanded into full definitions you can then invoke.\n", + "{\"query\":\"...\",\"maxResults\":5,\"categoryFilter\":\"...\"}"); + + double reduction = 100.0 * (1 - ((double) searchToolOnlyChars / fullLibraryChars)); + + try (Transcript t = new Transcript("05-tool-description-footprint.txt", + "Definition-text footprint: full library vs. one search-tool definition")) { + t.line("tools in the library: %d", library.size()); + t.line("total characters of name + description + input schema, ALL %d tools: %d", + library.size(), fullLibraryChars); + t.line("characters of name + description + input schema, toolSearchTool ONLY: %d", + searchToolOnlyChars); + t.line("reduction on the first call of a conversation: %.1f%%", reduction); + t.blank(); + t.line("caveat: characters are not tokens, and this is the FIRST call only -- once the model"); + t.line("has searched, the tools it found are added back in for the rest of that conversation."); + t.line("See ToolSearchProgressiveDisclosureTest / output/04 for what gets added back."); + + assertThat(fullLibraryChars).isGreaterThan(searchToolOnlyChars * 20); + assertThat(reduction).isGreaterThan(90.0); + } + } + + private static int definitionSize(ToolCallback callback) { + ToolDefinition definition = callback.getToolDefinition(); + return definitionSizeOf(definition.name(), definition.description(), definition.inputSchema()); + } + + private static int definitionSizeOf(String name, String description, String inputSchema) { + return name.length() + description.length() + inputSchema.length(); + } +} diff --git a/tool-calling/src/test/java/com/ankurm/toolcalling/ToolLibrarySizeTest.java b/tool-calling/src/test/java/com/ankurm/toolcalling/ToolLibrarySizeTest.java new file mode 100644 index 0000000..283a15f --- /dev/null +++ b/tool-calling/src/test/java/com/ankurm/toolcalling/ToolLibrarySizeTest.java @@ -0,0 +1,47 @@ +package com.ankurm.toolcalling; + +import com.ankurm.toolcalling.config.LargeToolLibrary; +import com.ankurm.toolcalling.support.Transcript; +import org.junit.jupiter.api.Test; +import org.springframework.ai.tool.ToolCallback; + +import java.util.List; +import java.util.Map; +import java.util.TreeMap; +import java.util.stream.Collectors; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * Proves the size and shape of the synthetic tool library that {@code ToolSearchProgressiveDisclosureTest} + * runs the search advisor against -- this is the number that makes "one tool advertised instead of + * the whole library" a real comparison instead of a made-up one. + */ +class ToolLibrarySizeTest { + + @Test + void libraryHasTheDocumentedShapeAndSize() { + List library = LargeToolLibrary.buildAll(); + + Map countsByDomain = new TreeMap<>(library.stream() + .collect(Collectors.groupingBy( + cb -> cb.getToolDefinition().name().split("_", 2)[0], + Collectors.counting()))); + + try (Transcript t = new Transcript("06-tool-library-size.txt", + "Synthetic tool library: size and per-domain breakdown")) { + t.line("total tools: %d", library.size()); + t.blank(); + countsByDomain.forEach((domain, count) -> t.line("%-10s %d tools", domain, count)); + t.blank(); + t.line("sample tool definitions:"); + library.stream().limit(4).forEach(cb -> t.line(" %-30s %s", cb.getToolDefinition().name(), + cb.getToolDefinition().description())); + + assertThat(library.size()).isEqualTo(230); + assertThat(countsByDomain).hasSize(6); + assertThat(library.stream().map(cb -> cb.getToolDefinition().name()).distinct().count()) + .isEqualTo(library.size()); + } + } +} diff --git a/tool-calling/src/test/java/com/ankurm/toolcalling/ToolSearchProgressiveDisclosureTest.java b/tool-calling/src/test/java/com/ankurm/toolcalling/ToolSearchProgressiveDisclosureTest.java new file mode 100644 index 0000000..d013725 --- /dev/null +++ b/tool-calling/src/test/java/com/ankurm/toolcalling/ToolSearchProgressiveDisclosureTest.java @@ -0,0 +1,85 @@ +package com.ankurm.toolcalling; + +import com.ankurm.toolcalling.config.ChatClientFactory; +import com.ankurm.toolcalling.config.LargeToolLibrary; +import com.ankurm.toolcalling.support.ScriptedChatModel; +import com.ankurm.toolcalling.support.Transcript; +import org.junit.jupiter.api.Test; +import org.springframework.ai.chat.client.ChatClient; +import org.springframework.ai.chat.messages.AssistantMessage; +import org.springframework.ai.tool.ToolCallback; +import org.springframework.ai.tool.toolsearch.ToolSearchTool; +import org.springframework.ai.tool.toolsearch.index.regex.RegexToolIndex; + +import java.util.List; +import java.util.Map; + +import static com.ankurm.toolcalling.support.TestSupport.toolNamesOfferedTo; +import static org.assertj.core.api.Assertions.assertThat; + +/** + * The point of {@code ToolSearchToolCallingAdvisor}: with a 230-tool library registered on the + * client, the model is offered exactly one tool -- {@code tool_search_tool} -- on the first call, + * not all 230. Only after it searches does the specific tool it needs appear on the next call. See + * {@code com.ankurm.toolcalling.config.LargeToolLibrary} for where the 230 comes from and + * {@code ToolLibrarySizeTest} for the exact count. + */ +class ToolSearchProgressiveDisclosureTest { + + private static final String SESSION_ID = "progressive-disclosure-test-session"; + + @Test + void modelIsOfferedOnlyTheSearchToolUntilItSearches() { + List library = LargeToolLibrary.buildAll(); + + ScriptedChatModel model = ScriptedChatModel.builder() + // The tool's real, model-facing name is "toolSearchTool" -- @Tool(name = "toolSearchTool") + // on ToolSearchTool.toolSearchTool(...), confirmed by disassembling the class rather than + // guessing from the advisor's class name. The advisor's own sessionId parameter is not part + // of the schema either: it is read out of ToolContext, not a model-supplied argument. + .thenCallTools(new AssistantMessage.ToolCall("call-1", "function", "toolSearchTool", + "{\"query\":\"look up an invoice\"}")) + .thenCallTools(new AssistantMessage.ToolCall("call-2", "function", "finance_lookup_invoice", + "{\"id\":\"INV-1001\"}")) + .thenRespond("Invoice INV-1001 was found in the finance system.") + .build(); + + ChatClient client = ChatClientFactory.toolSearchClient(model, library, new RegexToolIndex(), 5); + + // The session id the advisor uses to key its ToolIndex lookups comes from the + // ChatClientRequest's advisor-param context (AdvisorSpec.param), not from + // ChatOptions.getToolContext() / ChatClient.toolContext() -- confirmed by disassembling + // ToolSearchToolCallingAdvisor.initializeSession, which reads + // ChatClientRequest.context() directly rather than the request's ToolCallingChatOptions. + String answer = client.prompt() + .user("Can you look up invoice INV-1001 for me?") + .advisors(a -> a.param(ToolSearchTool.TOOL_SEARCH_TOOL_SESSION_ID_KEY, SESSION_ID)) + .call() + .content(); + + List offeredOnFirstCall = toolNamesOfferedTo(model.capturedPrompts().get(0)); + List offeredAfterSearch = toolNamesOfferedTo(model.capturedPrompts().get(1)); + + try (Transcript t = new Transcript("04-tool-search-progressive-disclosure.txt", + "ToolSearchToolCallingAdvisor: one tool offered up front instead of the whole library")) { + t.line("tools registered on the ChatClient: %d", library.size()); + t.blank(); + t.line("tools OFFERED to the model on call 1 (before any search): %d", offeredOnFirstCall.size()); + t.line(" %s", offeredOnFirstCall); + t.blank(); + t.line("tools OFFERED to the model on call 2 (after it searched for \"look up an invoice\"): %d", + offeredAfterSearch.size()); + t.line(" %s", offeredAfterSearch); + t.blank(); + t.line("final answer: %s", answer); + + assertThat(offeredOnFirstCall).containsExactly("toolSearchTool"); + assertThat(offeredAfterSearch).contains("finance_lookup_invoice"); + assertThat(offeredAfterSearch.size()) + .as("progressive disclosure should still offer far fewer than the full %d-tool library", + library.size()) + .isLessThan(10); + assertThat(answer).isEqualTo("Invoice INV-1001 was found in the finance system."); + } + } +} diff --git a/tool-calling/src/test/java/com/ankurm/toolcalling/support/ScriptedChatModel.java b/tool-calling/src/test/java/com/ankurm/toolcalling/support/ScriptedChatModel.java new file mode 100644 index 0000000..aeac3dd --- /dev/null +++ b/tool-calling/src/test/java/com/ankurm/toolcalling/support/ScriptedChatModel.java @@ -0,0 +1,117 @@ +package com.ankurm.toolcalling.support; + +import org.springframework.ai.chat.messages.AssistantMessage; +import org.springframework.ai.chat.model.ChatModel; +import org.springframework.ai.chat.model.ChatResponse; +import org.springframework.ai.chat.model.Generation; +import org.springframework.ai.chat.prompt.ChatOptions; +import org.springframework.ai.chat.prompt.Prompt; +import org.springframework.ai.model.tool.ToolCallingChatOptions; + +import java.util.ArrayDeque; +import java.util.ArrayList; +import java.util.Deque; +import java.util.List; +import java.util.Queue; +import java.util.concurrent.CopyOnWriteArrayList; + +/** + * A hand-written {@link ChatModel} that returns a pre-programmed queue of {@link ChatResponse}s, + * one per {@link #call(Prompt)} invocation, instead of calling a real LLM API. {@code ChatModel} + * has exactly one abstract method -- confirmed with {@code javap} against + * {@code spring-ai-model-2.0.1.jar}, everything else on the interface has a default implementation + * -- so this is the entire surface a deterministic test double needs to implement. + * + *

{@link org.springframework.ai.chat.client.advisor.ToolCallingAdvisor ToolCallingAdvisor} (and + * its subclass, {@code ToolSearchToolCallingAdvisor}) drive the tool-calling loop by calling the + * underlying {@link ChatModel} once per round: once to get the model's first response (which may + * contain tool calls), then once more per round of tool results fed back in, until a response + * comes back with no tool calls. Queuing responses here lets a test assert the exact shape of that + * loop -- how many rounds it took, what tool calls appeared, what the final answer was -- without + * an API key, network access, or the nondeterminism of an actual model. + * + *

Every {@link Prompt} the advisor sends is recorded in {@link #capturedPrompts()} so a test can + * also assert on what the advisor sent back on the next round -- in particular, that a + * {@code ToolResponseMessage} was appended after tool execution. + */ +public class ScriptedChatModel implements ChatModel { + + private final Queue script; + private final List capturedPrompts = new CopyOnWriteArrayList<>(); + + private ScriptedChatModel(Deque script) { + this.script = script; + } + + public static Builder builder() { + return new Builder(); + } + + @Override + public ChatResponse call(Prompt prompt) { + this.capturedPrompts.add(prompt); + ChatResponse next = this.script.poll(); + if (next == null) { + throw new IllegalStateException( + "ScriptedChatModel ran out of queued responses after " + this.capturedPrompts.size() + + " calls. Prompts so far: " + this.capturedPrompts); + } + return next; + } + + public List capturedPrompts() { + return List.copyOf(this.capturedPrompts); + } + + /** + * {@code DefaultChatClientUtils} builds every outgoing {@link Prompt}'s options from + * {@code chatModel.getOptions().mutate()} -- not {@code getDefaultOptions()}, which is a + * separate default method nobody in the request-building path actually calls. Confirmed by + * disassembling both: {@link ChatModel#getOptions()}'s default body is a bare + * {@code ChatOptions.builder().build()}, a plain {@link ChatOptions} that is not a + * {@link ToolCallingChatOptions}. Since {@code ToolCallingAdvisor.adviseCall} starts with an + * {@code instanceof ToolCallingChatOptions} check on that exact object and falls straight + * through to the underlying model with no tool loop at all when it fails, leaving this method's + * default in place silently turns every tool call in this repository into a no-op -- confirmed + * the hard way, by a first version of this class that overrode {@code getDefaultOptions()} + * instead and watched every test below get back an empty answer after exactly one model call. + * Real providers (OpenAI, Anthropic) return their own {@code ToolCallingChatOptions} + * implementation from {@code getOptions()} for the same reason. + */ + @Override + public ChatOptions getOptions() { + return ToolCallingChatOptions.builder().build(); + } + + public int callCount() { + return this.capturedPrompts.size(); + } + + public static final class Builder { + + private final Deque script = new ArrayDeque<>(); + + private Builder() { + } + + /** Queues a plain-text final answer with no tool calls -- ends the tool-calling loop. */ + public Builder thenRespond(String text) { + this.script.add(new ChatResponse(List.of(new Generation(new AssistantMessage(text))))); + return this; + } + + /** Queues an assistant turn that calls one or more tools, continuing the loop. */ + public Builder thenCallTools(AssistantMessage.ToolCall... toolCalls) { + AssistantMessage message = AssistantMessage.builder() + .content("") + .toolCalls(List.of(toolCalls)) + .build(); + this.script.add(new ChatResponse(List.of(new Generation(message)))); + return this; + } + + public ScriptedChatModel build() { + return new ScriptedChatModel(new ArrayDeque<>(this.script)); + } + } +} diff --git a/tool-calling/src/test/java/com/ankurm/toolcalling/support/TestSupport.java b/tool-calling/src/test/java/com/ankurm/toolcalling/support/TestSupport.java new file mode 100644 index 0000000..70cbf18 --- /dev/null +++ b/tool-calling/src/test/java/com/ankurm/toolcalling/support/TestSupport.java @@ -0,0 +1,35 @@ +package com.ankurm.toolcalling.support; + +import org.springframework.ai.chat.prompt.Prompt; +import org.springframework.ai.model.tool.ToolCallingChatOptions; + +import java.util.List; + +/** + * Small shared helpers for reading back what the advisor actually sent to the (scripted) model, so + * tests can assert on the wire shape of a request rather than trusting the advisor did the right + * thing. + */ +public final class TestSupport { + + private TestSupport() { + } + + /** + * The names of every tool the model was told about on a given call -- read from + * {@link ToolCallingChatOptions#getToolCallbacks()} on the {@link Prompt}'s options, not from + * anything the advisor logs. This is what makes the progressive-disclosure test a proof rather + * than an assertion about behaviour nobody can see: a plain {@code ToolCallingAdvisor} puts + * every registered tool here on every call, and {@code ToolSearchToolCallingAdvisor} puts only + * {@code tool_search_tool} here until the model has searched for something more specific. + */ + public static List toolNamesOfferedTo(Prompt prompt) { + if (!(prompt.getOptions() instanceof ToolCallingChatOptions toolOptions)) { + throw new IllegalStateException("Prompt options are not ToolCallingChatOptions: " + prompt.getOptions()); + } + return toolOptions.getToolCallbacks().stream() + .map(callback -> callback.getToolDefinition().name()) + .sorted() + .toList(); + } +} diff --git a/tool-calling/src/test/java/com/ankurm/toolcalling/support/Transcript.java b/tool-calling/src/test/java/com/ankurm/toolcalling/support/Transcript.java new file mode 100644 index 0000000..4434cbe --- /dev/null +++ b/tool-calling/src/test/java/com/ankurm/toolcalling/support/Transcript.java @@ -0,0 +1,47 @@ +package com.ankurm.toolcalling.support; + +import java.io.IOException; +import java.io.PrintWriter; +import java.io.StringWriter; +import java.nio.file.Files; +import java.nio.file.Path; + +/** + * Writes a numbered transcript under {@code output/} and echoes it to the console. Every console + * block quoted in the article comes out of one of these files verbatim. + */ +public final class Transcript implements AutoCloseable { + + private final Path path; + private final StringWriter buffer = new StringWriter(); + private final PrintWriter out = new PrintWriter(buffer); + + public Transcript(String fileName, String title) { + this.path = Path.of("output", fileName); + out.println("# " + title); + out.println(); + } + + public Transcript line(String format, Object... args) { + out.println(args.length == 0 ? format : String.format(format, args)); + return this; + } + + public Transcript blank() { + out.println(); + return this; + } + + @Override + public void close() { + out.flush(); + try { + Files.createDirectories(path.getParent()); + Files.writeString(path, buffer.toString()); + } + catch (IOException e) { + throw new IllegalStateException("could not write " + path, e); + } + System.out.print(buffer); + } +}