Versions, and an honest limit. Spring Boot 4.1.1, Spring AI 2.0.1, Resilience4j 2.4.0, Bucket4j 8.21.0 and Java 25. The module is llm-gateway in asmhatre/spring-ai. I did not call any vendor’s API: the tests use a scripted model or a local server that speaks the OpenAI and Anthropic wire formats. The command that starts a live gateway with real keys is in the README, and I did not run it.
What you need
A JDK 25 and Maven 3.9 or later. That is all. The older version of this page also asked for Docker and Redis; the module keeps its state in memory, so neither is needed, and the trade-off is that breakers, caps and the cache are per instance. The article’s last section says what that means for several replicas.Going deeper: why Redis is gone
The earlier code stored rate-limit buckets and the cache in Redis. I could not run Redis in the environment where I tested this, and I do not publish behaviour I have not seen, so the module uses Bucket4j and a list in memory instead. Bucket4j has Redis and other backends if you want shared limits, but I have not tested them here, and the breaker state would still be per replica.
README.md— the module overview, including the section on what is not covered.- The article — the “what is not here” box.
Run it in one command
Clone the repository and run the script. It deletestarget/, runs every test and leaves the results in the output/ directory. The tests are the demos: each one writes a transcript with the numbers it just asserted, so a transcript that is out of date would fail the build. The script is run-all.sh.
git clone https://ankurm.com/git.app/asmhatre/spring-ai.git
cd spring-ai/llm-gateway
scripts/run-all.shoutput/ after you edit the code is a real change in behaviour, and the diff is the review.
Where things are
There are eight classes you need to know, and each does one job. The first column links to the file; the article section that explains it is named in the last column.| File | Job | Article section |
|---|---|---|
Gateway.java | Token limit, cache, the ChatClient with the tool advisor, metering | A gateway is one door |
Failover.java | A ChatModel that tries providers in order, with a breaker and a budget hold per attempt | The tool loop is above the model |
Routes.java | Hint to provider chain; an unknown hint is an error | Route by a hint |
Failures.java | HTTP status out of the vendor SDK exceptions | Fail over for failures another provider can fix |
Budgets.java | Dollar caps: reserve before the call, settle after | Caps |
TokenLimiter.java | Tokens per hour per tenant | Caps |
SemanticCache.java | Embedding cache keyed by tenant and feature | A semantic cache |
GatewayController.java | The HTTP endpoint and its status codes | What the caller sees |
application.yml, which is also the file you edit to try your own setup.
The configuration is the product
A route is a list of provider names, in the order they are tried. A tenant is a name with a token limit and a dollar budget, and thedefaults apply to any tenant not listed. This is the whole file:
# Providers are plain data. Prices are dollars per million tokens (read 2026-10-09, see CostTest in the providers module).
gateway:
providers:
- { name: openai, type: openai, base-url: "https://api.openai.com/v1", api-key: "${OPENAI_API_KEY:unset}", model: gpt-6.1-sol, input-price: "2.00", output-price: "10.00" }
- { name: anthropic, type: anthropic, base-url: "https://api.anthropic.com", api-key: "${ANTHROPIC_API_KEY:unset}", model: claude-sonnet-5-5, input-price: "2.00", output-price: "10.00" }
routes:
smart: [openai, anthropic]
fast: [anthropic, openai]
defaults: { budget-micros: 5000000, tokens-per-hour: 100000 }
tenants:
free-tier-tenant: { budget-micros: 50000, tokens-per-hour: 10000 }
breaker: { sliding-window-size: 10, minimum-calls: 5, failure-rate-threshold: 50, wait-open-millis: 30000 }base-url must end in /v1; without it the request goes to a path that does not exist, and the failure looks like a provider outage. Prices are dollars per million tokens. And minimum-calls is set on purpose: Resilience4j’s default is 100, which silently stops a window of 10 from opening until the eleventh request (see 03-circuit-breaker.txt).
Going deeper: the dependencies
The
pom.xml is short. Spring Boot 4.1.1 is the parent; the Spring AI BOM manages the three Spring AI artifacts (spring-ai-client-chat, spring-ai-openai, spring-ai-anthropic); Resilience4j (resilience4j-circuitbreaker, used programmatically, no annotations) and Bucket4j (bucket4j_jdk17-core) are pinned by properties. There is no Redis, Micrometer or actuator dependency.
pom.xml— the full file.- ChatClient on Spring Boot 4.1 — the client the gateway builds on.
What the gateway answers
This is the output of the HTTP test,10-http.txt, written by HttpTest.java. It starts the real application on a random port, with the real OpenAI and Anthropic model classes pointed at a local fake. Each line is one request:
1. normal call -> 200 {"content":"30 days.","provider":"openai","model":"gpt-6.1-sol","promptTokens":47,"completionTokens":52,"costMicros":614,"modelCalls":1,"servedFromCache":false,"trail":["openai: ok"]}
2. openai returns 503 -> 200 {"content":"30 days.","provider":"anthropic","model":"claude-sonnet-5-5","promptTokens":47,"completionTokens":52,"costMicros":614,"modelCalls":1,"servedFromCache":false,"trail":["openai: failed with status 503","anthropic: ok"]}
3. both providers down -> 503 Retry-After=30 No provider could answer: [openai: failed with status 503, anthropic: failed with status 529]
4. unknown hint -> 400 Unknown model hint 'locla'. Known hints: [fast, smart]
5. tool not allow-listed -> 400 Tool 'drop_tables' is not on the gateway allow-list [refund_order]
6. tenant over its cap -> 429 Budget for poor would be exceeded: spent 0 of 1000 microdollars, this call may cost up to 2574
7. body claims tenant acme, header says poor -> 429 (the header wins)
8. no X-Tenant-Id header -> 400trail of what was tried. Case 2 is a failover: the first provider returned 503, and the answer came from the second. Case 3 is every provider down: a 503 with a Retry-After header. Cases 4 and 5 are requests that name something that does not exist, and case 6 is a tenant over its cap; in all three no provider is called. The header decides the tenant (case 7), and a missing header is a 400 (case 8).
The older page showed a cache hit with a latency of 41 ms against 1,240 ms for a normal call, and a Prometheus metric. Those numbers came from no run I can reproduce, so they are gone. The cache is exercised in its own test instead, and the gateway returns cost in the response rather than as a metric.
Going deeper: trying it by hand
With real keys,
mvn spring-boot:run starts the gateway on port 8080, and the request below sends the first case from the transcript. It needs OPENAI_API_KEY and ANTHROPIC_API_KEY in the environment. I did not run this against real providers; the same request through the test uses the local fake.
OPENAI_API_KEY=... ANTHROPIC_API_KEY=... mvn spring-boot:run
curl -X POST localhost:8080/v1/gateway/complete \
-H 'X-Tenant-Id: acme' -H 'Content-Type: application/json' \
-d '{"featureTag":"support","userMessage":"What is your return policy?","modelHint":"smart","maxTokens":256}'README.md. The gateway trusts the X-Tenant-Id header, so put it behind something that sets the header and do not expose it publicly.
README.md— quickstart, versions and the index of output files.HttpTest.java— how the fake vendors are wired in.
Which transcript answers which question
Each file inoutput/ is the evidence for one claim in the article. If you only want one answer, open that file.
| Question | Transcript |
|---|---|
| Does a typo in the hint go to a cloud provider? | 01-routing.txt |
| Which statuses fail over, and which stop? | 02-failover.txt |
| When does the breaker really open? | 03-circuit-breaker.txt |
| How many times does a tool run when failover happens mid loop? | 04-tool-loop.txt |
| What does the second provider receive? | 05-wire.txt |
| What do the vendor SDKs throw? | 06-real-exceptions.txt |
| What does a cap do to a request and to a tool loop? | 07-budget.txt |
| Does a cap hold under 64 simultaneous requests? | 07b-budget-concurrency.txt |
| How is the token limit settled and refilled? | 08-rate-limit.txt |
| Does a near-identical question hit the cache? | 09-cache.txt |
Should you even do this?
Run it, read it, then decide. This module is a teaching and testing vehicle, not a drop-in service. It has no authentication, no streaming and no shared state, it was never run against a real provider, and its semantic cache was tested with a toy embedding model. If you have one service and one provider, the article explains why a shared library is a better start than a service.
Frequently asked questions
Can I run the tests without Docker or API keys?
Yes. Nothing in the module needs either. The only network use is Maven downloading dependencies on the first build.Where did the Redis, Kubernetes and Prometheus parts go?
They were in the earlier version and are not in this one, because I could not run them and will not publish unverified setup. The article lists them under what it did not test.How do I add a provider?
Add an entry undergateway.providers with its type, URL, key, model and prices, then add its name to a route. The two types the module understands are openai and anthropic; anything else needs a new branch in GatewayConfig.java.
Further reading
- The LLM Gateway Pattern for Java Microservices — the design, the failures and the numbers.
- The code: the
llm-gatewaymodule of asmhatre/spring-ai and its output directory. - Anthropic Claude vs OpenAI vs Gemini in Spring AI 2.0 — per-provider options and cost.
- Tool calling in Spring AI 2.0 — the advisor the gateway sits under.
- Spring AI advisors reference
No Comments yet!