AI & Machine Learning

LLM Gateway Pattern for Java Microservices with Spring AI 2.0: Routing, Failover and Cost Caps

Build and test an LLM gateway on Spring AI 2.0: hint-based routing, failover that works with the new tool-calling advisor, one circuit breaker per provider, token and dollar caps, and a semantic cache. Every number comes from a run.

Ankur Apr 14, 2026 27 min read

Every team building AI features meets the same three problems sooner or later. One provider has a bad afternoon and a feature goes down with it. A loop calls a model ten thousand times overnight and the bill arrives in the morning. Two teams each paste their own API key into their own service and nobody can say who spent what. An LLM gateway is the usual answer: one internal service that every other service calls instead of calling a provider directly, so the rules live in one place. This article builds a small one on Spring AI 2.0 and tests the parts that are easy to get wrong: routing, failover, circuit breakers, spending caps, and a cache. There is one thing that is genuinely new in Spring AI 2.0 and changes where the failover code has to go: the loop that runs your tools is now an advisor, not part of the model. That gets its own section. The detail is in expandable boxes, so you can read straight through or open only what you need.

Versions, and an honest limit. Spring Boot 4.1.1, Spring AI 2.0.1, Resilience4j 2.4.0, Bucket4j 8.21.0 and Java 25. The code is the llm-gateway module of asmhatre/spring-ai, and every console block below is quoted from a file under its output/ directory. I did not call any vendor’s API. Where a test needs the real Spring AI OpenAI and Anthropic classes, they talk to a local server that answers in both wire formats, so the requests they send and the exceptions they throw are real. Everywhere else a scripted stand-in plays the provider. Token counts are numbers the tests chose, the cache test uses a toy embedding model, and nothing here measures latency, answer quality or a real provider’s behaviour. Prices are the illustrative sheet used in the provider comparison, read on 9 October 2026.

A gateway is one door, and every rule stands in it

Without a gateway, every service that uses a model carries its own copy of the same decisions: which provider, what to do when it fails, how much it may spend. The copies drift. With one, a service sends a single request and a tenant name, and the gateway decides the rest. The picture below is the whole design; the rest of the article takes the boxes one at a time.

The order matters and is the same every time: the cheap checks (the token limit, the cache) happen first, so a request that will be refused or answered from memory never touches a provider. The failover box is where the interesting work happens, and it is deliberately the part that talks to the providers. A caller only ever sees one of a few outcomes: an answer, a 429 when a cap is hit, a 400 when the request names something that does not exist, or a 503 when every provider is down.

Going deeper: the shape of the code

The module is small on purpose. Gateway.java is the one method callers use; Failover.java is a Spring AI ChatModel that wraps the providers; Routes.java, Budgets.java, TokenLimiter.java and SemanticCache.java are the rest. Providers, prices, routes and tenants are plain data in application.yml. Two older versions of this article, a design post and a code post, described a Redis-backed variant; this version keeps the same layers but keeps all state in memory, because that is what I could run and test here. The last section says what that costs you.

Module README — versions, files and the output index.
The runnable-code companion — quickstart and the HTTP transcript.

Route by a hint, and make a typo an error

A calling service should not name a model. It names an intent, a hint such as smart, fast or local, and the gateway maps it to an ordered list of providers. That mapping is a table in configuration. Changing which provider is first, or adding a cheaper one, is a config change and no calling service is touched. This is the whole of the lookup in Routes.java:

    public List<Target> get(String hint) {
        List<Target> t = table.get(hint);
        if (t == null) {
            throw new UnknownHintException(hint, table.keySet());
        }
        return t;
    }

The earlier version of this article returned the smart chain for any hint it did not recognise. That looks friendly and it is a trap: a request tagged locla (a typo for local, the hint that means “never leave our network”) silently went to a cloud provider. Here an unknown hint throws. The first block below is the new behaviour on a typo; the second is what the old default did (01-routing.txt, from RoutingTest.java):

hint local  -> answered by local; cloud calls made for it: 0

old registry, hint "locla" (typo for local): goes to claude, a cloud provider
this Routes, hint "locla": Unknown model hint 'locla'. Known hints: [local, smart]

The same test also checks the part of routing that is easy to forget. Each provider in a chain gets its own model id, and the portable settings (here the maximum number of tokens) are copied to each. A request for local made zero calls to the cloud targets.

A default route is a data-sovereignty decision. If any hint means “this data must stay inside”, an unknown hint must never fall through to a provider outside. Fail closed.

Going deeper: why each target rebuilds its options

When the gateway calls a provider it does not pass the caller’s options straight through. It starts from that provider’s default options (which carry its own model id), then copies over the portable settings and the tool callbacks. That is forTarget:

    static Prompt forTarget(Prompt prompt, Target t) {
        ChatOptions in = prompt.getOptions();
        ToolCallingChatOptions.Builder<?> b = (ToolCallingChatOptions.Builder<?>) t.chat().getOptions().mutate();
        b.model(t.model());
        if (in != null) {
            if (in.getMaxTokens() != null) {
                b.maxTokens(in.getMaxTokens());
            }
            if (in.getTemperature() != null) {
                b.temperature(in.getTemperature());
            }
            if (in.getTopP() != null) {
                b.topP(in.getTopP());
            }
            if (in.getStopSequences() != null) {
                b.stopSequences(in.getStopSequences());
            }
            if (in instanceof ToolCallingChatOptions tools) {
                b.toolCallbacks(tools.getToolCallbacks());
                b.toolContext(tools.getToolContext());
            }
        }
        return prompt.mutate().chatOptions(b.build()).build();
    }

The reason is the options trap from the provider comparison: an options object built for one vendor is the wrong class for the next, and with Anthropic it silently changed the model. Building the options per target avoids both. I did not test every provider-specific setting through this path; only the model id, maxTokens, temperature, top-p, stop sequences and tool callbacks are copied.

RoutingTest.java — the scenarios above.
01-routing.txt — the full capture.

Fail over for failures another provider can fix

When a provider fails, the gateway tries the next one. The skill is in deciding which failures qualify. An overloaded provider (429, 529), a timeout (408) or a server error (5xx) might well work elsewhere. A malformed request (400), a wrong key (401) or a missing model (404) will fail the same way everywhere, and trying another provider only hides a bug you need to see. Here is every case, with a scripted first provider (02-failover.txt, from FailoverTest.java):

status   second tried?  outcome      trail
408      true           answered     [first: failed with status 408, second: ok]
429      true           answered     [first: failed with status 429, second: ok]
500      true           answered     [first: failed with status 500, second: ok]
503      true           answered     [first: failed with status 503, second: ok]
400      false          rethrown     [first: rejected with status 400, not retried]
401      false          rethrown     [first: rejected with status 401, not retried]
404      false          rethrown     [first: rejected with status 404, not retried]

both down: No provider could answer: [a: failed with status 503, b: failed with status 429]

The four transient statuses reach the second provider; the three client errors stop at the first with the original exception. When every provider fails, the caller gets one exception that lists the whole trail, which is the line you want in an incident review. That table used a stand-in for the provider. The next one uses the real Spring AI OpenAI and Anthropic classes against a local server that answers each status, to check that the gateway can read a status out of the real exceptions (06-real-exceptions.txt):

vendor     status   exception                                            transient?  HTTP requests for one call
openai     400      com.openai.errors.BadRequestException                false       1
openai     401      com.openai.errors.UnauthorizedException              false       1
openai     429      com.openai.errors.RateLimitException                 true        1
openai     503      com.openai.errors.InternalServerException            true        1
anthropic  400      com.anthropic.errors.BadRequestException             false       1
anthropic  529      com.anthropic.errors.InternalServerException         true        1

The two vendors throw unrelated exception families, so Failures.java walks the cause chain for a status. The last column matters more than it looks: one gateway attempt made exactly one HTTP request, because the gateway builds the models with the SDKs’ own retries turned off. If you leave the SDK defaults on, each attempt can become several, and your retry policy is no longer the one you wrote down.

Decide where retries live, once. The module puts all of them in the gateway’s failover and none inside the vendor SDKs. The provider comparison measured what happens otherwise: with Gemini’s two retry layers, one failed call became nine requests.

Going deeper: what a status can and cannot tell you

A status code is a blunt tool. A 529 from Anthropic and a 503 from OpenAI both mean “try later”, but a 429 can mean your own rate limit (another provider may help) or an account-wide one (it may not). The gateway treats them all as transient. Network failures with no status (a dropped connection) are also treated as transient. Timeouts configured on the HTTP client are not set here; set them, because a hung connection is a failure the breaker can only see once it ends.

Failures.java — the classification.
WireTest.java — the real-exception test.
Anthropic vs OpenAI vs Gemini in Spring AI 2.0 — retries, options and caching per provider.

The Spring AI 2.0 twist: the tool loop is above the model

A model can answer, or it can ask for a tool to be run (“look up this order”) and wait for the result. Something has to run the tool, put the result back in the conversation, and call the model again. In Spring AI 2.0 that something is ToolCallingAdvisor, an advisor in the ChatClient chain, and it sits above the chat model. Each trip around the loop is one more call to the model underneath. That placement decides where failover must go. Suppose the first provider asks for a tool, the tool runs (it issues a refund, say), and then the first provider fails on the second call. If your failover wraps the whole ChatClient call, it starts again from the top on the next provider, and the model asks for the tool again. If your failover is the chat model, only the failed call moves.

On the left, the advisor has one model to talk to and does not know that two providers are behind it. On the right, each provider has its own advisor, so a failure throws away the loop’s progress. The test runs both with the same scripted providers and a tool that counts how often it runs (04-tool-loop.txt, from ToolLoopPlacementTest.java):

A. failover below the advisor (this module)
   tool executions:  1
   trail:            [openai: ok, openai: failed with status 503, claude: ok]
   answer:           Refunded as refund-A17-1 (from claude)
   what claude was sent: UserMessage, AssistantMessage, ToolResponseMessage

B. retry above the advisor (one ChatClient per provider)
   tool executions:  2
   answer:           Refunded as refund-A17-2

One execution against two. The second provider’s input in the first case is the original question, the assistant’s tool request and the tool’s response, which is the conversation up to the point of failure. A scripted stand-in cannot tell you whether a different vendor accepts a tool request that another vendor’s model made. So the next test uses the real classes: OpenAI asks for the tool and then fails with a 503, and the real Anthropic model takes over (05-wire.txt):

trail: [openai: ok, openai: failed with status 503, anthropic: ok]
answered by anthropic; tool executions 1; requests: openai 2, anthropic 1

request 1 to OpenAI: model=gpt-6.1-sol, max_tokens=256, tools=[refund_order]
request to Anthropic: model=claude-sonnet-5-5 max_tokens=256
  conversation it received: [assistant:tool_use(id=call_1, name=refund_order), user:tool_result(tool_use_id=call_1)]

The Anthropic request carries the other vendor’s tool call as a tool_use block with the same id and the result as a tool_result, and the tool ran once. That is the translation Spring AI does between wire formats. It is what the request contained; whether the real Anthropic service would accept it I did not test, which is why this sits in the “did not test” list at the end. The failover class that makes this work is Failover.java:

    public ChatResponse call(Prompt prompt) {
        CallContext ctx = CallContext.CURRENT.isBound() ? CallContext.CURRENT.get() : CallContext.anonymous();
        RuntimeException last = null;
        for (Target t : targets) {
            CircuitBreaker breaker = breakers.circuitBreaker(t.name());
            if (!breaker.tryAcquirePermission()) {
                ctx.meter().trail(t.name() + ": circuit open, not called");
                continue;
            }
            long estimate = t.price().micros(estimatePromptTokens(prompt), maxTokens(prompt, t));
            Budgets.Reservation reservation;
            try {
                reservation = budgets.reserve(ctx.tenant(), estimate);
            }
            catch (BudgetExceededException e) {
                breaker.releasePermission();
                throw e;
            }
            long started = System.nanoTime();
            try {
                ChatResponse response = t.chat().call(forTarget(prompt, t));
                breaker.onSuccess(System.nanoTime() - started, TimeUnit.NANOSECONDS);
                Usage u = response.getMetadata() == null ? null : response.getMetadata().getUsage();
                boolean missing = u == null || u.getTotalTokens() == null || u.getTotalTokens() == 0;
                long in = missing ? estimatePromptTokens(prompt) : u.getPromptTokens();
                long out = missing ? 0 : u.getCompletionTokens();
                long actual = t.price().micros(in, out);
                reservation.settle(actual);
                ctx.meter().record(t, in, out, actual, missing);
                ctx.meter().trail(t.name() + ": ok");
                return response;
            }
            catch (RuntimeException e) {
                reservation.release();
                if (Failures.isTransient(e)) {
                    breaker.onError(System.nanoTime() - started, TimeUnit.NANOSECONDS, e);
                    ctx.meter().trail(t.name() + ": failed with status " + Failures.status(e));
                    last = e;
                    continue;
                }
                breaker.releasePermission();
                ctx.meter().trail(t.name() + ": rejected with status " + Failures.status(e) + ", not retried");
                throw e;
            }
        }
        throw new AllProvidersUnavailableException(ctx.meter().trail(), last);
    }

Make the failover class a ChatModel, and return tool-calling options from getOptions(). The advisor decides whether to run its loop by looking at the options of the request. If your wrapper returns plain chat options, the loop is silently skipped and every tool call is ignored. The module’s getOptions() returns a tool-calling options object for that reason.

Going deeper: usage across a loop that changes provider

One more thing the loop changes is how you count tokens. The advisor adds up the usage of every model call and puts the total on the final response. Section C of 04-tool-loop.txt confirms it: two calls reporting 120 + 15 and 160 + 12 tokens give a final total of 280 and 27. That is convenient until a failover happens halfway, because a single total cannot be priced at two prices. Section D shows the difference for a loop whose first call was on a cheaper provider: pricing each model call at its own provider gives 587 microdollars, and pricing the total at the last provider’s rate gives 830. The gateway therefore meters inside the failover class, per call, and reports the sum of the per-call costs.

04-tool-loop.txt — sections A to D.
CallContext.java — the per-request meter, carried in a ScopedValue because the loop runs above the model and cannot pass arguments down.
Tool calling in Spring AI 2.0 — the advisor itself.

One circuit breaker per provider, and when it really opens

Failover alone has a cost: while a provider is down, every request still tries it first, waits for the failure, and only then moves on. A circuit breaker remembers recent failures and, once there are too many, stops calling that provider for a while so requests go straight to the next one. After a wait it lets one request through as a probe; if that works, the breaker closes again. Give each provider its own breaker. A single shared one would let OpenAI’s outage stop traffic to Anthropic too. Failover.java asks the registry for a breaker by provider name before each attempt. I tested the behaviour with Resilience4j 2.4.0 (03-circuit-breaker.txt, from CircuitBreakerTest.java):

old yml (window 10, threshold 50%): minimumNumberOfCalls is 100; the first request that skips openai is #11
this module (window 10, minimum 5, threshold 50%): the first request that skips openai is #6

after 8 requests: openai breaker OPEN, claude breaker CLOSED
openai was contacted 5 times of 8; claude answered 8 times

after the wait, one probe goes to openai (it has recovered): breaker CLOSED, openai calls 1

20 requests rejected with 400: breaker CLOSED, calls the breaker recorded: 0

Three results. First, the configuration in the older version of this article (a window of 10 and a 50% threshold, with no minimum set) opens only on the eleventh request, not after “5 failures” as that article said: the library’s default minimum is 100 calls, which a window of 10 caps at 10. This module sets a minimum of 5 and the breaker skips the provider from the sixth request. Second, once the provider’s breaker is open it is not contacted at all (5 contacts in 8 requests), and the other provider’s breaker stays closed. Third, a flood of 400s leaves the breaker closed with nothing recorded, because the gateway releases the permission instead of counting a caller’s mistake against the provider.

Set minimumNumberOfCalls explicitly. Its default is surprising, and with a small window it silently decides how many failures it takes to open. And do not count 4xx responses as failures: a bad request from one tenant should not take a provider out of rotation for everyone.

Going deeper: what I used and what I left out

The module uses Resilience4j’s CircuitBreakerRegistry programmatically (tryAcquirePermission, onSuccess, onError, releasePermission), not the @CircuitBreaker annotation the earlier version used. The reason is control: the failover class has to decide, per exception, whether to count a failure, release the permission, or rethrow, and doing that by hand is clearer than expressing it through annotation attributes. The half-open state is configured to allow one probe. I used a 200 ms open-state wait in the test so recovery can be observed; the sample configuration uses 30 seconds. Breaker state lives in memory, so with three gateway replicas each keeps its own view of a provider’s health. The earlier article’s Kubernetes comment said the state “lives in Redis”; it does not, in either version.

GatewayConfig.java — the registry built from configuration.
Resilience4j: CircuitBreaker

Caps: tokens per hour, and dollars

Two different limits protect two different things. A token limit per tenant per hour protects capacity and fairness: one tenant cannot use all of it. A dollar budget protects the bill. Count tokens, not requests, because one document summary can cost fifty simple lookups. Both work the same way: take an estimate before the call, then correct it once the real usage is known. For the token limit, TokenLimiter.java does it with a Bucket4j bucket per tenant. The test drives it with a fake clock (08-rate-limit.txt, from RateLimitTest.java):

capacity 10000 tokens per hour
take estimate 1500            -> available 8500
settle: prompt 100 + completion 1000 = 1100 -> available 8900
old rule (refund estimate minus PROMPT tokens only): would have refunded 1400 and left 100 charged

take 500, real total 2000     -> available 6900 (the shortfall is charged, not forgiven)

take 6900                     -> available 0
take 1000                     -> rejected: Token rate limit reached for acme

30 minutes later              -> available 5000 (greedy refill: 10000 per hour)
60 more minutes               -> available 10000 (capped at capacity)

The estimate is charged first, then the real total replaces it: a short answer refunds the difference, and a long one is charged the shortfall instead of being forgiven. The line about the “old rule” is a bug in the earlier version, which refunded the estimate minus the prompt tokens and never charged the completion. With a 1,000-token answer, 1,400 of 1,500 tokens came back and the answer was free. Refill is greedy, so after half an hour half the capacity is back. The dollar budget has a harder problem: concurrency. If you check the balance and record the spend as two steps, many simultaneous requests all pass the check before any of them records, and the cap is meaningless. Budgets.java reserves the estimated cost before the call and settles with the real cost after, so the check and the hold are one step:

    public Reservation reserve(String tenant, long estimateMicros) {
        Account a = accounts.computeIfAbsent(tenant, t -> new Account());
        long cap = capMicros.applyAsLong(tenant);
        if (!a.reserve(estimateMicros, cap)) {
            throw new BudgetExceededException("Budget for " + tenant + " would be exceeded: spent "
                    + a.spent() + " of " + cap + " microdollars, this call may cost up to " + estimateMicros);
        }
        return new Reservation(a, estimateMicros);
    }

To show the difference without relying on timing luck, the test makes every thread pass the check before any records (07b-budget-concurrency.txt):

cap 10000, each call 1000, 64 threads at once
check then record: admitted 64, spent 64000
reserve then settle: admitted 10

Sixty-four requests against a cap that fits ten: check-then-record admitted all 64 and spent 64,000; reserve-then-settle admitted exactly 10. In real life the gap between the check and the record is the duration of the provider call, which is seconds, so the race is not exotic. A reservation is only as good as its estimate, and that has limits worth seeing (07-budget.txt):

cap 5000 microdollars; every call really costs 2000; maxTokens 256 (estimate about 2,570)
  call 1: answered, cost 2000, spent so far 2000
  call 2: answered, cost 2000, spent so far 4000
  call 3: rejected before the provider was called (provider calls so far: 2)

cap 8000; prompt estimated at 100 tokens but the provider counts 3000 (code, other scripts, images do this)
  spent after 2 calls: 12200 (cap 8000); the second call was admitted on its estimate

cap 2700, a two-call tool loop: rejected on the SECOND model call; tool executions so far: 1

The first block is the happy path: the third call is refused before the provider is called, because the held estimate (about 2,570 microdollars from maxTokens 256) would not fit. The second block is the weakness: the prompt estimate is characters divided by four, and when the real prompt is much bigger (code, other languages, images) the call is admitted on its estimate and overshoots; the cap here was 8,000 and 12,200 was spent. The third block is the one that surprised me: a cap hit in the middle of a tool loop refuses the second model call, but the tool from the first call has already run.

A cap is a soft limit. It bounds spend to roughly the cap plus the error in the estimate, and it can stop a tool loop after a side effect. Use real token counts where the provider offers a counting call, keep a margin, and make tools idempotent or reversible if a budget might cut them off.

Going deeper: choices behind the numbers

Prices are held as exact decimals in dollars per million tokens, which is the same as microdollars per token, so all money is a whole number of microdollars rounded up. The reservation uses the target’s own price, so failing over from a cheap provider to an expensive one re-checks the cap at the higher price. A failed attempt releases its hold. The token bucket and the budget are separate on purpose: they fail differently (a tenant at its token limit recovers in an hour, a tenant over its budget does not) and you will want different messages. Both are in memory, per instance; for several replicas they need a shared store, which the next-to-last section discusses.

Price.java — the arithmetic.
BudgetTest.java — the three budget scenarios and the concurrency test.
Custom advisors: token budgets — a per-request cap inside one service.

A semantic cache, and two ways it goes wrong

A normal cache needs the same question word for word. A semantic cache turns each question into a list of numbers (an embedding) and treats two questions as the same when their lists point in nearly the same direction, measured by cosine similarity. If a stored question is close enough, the stored answer is returned and no provider is called. SemanticCache.java does this in memory with a threshold of 0.92. Before the numbers: the embedding model in my test is a toy, a hashed bag of words with no meaning in it. It shows how the threshold and the cache key behave. It says nothing about how a real embedding model scores paraphrases, and you must measure that with your own model and your own questions (09-cache.txt, from CacheTest.java):

probe                                  cosine   hit at 0.92?
same words, new order                  0.985    true
one word different (A18 for A17)       0.970    true
different question, shares a few words 0.395    false

tenant globex asks the stored question: miss
a cache keyed by feature only, tenant globex asks: 30 days, original packaging.

Two lessons. First, the threshold does not know which words matter: a long question with one changed word (order A18 for A17) scores 0.97 and is a hit, so the customer for A18 would be served A17’s answer. A high similarity is not the same as the same question, and the longer the question the less one word moves the score. Second, the cache key includes the tenant. A cache keyed only by feature, the last line, hands one tenant’s answer to another.

Do not cache answers that contain a person’s or tenant’s data across tenants, and do not cache anything keyed by an identifier inside the text. The earlier article claimed a cache cuts model calls by “40 to 60%”. I did not measure any such number, and it depends entirely on how repetitive your questions are.

Going deeper: the embedding API, and what a real deployment needs

One concrete correction to the earlier code: it called embed.embed(text) and treated the result as List<Double>. In Spring AI 2.0.1 EmbeddingModel.embed(String) returns float[] (the first line of 09-cache.txt reads it by reflection), so that code does not compile. The module’s cache scans a list per tenant and feature, which is fine for hundreds of entries and wrong for millions; past that, use a vector store with an index (see choosing a vector store). A real deployment also needs expiry, which this class does not have, and a way to evict an answer that turns out to be wrong.

SemanticCache.java — the cache.
Production RAG with Spring AI — the retrieval side of the same embeddings.

What the caller sees

Everything above sits behind one endpoint, GatewayController.java. The tenant comes from an X-Tenant-Id header that your service mesh or API gateway sets; the request body has no tenant field to spoof. Tools are named in the request but only from an allow-list the gateway owns. Here is the whole application over HTTP with real Spring beans, the real OpenAI and Anthropic models and the local fake at the far end (10-http.txt, from HttpTest.java):

1. normal call            -> 200 {"content":"30 days.","provider":"openai","model":"gpt-6.1-sol","promptTokens":47,"completionTokens":52,"costMicros":614,"modelCalls":1,"servedFromCache":false,"trail":["openai: ok"]}
2. openai returns 503     -> 200 {"content":"30 days.","provider":"anthropic","model":"claude-sonnet-5-5","promptTokens":47,"completionTokens":52,"costMicros":614,"modelCalls":1,"servedFromCache":false,"trail":["openai: failed with status 503","anthropic: ok"]}
3. both providers down    -> 503 Retry-After=30 No provider could answer: [openai: failed with status 503, anthropic: failed with status 529]
4. unknown hint           -> 400 Unknown model hint 'locla'. Known hints: [fast, smart]
5. tool not allow-listed  -> 400 Tool 'drop_tables' is not on the gateway allow-list [refund_order]
6. tenant over its cap    -> 429 Budget for poor would be exceeded: spent 0 of 1000 microdollars, this call may cost up to 2574
7. body claims tenant acme, header says poor -> 429 (the header wins)
8. no X-Tenant-Id header  -> 400

Read the status codes as the contract. 200 with a provider field is your only hint that failover happened (case 2). 503 with Retry-After means every provider failed. A 400 means the request named something that does not exist (a hint, a tool) or lacked the tenant header, and nothing was sent to a provider. A 429 means a cap, and again no provider was called. In case 7 the body claims another tenant and the header wins, so the cap that applied was the header tenant’s.

Going deeper: what is not here

No streaming: the per-request context travels in a ScopedValue, which does not cross the threads a reactive stream runs on, so a streaming endpoint needs a different way to carry the tenant. No Micrometer metrics: the earlier version recorded cost counters for a Grafana dashboard, and this one returns cost in the response instead; the observability article shows the metrics Spring AI already emits. No shared state: breakers, caps and the cache are per instance, so three replicas allow three times a tenant’s token limit unless you move the buckets to a shared store (Bucket4j has Redis and other backends) and accept that the breaker view will differ per replica. No authentication: the gateway trusts the tenant header, so it must not be reachable from outside your network.

HttpTest.java — the eight cases.
Prompt injection defense — what to add at the gateway for untrusted input; I did not test any injection filter here.

Should you even do this?

Not for one service with one provider. A gateway is another service to run, another hop on every call, and another thing that can fail. For a single service spending a few hundred dollars a month, a shared library with the failover class and a budget is enough. Build the service when several teams use models, when an outage of one provider would genuinely hurt, when you need one place for keys and spend, or when an audit asks for one record of every call. If you do build it, the lessons from the tests are the useful part: put failover under the tool advisor, give every provider its own breaker with an explicit minimum, reserve money before the call, and fail closed on routing. What I did not test: any real vendor API, latency or the hop’s cost, streaming, shared state across replicas, a real embedding model, whether the real Anthropic service accepts a tool call another vendor made, authentication, and the Kubernetes deployment the earlier article sketched. The earlier version also had a bar chart claiming failover in under a second against a 30-second timeout and a decision-tree image; I removed both because I could not reproduce the numbers.

Frequently asked questions

Should the gateway be synchronous or asynchronous?

This one is synchronous, and streaming is not implemented. For chat screens you will want streaming, which means rethinking how the tenant context travels (see above). For batch work such as classifying documents, put a queue in front of the gateway and let workers call it; the budget and token limit still apply per call.

How is an LLM gateway different from an API gateway?

An API gateway (Kong, a cloud API gateway) handles general HTTP: authentication, routing, request-count limits, TLS. An LLM gateway understands tokens, providers and cost: it counts tokens, keeps a breaker per provider and prices each call. Run both: the API gateway at the edge, the LLM gateway inside.

How do I stop prompt injection at the gateway?

A gateway is a convenient place for input and output checks, but I have not tested one here. The prompt-injection article measures what a phrase filter, a tool allow-list and output validation each stop. The one defence that is in this module is the tool allow-list: callers name tools but cannot supply them.

Is there an open-source Java gateway to use instead?

I did not survey the field for this rewrite. LiteLLM (Python) is a well-known gateway worth reading for provider abstraction ideas. Building on Spring gives you routing rules you can express in code, such as “never send documents tagged confidential to a cloud provider”.

LiteLLM