Every Java team building AI features eventually faces the same set of problems: a single LLM provider goes down during a critical demo, a runaway loop burns $500 of API credit overnight, or two product teams independently hard-code OpenAI keys into separate microservices and you lose all cost visibility. The LLM Gateway pattern solves all of these — and it’s the most underbuilt component in the Java AI ecosystem today. This post walks through designing and implementing a production-grade LLM Gateway in Spring Boot: multi-provider routing with automatic failover, circuit breaking, rate limiting, cost tracking, semantic caching, and tenant isolation.
What Is an LLM Gateway and Why Do You Need One?
An LLM Gateway is an internal service that sits between your application microservices and the LLM providers (OpenAI, Anthropic, Google Gemini, Ollama, etc.). It centralizes concerns that would otherwise be duplicated in every service that calls an LLM:
- API key management — one place to rotate keys, no keys scattered across microservices
- Multi-provider routing — route to OpenAI by default, fail over to Anthropic, fall back to a local Ollama instance
- Circuit breakers — detect provider degradation and stop hammering a slow endpoint
- Rate limiting — per-tenant and per-model token budgets to prevent cost explosions
- Cost tracking — real-time token usage aggregated by tenant, feature, and model
- Semantic caching — serve near-identical queries from cache, reducing LLM calls by 40–60%
- Prompt logging — audit trail of all LLM interactions for compliance and debugging
Without a gateway, each microservice reinvents these mechanisms inconsistently, or skips them entirely. The gateway makes the right behavior the default.
Architecture Overview
The gateway sits inside your private network as a ClusterIP Kubernetes service. Microservices call it over HTTP; the gateway routes to the best available LLM provider, applies circuit breaking, checks the semantic cache, enforces token budgets, and records cost metrics — all transparently.
Project Setup
Key dependencies (Spring Boot 3.4.5, Spring AI BOM 1.1.0 GA). See the companion code post for the complete pom.xml.
<!-- Spring AI BOM — manages all spring-ai-* versions -->
<!-- Add to <dependencyManagement> with version 1.1.0 -->
<!-- Spring AI: OpenAI (primary provider + embeddings for semantic cache) -->
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-openai-spring-boot-starter</artifactId>
</dependency>
<!-- Spring AI: Anthropic Claude (failover provider) -->
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-anthropic-spring-boot-starter</artifactId>
</dependency>
<!-- Spring AI: Ollama (local data-sovereignty fallback) -->
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-ollama-spring-boot-starter</artifactId>
</dependency>
<!-- Resilience4j 2.4.0: circuit breakers + retry per provider -->
<dependency>
<groupId>io.github.resilience4j</groupId>
<artifactId>resilience4j-spring-boot3</artifactId>
<version>2.4.0</version>
</dependency>
<!-- Bucket4j 0.12.7: token-bucket rate limiting per tenant -->
<dependency>
<groupId>com.giffing.bucket4j.spring.boot.starter</groupId>
<artifactId>bucket4j-spring-boot-starter</artifactId>
<version>0.12.7</version>
</dependency>
<!-- Redis: distributed rate-limit buckets + semantic cache storage -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
Core Abstraction: Provider-Agnostic LLM Request
The gateway exposes a single internal API surface. All microservices send requests in a provider-agnostic format; the gateway handles provider selection and translation internally.
/**
* Provider-agnostic LLM request model.
* Microservices send this to the gateway — they never know which provider handles it.
*
* modelHint: "fast" | "smart" | "local" — router picks the actual model.
* featureTag: identifies the calling feature (e.g. "rag-query", "summarizer")
* used in cost dashboards to break down spending by product area.
*/
public record LlmGatewayRequest(
String tenantId,
String featureTag,
String systemPrompt,
String userMessage,
String modelHint,
int maxTokens,
boolean streaming,
java.util.Map<String, String> metadata
) {
public LlmGatewayRequest {
if (maxTokens <= 0) maxTokens = 1024;
if (modelHint == null) modelHint = "smart";
if (metadata == null) metadata = java.util.Map.of();
}
}
/**
* Provider-agnostic LLM response.
* The providerUsed field tells callers which provider handled the request —
* useful for debugging latency anomalies and cost attribution.
*/
public record LlmGatewayResponse(
String content,
String providerUsed, // "openai", "anthropic", "ollama", "cache"
String modelUsed,
int promptTokens,
int completionTokens,
boolean servedFromCache,
long latencyMs
) {}
Provider Registry: Routing Logic
The registry maps each modelHint to an ordered list of providers. The gateway tries them in priority order, skipping any whose circuit breaker is open. The "local" hint implements data-sovereignty mode — requests tagged with it never leave your network.
import org.springframework.ai.chat.model.ChatModel;
import org.springframework.stereotype.Component;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
/**
* Maps model hints to ordered provider chains.
*
* "smart" → OpenAI GPT-4o → Anthropic Claude → Ollama llama3.1 (degraded fallback)
* "fast" → OpenAI GPT-4o-mini → Ollama llama3.1
* "local" → Ollama llama3.1 only (data-sovereignty mode)
*
* The ordering matters: first provider that responds successfully wins.
* If a circuit breaker is open the resilient caller returns null and the
* next provider in the list is tried automatically.
*/
@Component
public class LlmProviderRegistry {
public record ProviderConfig(
String providerName,
String modelName,
ChatModel chatModel,
int priority
) {}
private final Map<String, List<ProviderConfig>> routingTable;
public LlmProviderRegistry(
org.springframework.ai.openai.OpenAiChatModel openAi,
org.springframework.ai.anthropic.AnthropicChatModel anthropic,
org.springframework.ai.ollama.OllamaChatModel ollama) {
routingTable = new LinkedHashMap<>();
routingTable.put("smart", List.of(
new ProviderConfig("openai", "gpt-4o", openAi, 1),
new ProviderConfig("anthropic", "claude-3-5-sonnet-20241022", anthropic, 2),
new ProviderConfig("ollama", "llama3.1", ollama, 3)
));
routingTable.put("fast", List.of(
new ProviderConfig("openai", "gpt-4o-mini", openAi, 1),
new ProviderConfig("ollama", "llama3.1", ollama, 2)
));
routingTable.put("local", List.of(
new ProviderConfig("ollama", "llama3.1", ollama, 1)
));
}
public List<ProviderConfig> getProvidersForHint(String hint) {
return routingTable.getOrDefault(hint, routingTable.get("smart"));
}
}
Circuit Breaker and Retry with Resilience4j 2.4.0
Each provider gets its own circuit breaker. This is the critical design decision: if OpenAI’s circuit trips, it must not affect Anthropic’s circuit state. Shared-circuit designs are the most common production anti-pattern in gateway implementations — one provider’s outage cascades into all routes.
import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker;
import io.github.resilience4j.retry.annotation.Retry;
import org.springframework.ai.chat.messages.SystemMessage;
import org.springframework.ai.chat.messages.UserMessage;
import org.springframework.ai.chat.model.ChatModel;
import org.springframework.ai.chat.prompt.Prompt;
import org.springframework.stereotype.Component;
/**
* Wraps each provider call in its own named circuit breaker + retry.
* Per-provider circuit breaker isolation is the key design principle here:
* OpenAI tripping must never affect Anthropic's circuit state.
*
* Fallback returns null → the orchestrator tries the next provider.
* Only the Ollama fallback throws, because it is the last resort.
*
* Circuit breaker config (application.yml):
* slidingWindowSize: 10, failureRateThreshold: 50
* waitDurationInOpenState: 30s, permittedCallsInHalfOpenState: 3
*/
@Component
public class ResilientProviderCaller {
@CircuitBreaker(name = "openai", fallbackMethod = "openAiFallback")
@Retry(name = "openai")
public String callOpenAi(ChatModel model, String sys, String user) {
return invoke(model, sys, user);
}
public String openAiFallback(ChatModel m, String s, String u, Exception ex) { return null; }
@CircuitBreaker(name = "anthropic", fallbackMethod = "anthropicFallback")
@Retry(name = "anthropic")
public String callAnthropic(ChatModel model, String sys, String user) {
return invoke(model, sys, user);
}
public String anthropicFallback(ChatModel m, String s, String u, Exception ex) { return null; }
@CircuitBreaker(name = "ollama", fallbackMethod = "ollamaFallback")
public String callOllama(ChatModel model, String sys, String user) {
return invoke(model, sys, user);
}
public String ollamaFallback(ChatModel m, String s, String u, Exception ex) {
throw new AllProvidersUnavailableException("All LLM providers unavailable", ex);
}
private String invoke(ChatModel model, String sys, String user) {
return model.call(new Prompt(java.util.List.of(
new SystemMessage(sys), new UserMessage(user)
))).getResult().getOutput().getText();
}
}
class AllProvidersUnavailableException extends RuntimeException {
AllProvidersUnavailableException(String msg, Throwable cause) { super(msg, cause); }
}
Token-Bucket Rate Limiting per Tenant
Rate-limiting an LLM gateway by token count — not request count — is the only meaningful approach. A single document summarization can consume 50× the tokens of a simple lookup. Request-count limits give false safety and allow budget abuse via large prompts.
The design uses a pre-consume + refund pattern: estimate tokens before the LLM call, deduct them from the bucket, then refund the difference after the actual usage is known. This prevents bypass via request flooding while staying accurate.
import io.github.bucket4j.*;
import io.github.bucket4j.redis.lettuce.cas.LettuceBasedProxyManager;
import org.springframework.stereotype.Service;
import java.time.Duration;
/**
* Per-tenant token-bucket rate limiter (Bucket4j 0.12.7 + Redis).
*
* Each tenant has an independent bucket in Redis, so limits are enforced
* correctly across all gateway instances (distributed-safe).
*
* Tiers:
* FREE: 10,000 tokens / hour
* STANDARD: 100,000 tokens / hour
* ENTERPRISE: no limit
*
* Pre-consume + refund pattern:
* 1. tryConsume(estimatedTokens) — reject if bucket empty
* 2. Call LLM
* 3. refundUnused(estimated - actual) — keeps bucket accurate
*/
@Service
public class TokenBudgetRateLimiter {
private final LettuceBasedProxyManager<String> proxyManager;
private final TenantTierRepository tierRepo;
public TokenBudgetRateLimiter(LettuceBasedProxyManager<String> proxyManager,
TenantTierRepository tierRepo) {
this.proxyManager = proxyManager;
this.tierRepo = tierRepo;
}
public boolean tryConsume(String tenantId, int estimated) {
TenantTier tier = tierRepo.getTier(tenantId);
if (tier == TenantTier.ENTERPRISE) return true;
return proxyManager.builder()
.build(key(tenantId), () -> config(tier))
.tryConsume(estimated);
}
public void refundUnused(String tenantId, int estimated, int actual) {
int surplus = estimated - actual;
if (surplus <= 0) return;
TenantTier tier = tierRepo.getTier(tenantId);
if (tier == TenantTier.ENTERPRISE) return;
proxyManager.builder()
.build(key(tenantId), () -> config(tier))
.addTokens(surplus);
}
private BucketConfiguration config(TenantTier tier) {
long limit = switch (tier) {
case FREE -> 10_000L;
case STANDARD -> 100_000L;
default -> Long.MAX_VALUE;
};
return BucketConfiguration.builder()
.addLimit(Bandwidth.builder()
.capacity(limit)
.refillGreedy(limit, Duration.ofHours(1))
.build())
.build();
}
private String key(String tenantId) {
return "llm-gateway:rl:" + tenantId;
}
}
enum TenantTier { FREE, STANDARD, ENTERPRISE }
Semantic Cache with Embedding Similarity
String caches miss paraphrases entirely. A semantic cache embeds the incoming query and computes cosine similarity against every cached query. If similarity exceeds a threshold (0.92 is a practical default), the cached response is returned — no LLM call, no token cost.
The threshold controls the precision/recall trade-off: 0.95 is very strict (high cache-miss rate, no false hits); 0.88 is more permissive (more hits, occasional wrong answers for edge-case phrasings). Test with your specific domain vocabulary before tuning below 0.90.
import org.springframework.ai.embedding.EmbeddingModel;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.stereotype.Component;
import java.time.Duration;
import java.util.Comparator;
import java.util.List;
import java.util.Optional;
/**
* Semantic response cache backed by Redis.
*
* Write path: embed(query) → store (embedding, response) pair in Redis list
* Read path: embed(query) → scan list → return best match if similarity ≥ threshold
*
* O(n) scan over cached entries. Suitable for up to ~10k entries per tenant/feature key.
* Beyond that, replace the Redis list with a pgvector lookup (see RAG post).
*/
@Component
public class SemanticCache {
private static final double THRESHOLD = 0.92;
private static final Duration TTL = Duration.ofHours(24);
private final EmbeddingModel embed;
private final RedisTemplate<String, CacheEntry> redis;
public SemanticCache(EmbeddingModel embed, RedisTemplate<String, CacheEntry> redis) {
this.embed = embed;
this.redis = redis;
}
public Optional<String> lookup(String tenantId, String feature, String query) {
float[] qv = embedToFloat(query);
List<CacheEntry> entries = redis.opsForList().range(key(tenantId, feature), 0, -1);
if (entries == null || entries.isEmpty()) return Optional.empty();
return entries.stream()
.filter(e -> cosine(qv, e.embedding()) >= THRESHOLD)
.max(Comparator.comparingDouble(e -> cosine(qv, e.embedding())))
.map(CacheEntry::response);
}
public void put(String tenantId, String feature, String query, String response) {
redis.opsForList().rightPush(
key(tenantId, feature),
new CacheEntry(embedToFloat(query), query, response, System.currentTimeMillis())
);
redis.expire(key(tenantId, feature), TTL);
}
private float[] embedToFloat(String text) {
List<Double> v = embed.embed(text);
float[] arr = new float[v.size()];
for (int i = 0; i < v.size(); i++) arr[i] = v.get(i).floatValue();
return arr;
}
private double cosine(float[] a, float[] b) {
double dot = 0, na = 0, nb = 0;
for (int i = 0; i < a.length; i++) {
dot += a[i] * b[i]; na += a[i] * a[i]; nb += b[i] * b[i];
}
return dot / (Math.sqrt(na) * Math.sqrt(nb) + 1e-10);
}
private String key(String t, String f) { return "llm-cache:" + t + ":" + f; }
}
record CacheEntry(float[] embedding, String query, String response, long cachedAt)
implements java.io.Serializable {}
The Gateway Orchestrator
The orchestrator is the single method all microservices call. It enforces the execution order — rate limit → cache → route → record — and ensures that no step is skippable.
import io.micrometer.core.instrument.MeterRegistry;
import org.springframework.stereotype.Service;
import java.util.List;
/**
* LLM Gateway orchestrator: rate-limit → cache → route → record.
*
* The execution order is fixed and non-negotiable:
* 1. Rate limit check — reject cheaply before any expensive operation
* 2. Semantic cache — serve free if hit; full refund to rate-limit bucket
* 3. Provider routing — try in priority order with circuit-breaker isolation
* 4. Cost + cache write — always record cost; always cache success responses
*/
@Service
public class LlmGatewayOrchestrator {
private final LlmProviderRegistry registry;
private final ResilientProviderCaller caller;
private final TokenBudgetRateLimiter rateLimiter;
private final SemanticCache cache;
private final CostTracker cost;
private final MeterRegistry metrics;
public LlmGatewayOrchestrator(LlmProviderRegistry registry,
ResilientProviderCaller caller,
TokenBudgetRateLimiter rateLimiter,
SemanticCache cache,
CostTracker cost,
MeterRegistry metrics) {
this.registry = registry;
this.caller = caller;
this.rateLimiter = rateLimiter;
this.cache = cache;
this.cost = cost;
this.metrics = metrics;
}
public LlmGatewayResponse complete(LlmGatewayRequest req) {
long start = System.currentTimeMillis();
// 1. Rate limit — estimate conservatively (+200 for system prompt overhead)
int estimated = estimateTokens(req.userMessage()) + 200;
if (!rateLimiter.tryConsume(req.tenantId(), estimated)) {
metrics.counter("llm.gateway.rate_limited", "tenant", req.tenantId()).increment();
throw new RateLimitExceededException("Budget exhausted: " + req.tenantId());
}
// 2. Semantic cache lookup
var cached = cache.lookup(req.tenantId(), req.featureTag(), req.userMessage());
if (cached.isPresent()) {
metrics.counter("llm.gateway.cache_hits", "feature", req.featureTag()).increment();
rateLimiter.refundUnused(req.tenantId(), estimated, 0);
return new LlmGatewayResponse(cached.get(), "cache", "cache",
0, 0, true, System.currentTimeMillis() - start);
}
// 3. Provider routing with circuit-breaker failover
List<LlmProviderRegistry.ProviderConfig> providers =
registry.getProvidersForHint(req.modelHint());
String content = null; String usedProvider = null; String usedModel = null;
for (var provider : providers) {
try {
content = switch (provider.providerName()) {
case "openai" -> caller.callOpenAi(provider.chatModel(),
req.systemPrompt(), req.userMessage());
case "anthropic" -> caller.callAnthropic(provider.chatModel(),
req.systemPrompt(), req.userMessage());
case "ollama" -> caller.callOllama(provider.chatModel(),
req.systemPrompt(), req.userMessage());
default -> null;
};
if (content != null) {
usedProvider = provider.providerName();
usedModel = provider.modelName();
metrics.counter("llm.gateway.provider_calls",
"provider", usedProvider,
"tenant", req.tenantId(),
"feature", req.featureTag()).increment();
break;
}
} catch (AllProvidersUnavailableException ex) {
throw ex;
} catch (Exception ex) {
metrics.counter("llm.gateway.provider_failures",
"provider", provider.providerName()).increment();
}
}
if (content == null) throw new AllProvidersUnavailableException("All providers failed", null);
// 4. Record cost and populate cache
int promptTok = estimateTokens(req.systemPrompt() + req.userMessage());
int completeTok = estimateTokens(content);
rateLimiter.refundUnused(req.tenantId(), estimated, promptTok);
cost.record(req.tenantId(), req.featureTag(), usedProvider, usedModel,
promptTok, completeTok);
cache.put(req.tenantId(), req.featureTag(), req.userMessage(), content);
long latency = System.currentTimeMillis() - start;
metrics.timer("llm.gateway.latency",
"provider", usedProvider, "feature", req.featureTag())
.record(java.time.Duration.ofMillis(latency));
return new LlmGatewayResponse(content, usedProvider, usedModel,
promptTok, completeTok, false, latency);
}
// ~4 chars per token — replace with tiktoken-java for billing accuracy
private int estimateTokens(String text) {
return (text == null) ? 0 : text.length() / 4;
}
}
class RateLimitExceededException extends RuntimeException {
RateLimitExceededException(String msg) { super(msg); }
}
REST Controller
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;
/**
* Gateway REST API — deploy inside private network only (ClusterIP).
* Tenant ID arrives from the X-Tenant-Id header set by the service mesh or
* API gateway upstream — never trust the request body for tenant identity.
*
* POST /v1/gateway/complete — synchronous completion
*/
@RestController
@RequestMapping("/v1/gateway")
public class LlmGatewayController {
private final LlmGatewayOrchestrator orchestrator;
public LlmGatewayController(LlmGatewayOrchestrator orchestrator) {
this.orchestrator = orchestrator;
}
@PostMapping("/complete")
public ResponseEntity<LlmGatewayResponse> complete(
@RequestBody LlmGatewayRequest request,
@RequestHeader("X-Tenant-Id") String tenantId) {
// Tenant from header always wins over body — prevents tenant spoofing
LlmGatewayRequest enriched = new LlmGatewayRequest(
tenantId, request.featureTag(), request.systemPrompt(),
request.userMessage(), request.modelHint(), request.maxTokens(),
request.streaming(), request.metadata()
);
return ResponseEntity.ok(orchestrator.complete(enriched));
}
@ExceptionHandler(RateLimitExceededException.class)
public ResponseEntity<String> onRateLimit(RateLimitExceededException ex) {
return ResponseEntity.status(HttpStatus.TOO_MANY_REQUESTS).body(ex.getMessage());
}
@ExceptionHandler(AllProvidersUnavailableException.class)
public ResponseEntity<String> onProviderFailure(AllProvidersUnavailableException ex) {
return ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE)
.body("LLM service temporarily unavailable. Retry in 30 seconds.");
}
}
Cost Tracker
Token usage is converted to microdollar counters (1/1,000,000 USD) and stored in Micrometer. Using integer counters avoids floating-point accumulation errors in Prometheus. A Grafana dashboard subscribed to llm_gateway_cost_microdollars_total grouped by tenant and feature gives you real-time spend visibility.
import io.micrometer.core.instrument.MeterRegistry;
import org.springframework.stereotype.Component;
import java.util.Map;
/**
* Real-time cost tracker via Micrometer counters.
* Rates: approximate April 2026 values — externalise to config in production.
* Key metric: llm.gateway.cost_microdollars tagged by tenant/feature/provider/model.
*/
@Component
public class CostTracker {
private static final Map<String, ModelCost> RATES = Map.of(
"gpt-4o", new ModelCost(5.00, 15.00),
"gpt-4o-mini", new ModelCost(0.15, 0.60),
"claude-3-5-sonnet-20241022", new ModelCost(3.00, 15.00),
"llama3.1", new ModelCost(0.00, 0.00) // self-hosted
);
private final MeterRegistry m;
public CostTracker(MeterRegistry m) { this.m = m; }
public void record(String tenant, String feature, String provider,
String model, int prompt, int completion) {
ModelCost rate = RATES.getOrDefault(model, new ModelCost(0, 0));
double usd = (prompt / 1_000_000.0) * rate.promptPer1M()
+ (completion / 1_000_000.0) * rate.completionPer1M();
m.counter("llm.gateway.cost_microdollars",
"tenant", tenant, "feature", feature,
"provider", provider, "model", model
).increment((long)(usd * 1_000_000));
m.counter("llm.gateway.tokens_prompt",
"tenant", tenant, "model", model).increment(prompt);
m.counter("llm.gateway.tokens_completion",
"tenant", tenant, "model", model).increment(completion);
}
record ModelCost(double promptPer1M, double completionPer1M) {}
}
application.yml — Complete Configuration
# Spring AI 1.1.0 — Spring Boot 3.4.5 — Resilience4j 2.4.0
spring:
application:
name: llm-gateway
ai:
openai:
api-key: ${OPENAI_API_KEY}
chat.options.model: gpt-4o
anthropic:
api-key: ${ANTHROPIC_API_KEY:none}
chat.options.model: claude-3-5-sonnet-20241022
ollama:
base-url: ${OLLAMA_BASE_URL:http://localhost:11434}
chat.options.model: llama3.1
data:
redis:
host: ${REDIS_HOST:localhost}
port: 6379
resilience4j:
circuitbreaker:
instances:
openai:
slidingWindowSize: 10
failureRateThreshold: 50
waitDurationInOpenState: 30s
permittedCallsInHalfOpenState: 3
automaticTransitionFromOpenToHalfOpenEnabled: true
anthropic:
slidingWindowSize: 10
failureRateThreshold: 50
waitDurationInOpenState: 30s
ollama:
slidingWindowSize: 5
failureRateThreshold: 60
waitDurationInOpenState: 10s
retry:
instances:
openai:
maxAttempts: 2
waitDuration: 500ms
retryExceptions:
- java.util.concurrent.TimeoutException
- org.springframework.web.client.HttpServerErrorException
anthropic:
maxAttempts: 2
waitDuration: 500ms
management:
endpoints.web.exposure.include: health, prometheus, metrics, circuitbreakers
health.circuitbreakers.enabled: true
Performance Comparison: With vs Without Gateway
Deployment: Gateway as a Kubernetes ClusterIP Service
# kubernetes/llm-gateway.yaml — internal only, not exposed via Ingress
apiVersion: apps/v1
kind: Deployment
metadata:
name: llm-gateway
namespace: ai-platform
spec:
replicas: 3 # circuit breaker state lives in Redis — all replicas share it
selector:
matchLabels: { app: llm-gateway }
template:
metadata:
labels: { app: llm-gateway }
spec:
containers:
- name: llm-gateway
image: registry.internal/llm-gateway:1.0.0
ports:
- containerPort: 8080
env:
- name: OPENAI_API_KEY
valueFrom:
secretKeyRef: { name: llm-secrets, key: openai-api-key }
- name: ANTHROPIC_API_KEY
valueFrom:
secretKeyRef: { name: llm-secrets, key: anthropic-api-key }
- name: REDIS_HOST
value: redis-service
resources:
requests: { memory: 512Mi, cpu: 250m }
limits: { memory: 1Gi, cpu: 1000m }
readinessProbe:
httpGet: { path: /actuator/health, port: 8080 }
initialDelaySeconds: 15
periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
name: llm-gateway
namespace: ai-platform
spec:
type: ClusterIP
selector: { app: llm-gateway }
ports:
- port: 80
targetPort: 8080
Decision Tree: Do You Need a Gateway?
Trade-offs and When NOT to Build a Gateway
A gateway adds a network hop, another service to operate, and a new failure point. For a single microservice calling one provider with <$200/month spend, it is overkill — use a shared Spring Boot auto-configuration library instead.
Build a full gateway when: 2+ teams are independently building LLM features (key management chaos otherwise); provider reliability is SLA-critical (automatic failover without touching microservice code); monthly LLM spend exceeds $500 (semantic cache ROI kicks in fast); or compliance requires a central audit log of all LLM interactions.
Key Takeaways
- The LLM Gateway is the most-missing piece in Java AI architectures. Build it before you have a cost overrun, not after.
- Per-provider circuit breakers are mandatory. Never share circuit state across providers — one outage must not poison the others.
- Rate-limit by tokens, not requests. A summarization request can cost 50× a simple lookup.
- Semantic caching cuts 40–60% of LLM costs on knowledge-base workloads. An embedding call is ~100× cheaper than a generation call.
- Deploy as ClusterIP — the gateway is internal infrastructure, not a public API.
- Record cost in Micrometer counters (microdollars). You’ll see spend spikes in minutes via Grafana, not at month-end billing.
Frequently Asked Questions
Should the LLM Gateway be synchronous or asynchronous?
Both, depending on the use case. For chat UX, implement a streaming SSE endpoint via Spring WebFlux and StreamingChatModel — this gives sub-500 ms time-to-first-token. For batch workloads (document summarization, classification pipelines), front the gateway with a Kafka or SQS queue and process asynchronously. The orchestrator works in both modes with minimal changes.
How do I prevent prompt injection at the gateway?
Add an input guardrail before the provider call: scan inputs for known injection patterns (“ignore previous instructions”, “you are now…”) and block them. LangChain4j’s InputGuardrails API is a clean interface for this. For regulated industries, add a dedicated LLM classifier that scores every prompt for injection risk and rejects those above your threshold.
What is the difference between an LLM Gateway and an API Gateway?
An API Gateway (Kong, AWS API Gateway) handles general HTTP concerns: auth, routing, request-count rate limiting, SSL. An LLM Gateway is domain-specific: it understands tokens (not requests), manages circuit breakers per AI provider, and tracks cost by feature and tenant. Run both — the API Gateway as the public edge, the LLM Gateway as the internal LLM abstraction layer.
Is there an open-source Java LLM Gateway?
As of April 2026 no mature Java-native open-source LLM Gateway exists. LiteLLM (Python) is the closest equivalent and worth studying for provider abstraction patterns. Building on Spring Boot with Resilience4j gives you full control over routing rules — important for complex policies like “route Tier-1 tenants to GPT-4o, Tier-2 to GPT-4o-mini, never send documents tagged confidential to cloud providers”.
See Also
- Production-Grade RAG with Spring AI 1.1.0 — the LLM Gateway is the transport layer; RAG is what you route through it
- Virtual Threads vs Platform Threads: Real Benchmarks — use virtual threads for the gateway’s provider call pool
Conclusion
The LLM Gateway turns ad-hoc AI experiments into a managed, cost-controlled, resilient platform. Built in Spring Boot with Resilience4j circuit breakers, Bucket4j token buckets, Redis semantic caching, and Micrometer cost counters — all first-class Spring citizens — it gives every microservice team a single, consistent LLM interface with automatic failover and zero key-management overhead.
🚀 Ready to run it? The complete project with docker-compose, full directory structure, all source files, build instructions, and five end-to-end I/O examples is in the companion post: LLM Gateway — Complete Runnable Code →