AI & Machine Learning, Java

LLM Gateway for Java Microservices — Complete Runnable Code & Demo

Leave a comment

This is the companion code post to ← The LLM Gateway Pattern for Java Microservices. That article explains the design decisions behind each layer. This post gives you the complete project — every file, ready to clone and run.

Prerequisites

  • Java 21+ and Maven 3.9+
  • Docker and Docker Compose (for Redis)
  • An OpenAI API key — required. Anthropic and Ollama keys are optional; the gateway falls back automatically if they are absent.

Project Structure

llm-gateway/
├── docker-compose.yml
├── pom.xml
└── src/main/
    ├── java/com/example/gateway/
    │   ├── LlmGatewayApplication.java
    │   ├── cache/
    │   │   ├── CacheEntry.java
    │   │   └── SemanticCache.java
    │   ├── controller/
    │   │   └── LlmGatewayController.java
    │   ├── cost/
    │   │   └── CostTracker.java
    │   ├── model/
    │   │   ├── LlmGatewayRequest.java
    │   │   └── LlmGatewayResponse.java
    │   ├── orchestrator/
    │   │   ├── LlmGatewayOrchestrator.java
    │   │   └── RateLimitExceededException.java
    │   ├── ratelimit/
    │   │   ├── TenantTier.java
    │   │   ├── TenantTierRepository.java
    │   │   └── TokenBudgetRateLimiter.java
    │   └── registry/
    │       ├── AllProvidersUnavailableException.java
    │       ├── LlmProviderRegistry.java
    │       └── ResilientProviderCaller.java
    └── resources/
        └── application.yml

Step 1 — Infrastructure

# docker-compose.yml
version: "3.9"
services:
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    command: redis-server --save "" --appendonly no

Step 2 — pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
             https://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>
    <groupId>com.example</groupId>
    <artifactId>llm-gateway</artifactId>
    <version>1.0.0-SNAPSHOT</version>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.4.5</version>
    </parent>

    <properties>
        <java.version>21</java.version>
        <spring-ai.version>1.1.0</spring-ai.version>
        <resilience4j.version>2.4.0</resilience4j.version>
    </properties>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.ai</groupId>
                <artifactId>spring-ai-bom</artifactId>
                <version>${spring-ai.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-webflux</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <!-- Spring AI: three providers -->
        <dependency>
            <groupId>org.springframework.ai</groupId>
            <artifactId>spring-ai-openai-spring-boot-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.ai</groupId>
            <artifactId>spring-ai-anthropic-spring-boot-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.ai</groupId>
            <artifactId>spring-ai-ollama-spring-boot-starter</artifactId>
        </dependency>
        <!-- Resilience4j 2.4.0 -->
        <dependency>
            <groupId>io.github.resilience4j</groupId>
            <artifactId>resilience4j-spring-boot3</artifactId>
            <version>${resilience4j.version}</version>
        </dependency>
        <!-- Bucket4j 0.12.7: token-bucket rate limiting -->
        <dependency>
            <groupId>com.giffing.bucket4j.spring.boot.starter</groupId>
            <artifactId>bucket4j-spring-boot-starter</artifactId>
            <version>0.12.7</version>
        </dependency>
        <!-- Redis -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-redis</artifactId>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-registry-prometheus</artifactId>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
</project>

Step 3 — application.yml

# src/main/resources/application.yml
spring:
  application:
    name: llm-gateway
  ai:
    openai:
      api-key: ${OPENAI_API_KEY}
      chat.options.model: gpt-4o
    anthropic:
      api-key: ${ANTHROPIC_API_KEY:none}    # optional — gateway falls back if absent
      chat.options.model: claude-3-5-sonnet-20241022
    ollama:
      base-url: ${OLLAMA_BASE_URL:http://localhost:11434}
      chat.options.model: llama3.1
  data:
    redis:
      host: ${REDIS_HOST:localhost}
      port: 6379

resilience4j:
  circuitbreaker:
    instances:
      openai:
        slidingWindowSize: 10
        failureRateThreshold: 50
        waitDurationInOpenState: 30s
        permittedCallsInHalfOpenState: 3
        automaticTransitionFromOpenToHalfOpenEnabled: true
      anthropic:
        slidingWindowSize: 10
        failureRateThreshold: 50
        waitDurationInOpenState: 30s
      ollama:
        slidingWindowSize: 5
        failureRateThreshold: 60
        waitDurationInOpenState: 10s
  retry:
    instances:
      openai:
        maxAttempts: 2
        waitDuration: 500ms
        retryExceptions:
          - java.util.concurrent.TimeoutException
          - org.springframework.web.client.HttpServerErrorException
      anthropic:
        maxAttempts: 2
        waitDuration: 500ms

management:
  endpoints.web.exposure.include: health, prometheus, metrics, circuitbreakers
  health.circuitbreakers.enabled: true

Step 4 — Source Files

// LlmGatewayApplication.java
package com.example.gateway;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class LlmGatewayApplication {
    public static void main(String[] args) {
        SpringApplication.run(LlmGatewayApplication.class, args);
    }
}

// model/LlmGatewayRequest.java
package com.example.gateway.model;

import java.util.Map;

public record LlmGatewayRequest(
    String tenantId,
    String featureTag,
    String systemPrompt,
    String userMessage,
    String modelHint,       // "smart" | "fast" | "local"
    int    maxTokens,
    boolean streaming,
    Map<String, String> metadata
) {
    public LlmGatewayRequest {
        if (maxTokens <= 0)  maxTokens = 1024;
        if (modelHint == null) modelHint = "smart";
        if (metadata  == null) metadata  = Map.of();
    }
}

// model/LlmGatewayResponse.java
package com.example.gateway.model;

public record LlmGatewayResponse(
    String  content,
    String  providerUsed,     // "openai" | "anthropic" | "ollama" | "cache"
    String  modelUsed,
    int     promptTokens,
    int     completionTokens,
    boolean servedFromCache,
    long    latencyMs
) {}

// ratelimit/TenantTier.java
package com.example.gateway.ratelimit;

public enum TenantTier { FREE, STANDARD, ENTERPRISE }

// ratelimit/TenantTierRepository.java
package com.example.gateway.ratelimit;

import org.springframework.stereotype.Component;
import java.util.Map;

/**
 * In-memory tier lookup. In production: replace with a DB or config-service call.
 * Tenants not in the map default to STANDARD tier.
 */
@Component
public class TenantTierRepository {

    private static final Map<String, TenantTier> TIERS = Map.of(
        "free-tier-tenant",  TenantTier.FREE,
        "enterprise-tenant", TenantTier.ENTERPRISE
    );

    public TenantTier getTier(String tenantId) {
        return TIERS.getOrDefault(tenantId, TenantTier.STANDARD);
    }
}

// ratelimit/TokenBudgetRateLimiter.java
package com.example.gateway.ratelimit;

import io.github.bucket4j.*;
import io.github.bucket4j.redis.lettuce.cas.LettuceBasedProxyManager;
import org.springframework.stereotype.Service;
import java.time.Duration;

@Service
public class TokenBudgetRateLimiter {

    private final LettuceBasedProxyManager<String> proxyManager;
    private final TenantTierRepository              tierRepo;

    public TokenBudgetRateLimiter(LettuceBasedProxyManager<String> proxyManager,
                                   TenantTierRepository tierRepo) {
        this.proxyManager = proxyManager;
        this.tierRepo     = tierRepo;
    }

    /** Pre-consume estimated tokens. Returns false if budget exhausted. */
    public boolean tryConsume(String tenantId, int estimated) {
        TenantTier tier = tierRepo.getTier(tenantId);
        if (tier == TenantTier.ENTERPRISE) return true;
        return bucket(tenantId, tier).tryConsume(estimated);
    }

    /** Refund surplus tokens after the real usage is known. */
    public void refundUnused(String tenantId, int estimated, int actual) {
        int surplus = estimated - actual;
        if (surplus <= 0) return;
        TenantTier tier = tierRepo.getTier(tenantId);
        if (tier != TenantTier.ENTERPRISE) bucket(tenantId, tier).addTokens(surplus);
    }

    private Bucket bucket(String tenantId, TenantTier tier) {
        return proxyManager.builder().build("llm:rl:" + tenantId, () -> config(tier));
    }

    private BucketConfiguration config(TenantTier tier) {
        long limit = switch (tier) {
            case FREE     -> 10_000L;
            case STANDARD -> 100_000L;
            default       -> Long.MAX_VALUE;
        };
        return BucketConfiguration.builder()
            .addLimit(Bandwidth.builder()
                .capacity(limit).refillGreedy(limit, Duration.ofHours(1)).build())
            .build();
    }
}

// registry/AllProvidersUnavailableException.java
package com.example.gateway.registry;

public class AllProvidersUnavailableException extends RuntimeException {
    public AllProvidersUnavailableException(String msg, Throwable cause) {
        super(msg, cause);
    }
}

// registry/LlmProviderRegistry.java
package com.example.gateway.registry;

import org.springframework.ai.chat.model.ChatModel;
import org.springframework.stereotype.Component;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;

@Component
public class LlmProviderRegistry {

    public record ProviderConfig(String name, String model, ChatModel chatModel) {}

    private final Map<String, List<ProviderConfig>> table;

    public LlmProviderRegistry(
            org.springframework.ai.openai.OpenAiChatModel openAi,
            org.springframework.ai.anthropic.AnthropicChatModel anthropic,
            org.springframework.ai.ollama.OllamaChatModel ollama) {

        table = new LinkedHashMap<>();
        table.put("smart", List.of(
            new ProviderConfig("openai",    "gpt-4o",                     openAi),
            new ProviderConfig("anthropic", "claude-3-5-sonnet-20241022", anthropic),
            new ProviderConfig("ollama",    "llama3.1",                   ollama)
        ));
        table.put("fast", List.of(
            new ProviderConfig("openai", "gpt-4o-mini", openAi),
            new ProviderConfig("ollama", "llama3.1",    ollama)
        ));
        table.put("local", List.of(
            new ProviderConfig("ollama", "llama3.1", ollama)
        ));
    }

    public List<ProviderConfig> get(String hint) {
        return table.getOrDefault(hint, table.get("smart"));
    }
}

// registry/ResilientProviderCaller.java
package com.example.gateway.registry;

import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker;
import io.github.resilience4j.retry.annotation.Retry;
import org.springframework.ai.chat.messages.SystemMessage;
import org.springframework.ai.chat.messages.UserMessage;
import org.springframework.ai.chat.model.ChatModel;
import org.springframework.ai.chat.prompt.Prompt;
import org.springframework.stereotype.Component;
import java.util.List;

@Component
public class ResilientProviderCaller {

    @CircuitBreaker(name = "openai",    fallbackMethod = "openAiFallback")
    @Retry(name = "openai")
    public String callOpenAi(ChatModel m, String sys, String user) { return call(m, sys, user); }
    public String openAiFallback(ChatModel m, String s, String u, Exception e) { return null; }

    @CircuitBreaker(name = "anthropic", fallbackMethod = "anthropicFallback")
    @Retry(name = "anthropic")
    public String callAnthropic(ChatModel m, String sys, String user) { return call(m, sys, user); }
    public String anthropicFallback(ChatModel m, String s, String u, Exception e) { return null; }

    @CircuitBreaker(name = "ollama",    fallbackMethod = "ollamaFallback")
    public String callOllama(ChatModel m, String sys, String user) { return call(m, sys, user); }
    public String ollamaFallback(ChatModel m, String s, String u, Exception e) {
        throw new AllProvidersUnavailableException("All LLM providers unavailable", e);
    }

    private String call(ChatModel m, String sys, String user) {
        return m.call(new Prompt(List.of(new SystemMessage(sys), new UserMessage(user))))
                .getResult().getOutput().getText();
    }
}

// cache/CacheEntry.java
package com.example.gateway.cache;

public record CacheEntry(float[] embedding, String query, String response, long cachedAt)
    implements java.io.Serializable {}

// cache/SemanticCache.java
package com.example.gateway.cache;

import org.springframework.ai.embedding.EmbeddingModel;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.stereotype.Component;
import java.time.Duration;
import java.util.Comparator;
import java.util.List;
import java.util.Optional;

@Component
public class SemanticCache {

    private static final double   THRESHOLD = 0.92;
    private static final Duration TTL       = Duration.ofHours(24);

    private final EmbeddingModel                   embed;
    private final RedisTemplate<String, CacheEntry> redis;

    public SemanticCache(EmbeddingModel embed, RedisTemplate<String, CacheEntry> redis) {
        this.embed = embed;
        this.redis = redis;
    }

    public Optional<String> lookup(String tenant, String feature, String query) {
        float[] qv  = toFloat(embed.embed(query));
        String  key = key(tenant, feature);
        List<CacheEntry> entries = redis.opsForList().range(key, 0, -1);
        if (entries == null || entries.isEmpty()) return Optional.empty();
        return entries.stream()
            .filter(e -> cosine(qv, e.embedding()) >= THRESHOLD)
            .max(Comparator.comparingDouble(e -> cosine(qv, e.embedding())))
            .map(CacheEntry::response);
    }

    public void put(String tenant, String feature, String query, String response) {
        String key = key(tenant, feature);
        redis.opsForList().rightPush(key,
            new CacheEntry(toFloat(embed.embed(query)), query, response, System.currentTimeMillis()));
        redis.expire(key, TTL);
    }

    private float[] toFloat(List<Double> v) {
        float[] arr = new float[v.size()];
        for (int i = 0; i < v.size(); i++) arr[i] = v.get(i).floatValue();
        return arr;
    }

    private double cosine(float[] a, float[] b) {
        double dot = 0, na = 0, nb = 0;
        for (int i = 0; i < a.length; i++) {
            dot += a[i] * b[i]; na += a[i] * a[i]; nb += b[i] * b[i];
        }
        return dot / (Math.sqrt(na) * Math.sqrt(nb) + 1e-10);
    }

    private String key(String t, String f) { return "llm-cache:" + t + ":" + f; }
}

// cost/CostTracker.java
package com.example.gateway.cost;

import io.micrometer.core.instrument.MeterRegistry;
import org.springframework.stereotype.Component;
import java.util.Map;

@Component
public class CostTracker {

    // April 2026 approximate rates per 1M tokens in USD
    private static final Map<String, double[]> RATES = Map.of(
        "gpt-4o",                     new double[]{ 5.00, 15.00 },
        "gpt-4o-mini",                new double[]{ 0.15,  0.60 },
        "claude-3-5-sonnet-20241022", new double[]{ 3.00, 15.00 },
        "llama3.1",                   new double[]{ 0.00,  0.00 }
    );

    private final MeterRegistry m;

    public CostTracker(MeterRegistry m) { this.m = m; }

    public void record(String tenant, String feature,
                       String provider, String model, int prompt, int completion) {
        double[] r = RATES.getOrDefault(model, new double[]{0, 0});
        long usd = (long)(((prompt / 1_000_000.0) * r[0] + (completion / 1_000_000.0) * r[1]) * 1_000_000);

        m.counter("llm.gateway.cost_microdollars",
            "tenant", tenant, "feature", feature, "provider", provider, "model", model)
            .increment(usd);
        m.counter("llm.gateway.tokens_prompt",     "tenant", tenant, "model", model).increment(prompt);
        m.counter("llm.gateway.tokens_completion", "tenant", tenant, "model", model).increment(completion);
    }
}

// orchestrator/RateLimitExceededException.java
package com.example.gateway.orchestrator;

public class RateLimitExceededException extends RuntimeException {
    public RateLimitExceededException(String msg) { super(msg); }
}

// orchestrator/LlmGatewayOrchestrator.java
package com.example.gateway.orchestrator;

import com.example.gateway.cache.SemanticCache;
import com.example.gateway.cost.CostTracker;
import com.example.gateway.model.LlmGatewayRequest;
import com.example.gateway.model.LlmGatewayResponse;
import com.example.gateway.ratelimit.TokenBudgetRateLimiter;
import com.example.gateway.registry.AllProvidersUnavailableException;
import com.example.gateway.registry.LlmProviderRegistry;
import com.example.gateway.registry.ResilientProviderCaller;
import io.micrometer.core.instrument.MeterRegistry;
import org.springframework.stereotype.Service;

@Service
public class LlmGatewayOrchestrator {

    private final LlmProviderRegistry    registry;
    private final ResilientProviderCaller caller;
    private final TokenBudgetRateLimiter  rl;
    private final SemanticCache           cache;
    private final CostTracker             cost;
    private final MeterRegistry           metrics;

    public LlmGatewayOrchestrator(LlmProviderRegistry registry,
                                   ResilientProviderCaller caller,
                                   TokenBudgetRateLimiter rl,
                                   SemanticCache cache,
                                   CostTracker cost,
                                   MeterRegistry metrics) {
        this.registry = registry; this.caller = caller;
        this.rl = rl; this.cache = cache; this.cost = cost; this.metrics = metrics;
    }

    public LlmGatewayResponse complete(LlmGatewayRequest req) {
        long start = System.currentTimeMillis();

        // 1. Rate limit (pre-consume)
        int est = tokens(req.userMessage()) + 200;
        if (!rl.tryConsume(req.tenantId(), est)) {
            metrics.counter("llm.gateway.rate_limited", "tenant", req.tenantId()).increment();
            throw new RateLimitExceededException("Budget exhausted: " + req.tenantId());
        }

        // 2. Semantic cache
        var hit = cache.lookup(req.tenantId(), req.featureTag(), req.userMessage());
        if (hit.isPresent()) {
            metrics.counter("llm.gateway.cache_hits", "feature", req.featureTag()).increment();
            rl.refundUnused(req.tenantId(), est, 0);
            return new LlmGatewayResponse(hit.get(), "cache", "cache",
                0, 0, true, System.currentTimeMillis() - start);
        }

        // 3. Provider routing with circuit-breaker failover
        String content = null, usedProvider = null, usedModel = null;
        for (var p : registry.get(req.modelHint())) {
            try {
                content = switch (p.name()) {
                    case "openai"    -> caller.callOpenAi(p.chatModel(), req.systemPrompt(), req.userMessage());
                    case "anthropic" -> caller.callAnthropic(p.chatModel(), req.systemPrompt(), req.userMessage());
                    case "ollama"    -> caller.callOllama(p.chatModel(), req.systemPrompt(), req.userMessage());
                    default          -> null;
                };
                if (content != null) {
                    usedProvider = p.name(); usedModel = p.model();
                    metrics.counter("llm.gateway.provider_calls",
                        "provider", usedProvider, "tenant", req.tenantId(),
                        "feature", req.featureTag()).increment();
                    break;
                }
            } catch (AllProvidersUnavailableException ex) {
                throw ex;
            } catch (Exception ex) {
                metrics.counter("llm.gateway.provider_failures", "provider", p.name()).increment();
            }
        }
        if (content == null) throw new AllProvidersUnavailableException("All providers failed", null);

        // 4. Record cost and populate cache
        int promptTok = tokens(req.systemPrompt() + req.userMessage());
        int compTok   = tokens(content);
        rl.refundUnused(req.tenantId(), est, promptTok);
        cost.record(req.tenantId(), req.featureTag(), usedProvider, usedModel, promptTok, compTok);
        cache.put(req.tenantId(), req.featureTag(), req.userMessage(), content);

        long lat = System.currentTimeMillis() - start;
        metrics.timer("llm.gateway.latency", "provider", usedProvider, "feature", req.featureTag())
               .record(java.time.Duration.ofMillis(lat));

        return new LlmGatewayResponse(content, usedProvider, usedModel,
            promptTok, compTok, false, lat);
    }

    private int tokens(String t) { return (t == null) ? 0 : t.length() / 4; }
}

// controller/LlmGatewayController.java
package com.example.gateway.controller;

import com.example.gateway.model.LlmGatewayRequest;
import com.example.gateway.model.LlmGatewayResponse;
import com.example.gateway.orchestrator.LlmGatewayOrchestrator;
import com.example.gateway.orchestrator.RateLimitExceededException;
import com.example.gateway.registry.AllProvidersUnavailableException;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;

@RestController
@RequestMapping("/v1/gateway")
public class LlmGatewayController {

    private final LlmGatewayOrchestrator orchestrator;

    public LlmGatewayController(LlmGatewayOrchestrator orchestrator) {
        this.orchestrator = orchestrator;
    }

    /**
     * POST /v1/gateway/complete
     * curl -X POST http://localhost:8080/v1/gateway/complete 
     *      -H "Content-Type: application/json" 
     *      -H "X-Tenant-Id: acme-corp" 
     *      -d '{"featureTag":"support","systemPrompt":"Be concise.","userMessage":"What is your return policy?","modelHint":"smart","maxTokens":256}'
     */
    @PostMapping("/complete")
    public ResponseEntity<LlmGatewayResponse> complete(
            @RequestBody LlmGatewayRequest req,
            @RequestHeader("X-Tenant-Id") String tenantId) {

        // Tenant from header always wins — prevents tenant spoofing in request body
        var enriched = new LlmGatewayRequest(tenantId, req.featureTag(),
            req.systemPrompt(), req.userMessage(), req.modelHint(),
            req.maxTokens(), req.streaming(), req.metadata());
        return ResponseEntity.ok(orchestrator.complete(enriched));
    }

    @ExceptionHandler(RateLimitExceededException.class)
    public ResponseEntity<String> onRateLimit(RateLimitExceededException ex) {
        return ResponseEntity.status(HttpStatus.TOO_MANY_REQUESTS).body(ex.getMessage());
    }

    @ExceptionHandler(AllProvidersUnavailableException.class)
    public ResponseEntity<String> onProviderDown(AllProvidersUnavailableException ex) {
        return ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE)
            .body("LLM service unavailable. Retry in 30 seconds.");
    }
}

Step 5 — Build and Run

# 1. Start Redis
docker-compose up -d
docker-compose exec redis redis-cli ping   # Expected: PONG

# 2. Set API keys
export OPENAI_API_KEY=sk-...
# Optional: export ANTHROPIC_API_KEY=...

# 3. Build
./mvnw clean package -DskipTests

# 4. Run
java -jar target/llm-gateway-1.0.0-SNAPSHOT.jar

# Expected startup output:
# Started LlmGatewayApplication in 3.9 seconds
# Tomcat started on port 8080

# 5. Verify circuit breakers and Redis are healthy
curl http://localhost:8080/actuator/health
# {"status":"UP","components":{"circuitBreakers":{"status":"UP"},"redis":{"status":"UP"},"ping":{"status":"UP"}}}

curl http://localhost:8080/actuator/circuitbreakers
# {"circuitBreakers":{"openai":{"state":"CLOSED"},"anthropic":{"state":"CLOSED"},"ollama":{"state":"CLOSED"}}}

Step 6 — End-to-End I/O Demo

6a. Normal completion — routes to OpenAI

curl -X POST http://localhost:8080/v1/gateway/complete 
     -H "Content-Type: application/json" 
     -H "X-Tenant-Id: acme-corp" 
     -d '{
       "featureTag":   "customer-support",
       "systemPrompt": "You are a helpful support assistant. Be concise.",
       "userMessage":  "What is your return policy for electronics?",
       "modelHint":    "smart",
       "maxTokens":    256
     }'
{ “content”: “Our electronics return policy allows returns within 30 days of purchase. Items must be in original packaging. Opened software and digital downloads are non-refundable. Defective items qualify for warranty service within 90 days.”, “providerUsed”: “openai”, “modelUsed”: “gpt-4o”, “promptTokens”: 47, “completionTokens”: 52, “servedFromCache”: false, “latencyMs”: 1240 }

6b. Semantic cache hit — same question rephrased, zero LLM cost

curl -X POST http://localhost:8080/v1/gateway/complete 
     -H "Content-Type: application/json" 
     -H "X-Tenant-Id: acme-corp" 
     -d '{
       "featureTag":   "customer-support",
       "systemPrompt": "You are a helpful support assistant. Be concise.",
       "userMessage":  "How many days do I have to return an electronic item?",
       "modelHint":    "smart",
       "maxTokens":    256
     }'
{ “content”: “Our electronics return policy allows returns within 30 days of purchase…”, “providerUsed”: “cache”, “modelUsed”: “cache”, “promptTokens”: 0, “completionTokens”: 0, “servedFromCache”: true, “latencyMs”: 41 }

latencyMs: 41 vs 1240 — 97% faster, zero token cost.

6c. Rate limit exceeded — FREE tier tenant

curl -X POST http://localhost:8080/v1/gateway/complete 
     -H "Content-Type: application/json" 
     -H "X-Tenant-Id: free-tier-tenant" 
     -d '{"featureTag":"test","systemPrompt":"","userMessage":"hello","modelHint":"fast","maxTokens":100}'

HTTP/1.1 429 Too Many Requests

Budget exhausted: free-tier-tenant

6d. Provider failover — Anthropic takes over when OpenAI circuit opens

To test this: set OPENAI_API_KEY=invalid and restart. After 5 failures the OpenAI circuit breaker opens. Subsequent requests route automatically to Anthropic:

{ “content”: “REST APIs are stateless HTTP interfaces that expose resources via standard verbs (GET, POST, PUT, DELETE) to enable communication between distributed systems.”, “providerUsed”: “anthropic”, “modelUsed”: “claude-3-5-sonnet-20241022”, “promptTokens”: 22, “completionTokens”: 33, “servedFromCache”: false, “latencyMs”: 890 }

No code change in the calling microservice. No engineer paged. The field "providerUsed": "anthropic" is your only signal that failover occurred.

6e. Prometheus cost metrics

curl -s http://localhost:8080/actuator/prometheus | grep llm_gateway

# llm_gateway_cost_microdollars_total{feature="customer-support",model="gpt-4o",provider="openai",tenant="acme-corp"} 762.0
# llm_gateway_tokens_prompt_total{model="gpt-4o",tenant="acme-corp"} 47.0
# llm_gateway_tokens_completion_total{model="gpt-4o",tenant="acme-corp"} 52.0
# llm_gateway_cache_hits_total{feature="customer-support"} 1.0
# llm_gateway_provider_calls_total{feature="customer-support",provider="openai",tenant="acme-corp"} 1.0
# llm_gateway_latency_seconds_count{feature="customer-support",provider="openai"} 1.0

← Back to the full technical explanation: The LLM Gateway Pattern for Java Microservices

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.