I spent several weeks running Java AI inference handlers across all three major clouds — AWS Lambda, Azure Functions, and Google Cloud Run — testing cold start behaviour, token cost at scale, and multi-model routing under realistic load. The short version: the right choice depends on which layer is your bottleneck, and the three clouds diverge more sharply than any generic “serverless comparison” post will tell you.

This post covers everything I found: measured cold start numbers with sources, honest cost models at 50k–100k requests/day, multi-model routing patterns, observability trade-offs, and runnable Java code for RAG endpoints and function-based agents. Skip to the Winner Section if you want the bottom line immediately.

TL;DR: Quick Picks

🏆 The Winner (And Why It Depends)

Why Generic Serverless Posts Don’t Help You Here

Serverless Java is genuinely fast now. AWS Lambda ships Java 25 with AOT caches enabled by default — the first production wave of Project Leyden. GraalVM native image cold starts under 100 ms are routine with Quarkus. Published data from Liberty Mutual, cited in the AWS Lambda Java 25 launch post, shows a Spring Boot Lambda dropping from 5.7s to 655ms as a native image — roughly 9×.

But none of that on its own makes serverless AI inference fast. The right question is: which layer holds the cold start, and which layer holds the cost? The three clouds answer this completely differently.

The Three Architectures at a Glance

1. AWS Lambda + SageMaker (or Bedrock)

The canonical AWS pattern: API Gateway → Lambda (Java) → Bedrock or SageMaker. Lambda has no GPU support — no GPU resource type, no CUDA drivers — so Lambda is always the coordination layer, never the inference layer.

SageMaker offers three relevant deployment types. SageMaker Serverless Inference docs make clear it does not support GPUs — which rules out most generative models. Real-time endpoints (always-on instances) and Async endpoints (scale-to-zero with cold start consequences) are the alternatives. For most Java teams, Lambda + Bedrock is the honest serverless AI answer: token-priced, no SageMaker endpoint lifecycle to manage, real scale-to-zero. SageMaker shines only when you have a custom-trained model.

2. Azure Functions + Microsoft Foundry

Microsoft renamed Azure AI Studio to Microsoft Foundry. The Azure AI Inference beta SDK is being retired — check the official Azure Foundry supported languages page for the current deprecation date. For new projects, use the OpenAI-compatible /openai/v1 endpoint with the standard OpenAI Java SDK plus azure-identity. The Java azure-ai-projects SDK covers agents, evaluations, memory, and inference under a single AIProjectClient.

3. Google Cloud Run + Vertex AI

Cloud Run is container-first serverless — the closest of the three to “deploy any Linux binary, scale to zero.” That makes it the natural home for Quarkus or Micronaut native images. Vertex AI provides a unified Google Gen AI SDK for Java (com.google.genai:google-genai). See the Vertex AI overview for the platform’s current state as it transitions into the Gemini Enterprise Agent Platform.

Cold Start Benchmarks: Java AI Handlers

P50 and P99 figures below are for a Java 25 inference handler that parses JSON, calls a chat-completion endpoint, and returns JSON. Function memory: 1024 MB unless noted. Sources are named per row — treat these as directional ranges, not guaranteed numbers. Your dependency graph dominates: a lightweight Quarkus handler is faster than a full Spring Boot app with 30 starters. Always measure with your actual production payload.

Cost Models: Where Your Money Actually Goes

Illustrative estimates for 100,000 inference requests/day, ~800 input + 200 output tokens average, P95 latency target ≤ 500 ms. Calculated from Bedrock pricing, Azure OpenAI pricing, and Vertex AI pricing pages (May 2026). Use each provider’s pricing calculator for your exact numbers.

Multi-Model Routing: Where the Clouds Diverge Most

Multi-model routing — picking which model serves a request based on cost, latency, capability, or tenant — is what separates a one-vendor demo from a real LLM platform. In my experience building routing layers, this is the capability teams underinvest in most, then regret at scale.

❌ Bad: One Provider Hardcoded

// Bad: tightly coupled to one vendor, no fallback, no cost controls.
// SDK client initialized on every cold start — pays the reflection cost every time.
public class InferenceHandler implements RequestHandler<APIGatewayProxyRequestEvent, APIGatewayProxyResponseEvent> {

    // PROBLEM: built inside the class, not as a static field — rebuilt on every invocation
    @Override
    public APIGatewayProxyResponseEvent handleRequest(APIGatewayProxyRequestEvent req, Context ctx) {
        OpenAIClient openai = OpenAIOkHttpClient.builder()
            .apiKey(System.getenv("OPENAI_API_KEY"))
            .build();  // expensive — connection pools, reflection, DNS resolution

        ChatCompletion completion = openai.chat().completions().create(
            ChatCompletionCreateParams.builder()
                .model("gpt-4.1-mini")
                .addUserMessage(req.getBody())
                .build()
        );
        return new APIGatewayProxyResponseEvent()
            .withStatusCode(200)
            .withBody(completion.choices().get(0).message().content().orElse(""));
    }
}
// Problems: provider failure = total outage | no fallback | no per-tenant routing
// | client rebuilt on EVERY invocation (not just cold starts) | zero cost visibility

✅ Better: Provider-Agnostic Router with Cost-Aware Selection

// A minimal LLM gateway suitable for Lambda, Functions, or Cloud Run.
// Clients initialized ONCE (static or constructor) — reused across all warm invocations.
public final class ModelRouter {

    // Sealed types: routing logic is exhaustive at compile time — no missed cases
    public sealed interface Provider permits Bedrock, Foundry, Vertex {}
    public record Bedrock(String modelId) implements Provider {}
    public record Foundry(String deploymentName) implements Provider {}
    public record Vertex(String modelId) implements Provider {}

    private final BedrockRuntimeClient bedrock;
    private final OpenAIClient foundryClient;
    private final Client vertexClient;

    public ModelRouter() {
        // Expensive initialization happens once per execution environment.
        // With SnapStart, these survive snapshot/restore (add CRaC hooks for
        // anything that must be unique per env, like connection pools).
        this.bedrock = BedrockRuntimeClient.builder().build();
        this.foundryClient = OpenAIOkHttpClient.builder()
            .baseUrl(System.getenv("FOUNDRY_BASE_URL"))  // .../openai/v1/
            .apiKey(System.getenv("FOUNDRY_API_KEY"))
            .build();
        this.vertexClient = Client.builder().build();
    }

    /** Route based on cost, capability, and tenant tier. */
    public Provider route(InferenceRequest r) {
        // Short free-tier prompts: cheapest model
        if (r.tenantTier() == Tier.FREE && r.inputTokens() < 600) {
            return new Vertex("gemini-2.5-flash");
        }
        // Tool/function calling: needs reliable structured output
        if (r.requiresFunctionCalling()) {
            return new Bedrock("anthropic.claude-haiku-4-5");
        }
        // Paid tier default
        return new Foundry("gpt-4.1-mini");
    }

    public String invoke(InferenceRequest r) {
        return switch (route(r)) {
            case Bedrock(var id)  -> callBedrock(id, r);
            case Foundry(var dep) -> callFoundry(dep, r);
            case Vertex(var id)   -> callVertex(id, r);
        };
    }
    // callBedrock/callFoundry/callVertex: each logs tenant_id, model, input_tokens,
    // output_tokens, latency_ms to structured JSON for cost attribution
}

This is the same pattern documented in our LLM Gateway pattern for Java microservices and the complete runnable LLM gateway demo.

Runnable RAG Endpoint (Works on All Three Clouds)

RAG is the most common real-world inference pattern: receive query → embed → search vector store → stuff passages into prompt → call LLM → return. The serverless challenge is keeping all of this fast on a cold start.

// Vendor-neutral RAG core. Wrap with:
//   Lambda: implement RequestStreamHandler
//   Azure Functions: @HttpTrigger
//   Cloud Run: Quarkus @Path or Spring @RestController
public class RagCore {

    private final EmbeddingClient embedder;  // e.g. Vertex text-embedding-005
    private final VectorStore vectorStore;   // pgvector on AlloyDB / Aurora / Cosmos
    private final ModelRouter router;

    // All initialized once per execution environment — SnapStart/Leyden preserve these
    public RagCore(EmbeddingClient e, VectorStore v, ModelRouter r) {
        this.embedder = e;
        this.vectorStore = v;
        this.router = r;
    }

    public String answer(String query, String tenantId) {
        // Step 1: embed the query (~50–150 ms, single network call)
        float[] embedding = embedder.embed(query);

        // Step 2: retrieve top-4 passages scoped to this tenant
        List<Passage> context = vectorStore.search(embedding, tenantId, 4);

        // Step 3: build a grounded, hallucination-resistant prompt
        String prompt = """
            Answer using ONLY the context below.
            If the context doesn't contain the answer, say "I don't know."

            Context:
            %s

            Question: %s
            """.formatted(formatContext(context), query);

        // Step 4: route to cost/latency-appropriate model
        return router.invoke(new InferenceRequest(
            prompt, query.length() / 4, Tier.PAID, false));
    }

    private String formatContext(List<Passage> passages) {
        return passages.stream()
            .map(p -> "- [" + p.sourceId() + "] " + p.text())
            .collect(Collectors.joining("n"));
    }
}

For a fully-runnable Spring AI version with chunking, embedding generation, and a real vector store, see our complete Spring AI RAG runnable demo.

Function-Based Agents: Where Cold Start Optimisation Pays Off Most

The pattern gaining traction: the agent loop runs in Bedrock Agents, Foundry Agent Service, or Vertex Agent Engine. Each tool the agent calls is a separate serverless function. With a 4-second cold-starting Java Lambda tool, a 12-tool agent loop takes minutes. With Java 25 + Leyden AOT cache, each tool call wrapper is sub-200 ms — comparable to a database query.

// Function-based agent tool: receives structured JSON from the agent platform,
// executes business logic, returns typed JSON the agent reasons over.
//
// Tool schema registered with agent platform:
//   name: "lookup_order"
//   parameters: { order_id: string, include_lines: boolean }
//   returns:    { status, total, customer_email, lines? }
public class LookupOrderTool {

    private final OrderRepository orders;  // initialized once, reused across warm calls

    public LookupOrderTool(OrderRepository orders) { this.orders = orders; }

    public ToolResult invoke(ToolInvocation call) {
        String orderId = call.requireString("order_id");
        boolean includeLines = call.optionalBoolean("include_lines", false);

        Order o = orders.findById(orderId)
            .orElseThrow(() -> ToolError.notFound("order_id=" + orderId));

        var result = new HashMap<String, Object>();
        result.put("status", o.status().name());
        result.put("total", o.total().toPlainString());
        result.put("customer_email", o.customerEmail());
        if (includeLines) result.put("lines", o.lines());

        return ToolResult.json(result);  // agent receives this and continues reasoning
    }
}

Observability Compared

Under the Hood: How AOT Caches and Native Images Cut Cold Starts

A cold-starting Java Lambda spends most INIT time on four things:

1. Class loading and linking — A Spring Boot app touches 15,000–25,000 classes at startup. Parsing bytecode, verifying, linking: a measurable fraction of every cold start.
2. Static initializer execution — Every @PostConstruct, every static block, every reflection-heavy library scan happens here.
3. JIT warmup — HotSpot compiles hot methods after observing them. On a cold start, nothing is hot yet — early requests run interpreted or at C1 tier.
4. Application logic — Connection pools open, schema loads, caches prime.

The practical pick: For a Spring Boot Lambda calling a managed AI API, Leyden’s cache (now default in Lambda Java 25) is the highest-ROI path — free improvement, no trade-offs. For tight tool-handlers in agent loops, GraalVM native with Quarkus is worth the build complexity. SnapStart fills the middle: full application, sub-200 ms cold starts, but requires CRaC hooks for anything unique per environment (DB connections, JWT keys, random seeds). Our Spring Boot + GraalVM native image guide walks the full build pipeline.

Gotchas That Cost Teams Real Time

• ⚠️ SageMaker Serverless Inference has no GPU support — most blog posts don’t say this clearly. If your model needs a GPU (most generative models do), SageMaker Serverless is not your answer. Use Bedrock, SageMaker Real-time, or Cloud Run GPU.
• ⚠️ Azure Functions Java cold starts haven’t improved on standard plans. The “Java is fast on serverless now” narrative is true on Lambda and Cloud Run. On Azure, you’re still at 1.5–4 s on Flex Consumption. Microsoft’s investment is in C# AOT and the Foundry runtime.
• ⚠️ Leyden AOT cache is invalidated when AWS patches the managed runtime. Don’t ship custom caches with managed runtimes — use container image deployment (where the cache is immutable) if you need a predictable cache. See Lambda runtime update docs.
• ⚠️ Vertex AI’s Java SDK is com.google.genai, not google-cloud-aiplatform. Many tutorials still point at the deprecated SDK. Check the Vertex AI overview for the current recommendation.
• ⚠️ Check the Azure AI Inference beta SDK deprecation date. See the official Foundry supported languages page before starting a new project.
• ⚠️ “Scale to zero” on AWS SageMaker has a catch. If you need P99 < 500 ms, you need MinCapacity=1 or provisioned concurrency on the endpoint — that’s a hard cost floor. Bedrock is the real scale-to-zero option on AWS for most teams.
• ⚠️ Cloud Run GPU is GA but not in every region. Check Cloud Run AI overview for current L4 GPU regional availability before committing to this architecture for data-residency-constrained workloads.
• ⚠️ API Gateway REST APIs buffer the full LLM response. For streaming, use Lambda Function URLs, HTTP API with response streaming, Cloud Run’s native streaming, or Azure Functions’ OpenAI streaming extension.

Best Practices at a Glance

Common Mistakes

• ❌ Loading a model from S3 inside the Lambda handler. A 1.5 GB HuggingFace model takes 30–45 s to load — Lambda’s 30-second handler timeout will fire first. The fix: don’t run the model in Lambda. Use SageMaker, Bedrock, or Cloud Run GPU.
• ❌ Enabling SnapStart without auditing INIT phase code. If INIT generates a JWT signing key, every snapshot-restored environment signs with the same key. Audit for unique-per-env state → add CRaC Resource hooks → then enable.
• ❌ Streaming responses with API Gateway REST. REST APIs buffer the whole response. For LLM token streaming, switch to Lambda Function URLs or HTTP API with response streaming.
• ❌ Trusting the AOT cache without measuring with production payloads. If your training run skipped a code path that production hits, you get cache misses and degraded performance. Benchmark with actual production-representative requests.
• ❌ Leaving Spring’s full auto-configuration in a serverless function. Even with Leyden’s cache, a full Spring Boot app with 30 starters pays a meaningful startup cost. For serverless, prefer hand-wired configuration or switch to Quarkus/Micronaut.
• ❌ Optimising function cold start but ignoring LLM TTFT. A 250 ms cold-start function calling a model with 4 s time-to-first-token is a 4.25 s user experience. Optimise for TTFT, not time-to-completion.

Illustrative Scenario: Multi-Tenant Customer-Support RAG API

Setup: A B2B SaaS application serves a customer-support AI assistant for several hundred tenants. Traffic is highly bursty — most tenants quiet most of the day, a handful active during business hours. Average inference: ~1,500 input tokens, ~300 output tokens. Latency target: P95 < 1.5 s end to end.

Starting point: single always-on SageMaker ml.g5.xlarge real-time endpoint. Cost: ~$730/month regardless of traffic. P95 latency: ~2.4 s (model backend bottleneck, not the Lambda wrapper).

Revised architecture:

• Cloud Run service with Quarkus native image (Java 25). Min instances=0 off-hours; min=2 during business hours per tenant cluster.
• pgvector on AlloyDB for the RAG store, scoped per tenant.
• ModelRouter: short queries (<600 tokens) → Vertex Gemini 2.5 Flash; citation-heavy queries → a more capable model; tool-calling → a model with reliable function-calling.
• Observability: OpenTelemetry → Cloud Trace + tokens-per-tenant in BigQuery from structured gateway logs.

Expected trade-off direction: the always-on ml.g5.xlarge floor (~$730/month) disappears when you switch to token-priced backends that scale with actual usage. Cold start improves from seconds to sub-200 ms with native images. Token routing to a cheaper model for short queries cuts total token spend meaningfully. The exact magnitude depends on your traffic distribution — model it with your numbers before migrating.

10 AI Prompts You Can Use to Build, Validate, or Migrate

Copy these into your assistant of choice (Claude, ChatGPT, Gemini, Cursor) when working on serverless Java AI inference. Written to produce specific, actionable answers — not generic explanations.

1. “Review my AWS Lambda Java handler for AI inference and identify everything in the cold start path that should move to static initialization or out of the handler entirely. Show me the before/after code with comments explaining the cold-start impact of each change.”
2. “Given this Spring Boot 3.4 application targeting AWS Lambda with Java 25, generate a working configuration for both Project Leyden AOT cache and Lambda SnapStart, plus the CRaC Resource hooks I need for the database connection pool and JWT key generator. Flag any incompatibilities.”
3. “Convert this Quarkus REST endpoint into a GraalVM native image build that deploys to Google Cloud Run with sub-150 ms cold start. Include the multi-stage Dockerfile, application.properties for Cloud Run’s PORT environment variable, and the reflection configuration for the Vertex AI Gen AI SDK.”
4. “Compare the cost of running this RAG endpoint on (a) Lambda + Bedrock Claude Haiku 4.5, (b) Azure Functions Java + Foundry GPT-4.1-mini, and (c) Cloud Run + Vertex Gemini 2.5 Flash, given 100,000 requests/day, 1,500 input + 300 output tokens average. Show your math, including provisioned concurrency for cold-start mitigation where applicable.”
5. “Write a Java ModelRouter using sealed types that routes requests across AWS Bedrock, Microsoft Foundry, and Google Vertex AI based on input length, tenant tier, and whether function calling is required. Include unit tests with fakes for each provider.”
6. “My SageMaker Serverless Inference endpoint is cold-starting in 30+ seconds for a 1.5 GB HuggingFace model. List five concrete techniques to reduce this — for each, give the expected reduction range, the implementation effort, and any trade-offs.”
7. “Generate an OpenTelemetry-based observability layer for a Java AI gateway that captures, per request: model name, provider, input tokens, output tokens, total latency, time-to-first-token, tenant ID, and tool calls if any. Show how to export this to AWS X-Ray, Application Insights, and Cloud Trace.”
8. “Audit this Lambda Java function for SnapStart compatibility. Flag every line that generates state during INIT that should be unique per execution environment (random IDs, DB connections, signed tokens, etc.). Suggest CRaC Resource implementations for each.”
9. “My function-based agent on Bedrock Agents has 12 tools, each a Java Lambda. Average tool call latency is 1.8 s and the agent loop takes ~25 s end-to-end. Profile this conceptually — where is the time going, and which optimization (Leyden AOT cache vs SnapStart vs GraalVM native vs moving tools to a single shared container) gives the best win for the least effort?”
10. “Compare Microsoft Foundry’s tracing for AI agents to OpenTelemetry-only instrumentation on Cloud Run + Vertex AI. For a Java Spring AI app that needs to debug tool-call failures and track token spend per tenant, which is more useful in production, and what is the migration effort if I need to switch later?”

See Also

• Project Leyden Explained: AOT Compilation and Smart Caching to Finally Fix Java’s Cold Start
• Building Native Images of Spring Boot Applications with GraalVM
• Spring AI RAG in Java — Complete Runnable Code & End-to-End Demo
• The LLM Gateway Pattern for Java Microservices: Multi-Provider Failover, Cost Control, and Rate Limiting
• LLM Gateway for Java Microservices — Complete Runnable Code & Demo
• Spring AI + LangChain4j: Building Production-Ready AI Microservices in Java
• Virtual Threads vs Reactive (WebFlux) vs Platform Threads in Spring Boot 3.4
• Java 25 LTS: Every JEP That Matters (with AI Prompts for Each Migration)

Conclusion

After testing all three stacks against realistic bursty workloads, the headline finding is simple: the function layer cold start is no longer the bottleneck for Java in 2025–2026 — the inference backend and the token cost model are. Java 25’s default AOT cache on Lambda and sub-150 ms Quarkus native cold starts on Cloud Run have closed the gap with Python. What differentiates the three clouds now is cost shape at scale, multi-model routing ergonomics, and observability depth.

FAQs

Is Java actually competitive with Python for serverless AI inference?

Yes — for the orchestration/gateway/tool-handler layer. With Java 25’s default AOT cache on Lambda, plus SnapStart or GraalVM native, cold starts are on par with Python (often faster). For the model itself, both languages call the same managed APIs (Bedrock, Foundry, Vertex) — the language is irrelevant at that layer. Where Python still wins: serving custom models with PyTorch/TF natively in the function, which isn’t a serverless-friendly workload anyway.

Should I use Project Leyden’s AOT cache or GraalVM native image?

Different tools for different budgets. Leyden: 40–60% startup improvement, no code changes, full JVM dynamism, free. GraalVM: ~10× improvement, reflection configuration required, constrains dynamic features, longer build. For a Spring Boot Lambda calling a managed AI API, Leyden (now default in Lambda Java 25) is the higher ROI path. For tight tool-handlers in agent loops, GraalVM native with Quarkus is worth the build complexity.

Can I run a small LLM directly in Cloud Run instead of calling Vertex AI?

Yes. Cloud Run GPU (L4) is GA in supported regions. You can host a quantized Gemma or similar small model with true scale-to-zero. The break-even vs Vertex Gemini Flash pricing depends on QPS — Cloud Run GPU typically wins at sustained high request rates on a single instance; below that, Vertex’s per-token pricing is usually cheaper.

Why is SageMaker often discouraged for serverless AI?

Because SageMaker Serverless Inference (a) doesn’t support GPUs, ruling out most generative models, and (b) cold-starts a 1+ GB model in 20–45 seconds — which can exceed Lambda’s handler timeout. Most teams end up on SageMaker Real-time (always-on cost ~$730+/month) or Async (with queue complexity). Bedrock fixes both: token-priced, no infrastructure, real scale-to-zero.

How do I track tokens-per-tenant for cost attribution?

Build it in your model router. Every provider’s response includes a usage object with input/output tokens. Log structured JSON: {tenant_id, model, provider, input_tokens, output_tokens, latency_ms} → ship to your data warehouse (BigQuery, Snowflake, Redshift). APMs show latency; they’re not designed for token-based billing reconciliation — your gateway is.

Is Azure Functions Java really that bad for AI inference?

“Bad” is too strong. For HTTP APIs with strict latency SLAs, it’s the weakest of the three — 1.5–4 s cold starts on Flex Consumption with no SnapStart equivalent. For event-driven workloads (Service Bus queues, Cosmos DB change feeds, scheduled jobs) where a 2-second cold start is invisible to users, Azure Functions Java + Foundry is a perfectly viable stack — and Foundry’s AI observability is genuinely best-in-class among the three.

Further Reading

• AWS Lambda now supports Java 25 (AWS Compute Blog) — official announcement with Liberty Mutual case study data (5.7s → 655ms as native image)
• Run Into the New Year with Java’s Ahead-of-Time Cache Optimizations (inside.java) — JEP-by-JEP walkthrough of the Leyden cache mechanism (JEP 483, 514, 515)
• SageMaker Serverless Inference docs — including GPU limitations and supported instance types
• AWS Lambda SnapStart documentation — including CRaC hooks and priming strategies
• Azure Foundry supported languages (Java) — current SDK options and deprecation timeline
• Vertex AI overview — platform transition into Gemini Enterprise Agent Platform
• Run AI solutions on Cloud Run — GPU availability, patterns, and pricing for Cloud Run AI workloads
• aws-samples/serverless-graalvm-demo — reference Java + GraalVM Lambda project with benchmark methodology
• Quarkus: Deploying to Google Cloud — native image build and Cloud Run deployment guide
• OpenJDK Project Leyden — the JEP roadmap for Java startup and footprint improvements