Streaming AI Chat UI with Spring AI, SSE and htmx (No React Needed)

Every “spring ai chatbot” tutorial gets to the same place: the controller returns a Flux<String>, you curl it, tokens arrive, and the tutorial ends. A real Spring AI streaming chat UI needs the rest — something in the browser that appends tokens as they come, a conversation that remembers the last question, a Stop button that stops the model and not just the display, and an answer for what happens when the reader is slower than the writer. This article builds all of that in one Spring Boot module with no React, no build step and no JavaScript of our own: the browser side is htmx and its SSE extension, both vendored into the repository.

It is also a measurement exercise, and the measurements found three things I did not expect. EventSource silently strips a leading space from every token, so “hello world” becomes “helloworld”. A client abort cancels the Reactor Flux every time, but in my runs it left the upstream model connection open in most of them — the cancel log line proves less than it appears to. And pushing the stream through limitRate does not slow the model down at all, because the producer is a thread that pushes.

The model in this article is a stub. No real language model and no Docker were available where I wrote this, so the model side is a small OpenAI-compatible server I wrote (OpenAiStubServer) that streams real chat.completion.chunk SSE frames with a delay I control. Its answers are templates built from the request it received — it does no inference. Everything on the Spring side is real: the real OpenAiChatModel talks to it over real HTTP. The base URL and model name are switchable to any OpenAI-compatible endpoint, but I did not run a real provider, and that matters for the cancel and back-pressure sections (see “Not tested” at the end).

Versions this was built and run against. Spring Boot 4.1.2, Spring AI 2.0.2 (spring-ai-starter-model-openai; the 2.0.2 BOM was published to Maven Central on 2026-10-22), which brings openai-java 4.80.0 and OkHttp 4.12.0, JDK 25.0.5 (LTS), htmx 2.0.11 (published 2026-09-22, 0BSD licence) and htmx-ext-sse 2.2.4 (published 2025-10-18), both vendored with checksums. The page was driven in headless Chromium 153.0.8010.12 through Playwright 1.63.0. Companion code: chat-ui-htmx in asmhatre/spring-ai.

A chat page is two requests per turn: one returns markup, one returns tokens

The smallest correct picture has three parts. A form posts what you typed. The server answers with a fragment of HTML: your message as a bubble, plus an empty bubble for the answer that is wired to open a second request. That second request is a Server-Sent Events stream — a plain HTTP response that never ends, carrying lines like event:token and data: hello — and every event is appended into the empty bubble. htmx does the posting, the swapping and the appending; its SSE extension wraps the browser’s EventSource.

The diagram is the whole app. On the server, request 1 is POST /chat and request 2 is GET /chat/stream. The second one is a GET because EventSource can only issue GETs, which has a consequence worth knowing now: the message travels in the query string of that second request (see the “should you do this” box at the end).

Going deeper

• The Server-Sent Events and WebSocket on Spring Boot 4 article covers SseEmitter, STOMP and when to pick which; this one stays on the reactive Flux route because that is what ChatClient hands you.
• Spring AI 2.0 in 10 Minutes: ChatClient on Spring Boot 4.1 introduces the ChatClient and the one-line .stream().content() used below.
• WHATWG: Server-sent events (the wire format and EventSource rules)

The model side: a stub you can run, and a switch to a real one

Streaming bugs show up on the wire, so the model had to be something I could slow down, speed up, and watch from the other side. OpenAiStubServer is about 200 lines on the JDK’s built-in HttpServer: it accepts POST /v1/chat/completions with stream:true, writes data: {…chat.completion.chunk…} frames and a final data: [DONE], sleeps a configurable number of milliseconds between chunks, and counts how many streams finished and how many ended because the client went away. That last counter is the instrument the cancel section depends on.

Start of your message	What the stub streams
anything	[stub] saw N messages. Earlier user turns: a | b. You said: c. — so the reply shows exactly what the client sent, which is how memory is observed
count N	N one-word chunks, w1 w2 … wN
big N	N chunks of 32 KiB each, to make a slow reader visible
html	text containing <b> and an <img onerror>, to test escaping

The application points the real Spring AI OpenAI starter at it with two lines of configuration, which are also the lines you change for a real provider:

spring:
  ai:
    openai:
      base-url: ${CHAT_BASE_URL:http://localhost:38181/v1}
      api-key: ${OPENAI_API_KEY:not-needed}
      # SDK default read timeout is 60 s; a slow model can exceed it.
      timeout: 5m
      chat:
        options:
          model: ${CHAT_MODEL:stub-gpt}

That is from src/main/resources/application.yml. Two things carry over from the Docker Model Runner article and are easy to trip on when you swap the stub out: the starter does not append /v1 to the base URL (so it must already end in /v1), and the SDK’s default read timeout is about 60 seconds, which is why timeout: 5m is set for slow models.

Real provider: not run. To try one, start the jar with CHAT_BASE_URL=https://api.openai.com/v1 CHAT_MODEL=<model> OPENAI_API_KEY=<key>. I did not do that, so I make no claim about how a real provider’s chunk sizes, usage chunks or HTTP/2 transport interact with anything below.

The smallest thing that works: ChatClient to Flux to server-sent events

Spring AI’s ChatClient has two ways to finish a prompt: .call() waits for the whole answer, .stream() returns a Flux that emits as the model produces. .stream().content() gives a Flux<String> of text fragments. Returning that from a controller method that produces text/event-stream is enough for Spring MVC to write each element as an SSE event. The controller here maps each fragment to a ServerSentEvent so it can name the event token and end with one named done.

        return content
                .doOnRequest(n -> {
                    requested.addAndGet(n);
                    log.info("turn {} web-side demand request({})", turn, n == Long.MAX_VALUE ? "unbounded" : n);
                })
                .map(token -> {
                    tokens.incrementAndGet();
                    // Model text is untrusted: htmx inserts the data as HTML, so escape it here.
                    // A browser strips ONE leading space from every data line (SSE spec), and Spring writes
                    // "data:" with no space of its own, so a token like " world" would lose its space.
                    // Prefix every line with a space that the client strips again.
                    String data = " " + HtmlUtils.htmlEscape(token).replace("\n", "\n ");
                    return ServerSentEvent.builder(data).event("token").build();
                })
                .concatWith(Flux.just(ServerSentEvent.<String>builder("").event("done").build()))
                .doOnSubscribe(s -> log.info("turn {} start conversation={} message={}", turn, cid, message))
                .doOnCancel(() -> cancelled.incrementAndGet())
                .doOnCancel(() -> log.info("turn {} CANCELLED by client after {} tokens", turn, tokens.get()))
                .doOnComplete(() -> log.info("turn {} complete, {} tokens", turn, tokens.get()))
                .doOnError(e -> log.info("turn {} error {}", turn, e.toString()));

That is the tail of stream(…) in ChatController.java; the doOn* operators are only logging. Here is one turn on the wire, captured by a test that reads the stream the way a browser does (output/01-sse-transcript.txt):

$ GET /chat/stream?message=hello world   (raw lines, first 14 and last 5)
event:token
data: [stub]

event:token
data:  saw

event:token
data:  2

event:token
data:  messages.

event:token
data:  Earlier
...
data:  world
<blank>
event:done
data:
<blank>
event names seen: [token, done]

The space that EventSource eats

Look at the data: lines. Real model tokens carry their own leading space — the stub’s second token is  saw — and the SSE specification says a client must remove one leading space from every field value. Spring writes data: with no space of its own, so without help the token’s space is the one that gets removed. The controller therefore puts an extra space in front of every data line, and the browser strips that one. The same test computed both outcomes:

joined token text as EventSource delivers it: [stub] saw 2 messages. Earlier user turns: none. You said: hello world
same text if the server did not pad:        [stub]saw2messages.Earlieruserturns:none.Yousaid:helloworld

Same events, same parser; the only difference is the padding (output/01-sse-transcript.txt). A multi-line token needs the padding on every line, which is why the controller replaces each newline with a newline plus a space.

Fingerprint of this bug: the streamed answer reads fine in curl -N and is glued together in the browser — [stub]saw2messages. — and code blocks lose their indentation. It never shows up in a Java test that compares tokens, because the Java side is correct.

Going deeper

• The padding is in the map step of ChatController; the parser used in the test is a few lines in ChatFlowTest.
• WHATWG SSE: parsing an event stream (the leading-space rule)

The htmx front end: one HTML file and one returned fragment

The page itself is static. It loads two vendored scripts and declares a form; hx-post sends the form, hx-target says where the response goes, and beforeend means “append” so the log grows instead of being replaced.

<script src="/vendor/htmx.min.js"></script>
<script src="/vendor/htmx-ext-sse.js"></script>
<style>
  body { font: 16px/1.5 system-ui, sans-serif; max-width: 42rem; margin: 2rem auto; padding: 0 1rem; }
  #log { min-height: 12rem; }
  .msg { padding: .5rem .75rem; margin: .5rem 0; border-radius: .5rem; white-space: pre-wrap; }
  .user { background: #e8eefc; }
  .bot { background: #f4f5f7; }
  .stop { float: right; }
  form { display: flex; gap: .5rem; }
  input[name=message] { flex: 1; padding: .5rem; }
</style>
</head>
<body>
<h1>Chat</h1>
<div id="log"></div>
<form hx-post="/chat" hx-target="#log" hx-swap="beforeend" hx-on::after-request="this.reset()">
  <input name="message" autocomplete="off" placeholder="Say something" required autofocus>
  <button>Send</button>
</form>

Source: src/main/resources/static/index.html. The vendored files and their SHA-256 sums are in vendor/SHA256SUMS; vendoring means the page works with no CDN and the version I tested is the version you get.

The interesting part is the fragment the server returns for each message. Three attributes do the streaming: sse-connect opens the EventSource; sse-swap="token" says “every event named token is content to swap”; and hx-target with hx-swap="beforeend" sends each token to the output span and appends. Without beforeend every token would replace the previous one and you would watch a single word flicker.

        return """
                <div class="msg user">%s</div>
                <div class="msg bot">
                  <span id="out-%d"></span>
                  <span id="sse-%d" hx-ext="sse" sse-connect="%s" sse-swap="token"
                        sse-close="done" hx-target="#out-%d" hx-swap="beforeend">
                    <span sse-swap="done" hx-target="#stop-%d" hx-swap="delete"></span>
                  </span>
                  <button id="stop-%d" type="button" class="stop"
                          hx-post="/chat/stop?turn=%d" hx-target="#sse-%d" hx-swap="outerHTML">Stop</button>
                </div>
                """.formatted(HtmlUtils.htmlEscape(message), turn, turn,
                HtmlUtils.htmlEscape(url), turn, turn, turn, turn, turn);

From ChatController.send. The message is escaped twice on purpose: once as text inside the bubble, once inside the URL attribute.

The diagram shows the wiring after the fragment lands. Tokens go to #out-N, which sits outside the element that owns the connection — that placement is what lets the Stop button, later, remove the connection without erasing the partial answer. The done event does two jobs: sse-close="done" closes the EventSource (without it the browser would treat the finished response as a dropped connection and reconnect, which is how the specification defines EventSource; I relied on the attribute rather than reproducing the reconnect), and a small sse-swap="done" span deletes the Stop button.

== turn 1 (session A)
A1: '[stub] saw 2 messages. Earlier user turns: none. You said: My name is Asha and I like tea'
== turn 2 (session A, same cookie)
A2: '[stub] saw 4 messages. Earlier user turns: My name is Asha and I like tea. You said: What did I just tell you?'

That is the real page in headless Chromium: two turns typed into the form and streamed back (output/06-browser.txt, produced by scripts/browser-test.py).

Going deeper

• htmx SSE extension reference (sse-connect, sse-swap, sse-close)
• The extension creates its connection with new EventSource(url, { withCredentials: true }) (read in the vendored htmx-ext-sse.js), which is why the session cookie reaches the stream request — the next section depends on it.

Model text is untrusted: escape it before htmx puts it in the page

htmx swaps what it receives as HTML. A language model’s output is whatever the model, or a document it was shown, says it is. The prompt-injection article demonstrated a markdown-image leak; this is the same family one layer down. The controller escapes every token with HtmlUtils.htmlEscape before it becomes an event. The stub’s html directive streams a bold tag and an <img onerror>:

data sent to the browser: Here is <b>bold</b> and <img src=x onerror=alert(1)> & done

Bytes sent to the browser (output/03-escaping.txt). In Chromium the answer rendered as literal text and the checks in output/06-browser.txt came out clean:

== model output containing markup
rendered text: 'Here is <b>bold</b> and <img src=x onerror=alert(1)> & done'
<b> elements created inside the answer: 0
<img> elements created inside the answer: 0
alert dialogs fired: 0

Escaping on the server has a cost: you cannot send HTML back as formatting. If you want rendered markdown, render it on the server from the complete or partially complete text and sanitise the result, rather than un-escaping the tokens.

Memory per browser session: one conversation id, taken from the HTTP session

A chat model has no memory; “memory” is the application re-sending earlier messages with each request. Spring AI’s MessageChatMemoryAdvisor does that: before each call it loads the stored messages for a conversation id, prepends them, and after the call stores the new pair. Two decisions are yours: where the messages are kept (here MessageWindowChatMemory, the last 20 messages, in a map in the JVM) and where the id comes from.

    @Bean
    ChatMemory chatMemory(@Value("${chat.memory.max-messages:20}") int maxMessages) {
        return MessageWindowChatMemory.builder().maxMessages(maxMessages).build();
    }

    @Bean
    ChatClient chatClient(ChatModel model, ChatMemory memory) {
        return ChatClient.builder(model)
                .defaultSystem("You are a concise assistant.")
                .defaultAdvisors(MessageChatMemoryAdvisor.builder(memory).build())
                .build();
    }

From ChatConfig.java. The id comes from the servlet session, so one browser is one conversation:

    static String conversationId(HttpSession session) {
        Object id = session.getAttribute(CID);
        if (id == null) {
            id = UUID.randomUUID().toString();
            session.setAttribute(CID, id);
        }
        return id.toString();
    }

The diagram is the whole scheme: cookie, session attribute, memory key. It is not a security boundary — anyone with the cookie is that conversation — and it is not shared between server instances (see the going-deeper list). The id is passed per call with .advisors(a -> a.param(ChatMemory.CONVERSATION_ID, cid)).

The proof is the stub echoing what it received (output/02-memory.txt). Turn 2 of session A arrives with four messages (system, the first user message, the first answer, the new question); the same question from a second browser arrives with two:

session A, turn 1: [stub] saw 2 messages. Earlier user turns: none. You said: My name is Asha and I like tea
session A, turn 2: [stub] saw 4 messages. Earlier user turns: My name is Asha and I like tea. You said: What did I just tell you?
session B, turn 1: [stub] saw 2 messages. Earlier user turns: none. You said: What did I just tell you?
session A, turn 3: [stub] saw 6 messages. Earlier user turns: My name is Asha and I like tea | What did I just tell you?. You said: And now?

The same result in Chromium, including the “New conversation” button that drops the session attribute (output/06-browser.txt):

== turn 1 (session B, fresh browser context, no cookie)
B1: '[stub] saw 2 messages. Earlier user turns: none. You said: What did I just tell you?'
== New conversation button in session A, then ask again
A3: '[stub] saw 2 messages. Earlier user turns: none. You said: Do you remember me?'

Forget the id and it fails loudly, but not where you look. In Spring AI 2.0.2 a memory advisor with no CONVERSATION_ID parameter does not fall back to a default; the stream fails with the message below (output/11-missing-conversation-id.txt). In my first attempt this surfaced as an “Aggregation Error” log line from a boundedElastic thread rather than as an exception in the controller.

stream failed: IllegalArgumentException: conversationId cannot be null

What memory keeps when the reader hangs up

One more thing I wanted to know before building a Stop button: if the client abandons an answer halfway, does the half answer enter the history? A test opens a raw socket for count 300, reads a few hundred bytes and hangs up, then asks a third question in the same session (output/12-memory-after-cancel.txt):

turn 1 'warm' completed, turn 2 'count 300' cancelled after a few tokens, then turn 3:
[stub] saw 5 messages. Earlier user turns: warm | count 300. You said: What do you remember?

Five messages: the system prompt, the first user turn, its answer, the cancelled user turn, and the new question. The cancelled turn’s question was stored; its partial answer was not, so the next prompt contains two user messages in a row. That is probably what you want (nothing half-finished is replayed as if the model had said it), but it is worth knowing before you decide how a “Stop” should look in the transcript.

Going deeper

• Chat Memory in Spring AI 2.0: JDBC, Redis and Windowed Conversations covers durable repositories, the 36-character conversation_id limit and per-user ids.
• Writing Custom Advisors in Spring AI 2.0 explains where the memory advisor sits relative to the others in the chain.
• In-memory means a restart or a second instance starts everyone from zero; with several instances you need a shared repository or sticky sessions. Neither was tested here.
• Spring AI reference: Chat Memory

The Stop button: cancelling the Flux is easy, stopping the model is the trap

The Stop button is a normal htmx button that posts to /chat/stop and replaces the connection element with a short “[stopped]” note. When htmx removes an element it runs the extensions’ cleanup, and the SSE extension closes the EventSource in that cleanup. The browser then closes the socket.

    /** Replaces the stream element; htmx cleans the removed element up and closes its EventSource. */
    @PostMapping(value = "/chat/stop", produces = MediaType.TEXT_HTML_VALUE)
    public String stop(@RequestParam int turn) {
        return "<em id=\"sse-" + turn + "\"> [stopped]</em>"
                + "<span id=\"stop-" + turn + "\" hx-swap-oob=\"delete\"></span>";
    }

From ChatController.stop. On the server, Spring MVC notices the closed socket when it next tries to write, and cancels the subscription. The doOnCancel log line is the usual “proof” that it worked. Here is the Stop click from the Chromium run, with the server log and the stub log side by side (output/07-stop-logs.txt):

-- app log (the turn started by the Stop-button test)
turn 6 start conversation=def8b65c-b58a-4677-b149-e71575426ebd message=count 200
turn 6 CANCELLED by client after 14 tokens
-- stub log
stub: request #6 model=stub-gpt stream=true messages=6 roles=system,user,assistant,user,assistant,user latest="count 200"
stub: stream #6 client closed the connection after 15 chunks

The Flux was cancelled after 14 tokens and the stub saw its connection close after 15 chunks. That looks like success, and with the fix described below it is. Without it, that was not what happened — I only saw the real picture because the stub counts how streams end, not just that they started.

The cancel that did not reach the model

With the fix switched off (--chat.cancel-fix=false), scripts/cancel-upstream.sh aborts a streaming request after one second, eight times, and asks the stub whether the upstream connection was closed. doOnCancel fired every time. The upstream connection was closed in one of the eight runs and left open in seven; in those seven the stub kept the stream open (its writes blocked once the socket buffers filled because nobody was reading). With the fix: closed in eight of eight (output/08-cancel-upstream.txt).

== chunk delay 20 ms, 8 runs, app args: --chat.cancel-fix=false
upstream closed: 1 of 8, left open: 7 of 8
== chunk delay 20 ms, 8 runs, app args: (none)
upstream closed: 8 of 8, left open: 0 of 8

The diagram shows where the chain splits. Everything left of the last box is Reactor and is proven by doOnCancel. The last hop is the HTTP client: Spring AI 2.0.2 ships its own OkHttp wrapper whose response close is response.body().close(). I took the problem apart without Spring AI at all: plain OkHttp 4.12.0, a reader thread blocked in a read, and a second thread closing the response.

plain OkHttp 4.12.0, 10 runs each, stub streaming 60 chunks at 20 ms, reader thread blocked in a read
  response.close() threw: java.lang.IllegalStateException: Unbalanced enter/exit
response.close() from another thread: upstream connection closed in 0 of 10 runs
call.cancel()     from another thread: upstream connection closed in 10 of 10 runs

That is OkHttpCloseReproTest (output/05-okhttp-close-vs-cancel.txt). Closing a response from a thread other than the one blocked reading it throws okio’s Unbalanced enter/exit and leaves the connection open in nearly every run (this committed run closed it in none of ten; earlier runs closed it in one of ten), while Call.cancel(), which OkHttp documents as safe from any thread, closed it ten times in ten. In the Spring AI path the cancel arrives on whichever thread Reactor uses, so the same failure shows up as the seven-of-eight leak above. I have not isolated why a few runs get lucky, and I have not reported this upstream.

Spring AI offers an extension point for exactly this: an OpenAiHttpClientBuilderCustomizer bean that can add an OkHttp interceptor. The interceptor wraps SSE response bodies so that close() calls Call.cancel() first:

    @Bean
    OpenAiHttpClientBuilderCustomizer cancelOnClose() {
        return builder -> builder.interceptor(chain -> {
            Response response = chain.proceed(chain.request());
            ResponseBody body = response.body();
            if (body == null || body.contentType() == null || !"event-stream".equals(body.contentType().subtype())) {
                return response;
            }
            var call = chain.call();
            return response.newBuilder().body(new ResponseBody() {
                @Override public MediaType contentType() { return body.contentType(); }
                @Override public long contentLength() { return body.contentLength(); }
                @Override public BufferedSource source() { return body.source(); }
                @Override public void close() {
                    call.cancel();
                    try {
                        body.close();
                    } catch (RuntimeException ignored) {
                        // the call is already cancelled; closing the body can only complain
                    }
                }
            }).build();
        });
    }

From CancelOnCloseConfig.java. With it, a test that opens a socket, reads 40,000 bytes and hangs up sees the Flux cancelled and the stub’s upstream stream closed within about 50 ms (output/04-cancel-with-fix.txt):

client read 40000 bytes then closed the socket
Flux cancelled on the server: 1
stub saw the upstream connection close: 1 (within 50 ms)
stub chunks written for this request: 5 of 400 requested

Why this matters in money. A model server that keeps generating after the user pressed Stop is still producing tokens you are billed for, and holding a GPU slot. Only the stub’s counters measured the leak here; I could not measure a real provider’s billing, and a provider speaking HTTP/2 over TLS closes a cancelled stream differently from the plain-HTTP/1.1 chunked stream my stub uses. Treat the 7-of-8 as a result for HTTP/1.1 chunked responses (what local OpenAI-compatible servers speak), and set a max_tokens ceiling regardless — I did not test that option.

Intermediate: why Spring MVC only notices a closed socket on the next write

A servlet container learns the client has gone when a write to the socket fails. While the model is silent nothing is written, so nothing fails and the Flux is not cancelled until the next token. The raw-socket test above hung up while tokens arrived every 50 ms, so the next write failed almost at once and the stub saw the close within 50 ms. With a slow first token or a long pause in a tool call, the same hang-up would be noticed late; I did not measure that case.

Going deeper

• Detection is on write: MVC notices a closed socket only when it next writes, so a model that is silent for ten seconds is not cancelled for ten seconds. The raw-socket test above saw 50 ms only because tokens were arriving every 50 ms.
• With a heartbeat comment line (: ping) every few seconds you would notice sooner; I did not add one.
• OkHttp documentation (Call.cancel and response handling)

Back-pressure: the slow reader and the push source

Reactive streams has a rule: a subscriber says how many items it can take (request(n)) and the publisher must not send more. Spring MVC’s adapter for a returned Flux asks for one element at a time. The model side is the problem: Spring AI’s OpenAI stream is built with Flux.create over an SDK reader thread that pushes chunks as they arrive from the socket. A push source cannot be told to wait; Flux.create’s default is to queue whatever the subscriber has not yet asked for.

To see it, scripts/backpressure.sh asks the stub for 600 chunks of 32 KiB (19.6 MB) written with no delay, while scripts/slow_client.py reads at most 64 KiB every 0.2 seconds on a raw socket. The script samples the stub’s progress, the client’s byte count and the app’s heap after a forced GC, for four ways of shaping the Flux (output/09-backpressure.txt). The default (none):

== chat.backpressure=none  (stub delay 0 ms, 600 chunks x 32 KiB, client reads 64 KiB every 0.2 s)
t= 1.0s  client got     0 KiB   stub: chunksWritten=0 finished=0 abortedByClient=0   app heap after GC: used 34669K
t= 2.8s  client got   197 KiB   stub: chunksWritten=452 finished=0 abortedByClient=0   app heap after GC: used 134096K
t= 5.5s  client got   896 KiB   stub: chunksWritten=603 finished=1 abortedByClient=0   app heap after GC: used 86306K
t= 8.9s  client got  1832 KiB   stub: chunksWritten=603 finished=1 abortedByClient=0   app heap after GC: used 109343K
t=13.3s  client got  2967 KiB   stub: chunksWritten=603 finished=1 abortedByClient=0   app heap after GC: used 116565K
-- app log (demand + outcome):
    213 web-side demand request(1)
    213 model-side demand request(1)
      1 CANCELLED by client after 213 tokens

The reading is in the numbers and the diagram. By 5.5 seconds the stub had written everything it was going to write while the client had received 896 KiB; the model server was never slowed. The app’s heap after GC went from 35–61 MB before the request to 86–147 MB across the unbounded modes, which is the queue, and the web-side demand really was request(1) 213 times. I sampled heap once per mode on a shared 2-vCPU machine, so treat those megabytes as indicative.

limitRate changes the demand, not the behaviour

limitRate(8) is the operator people reach for. It does what its name says — the model-side log shows request(8) once and then request(6) thirty-five times — and it changes nothing that matters, because the producer ignores demand:

== chat.backpressure=limitRate  (stub delay 0 ms, 600 chunks x 32 KiB, client reads 64 KiB every 0.2 s)
t= 1.0s  client got     0 KiB   stub: chunksWritten=0 finished=0 abortedByClient=0   app heap after GC: used 42630K
t= 2.8s  client got   242 KiB   stub: chunksWritten=603 finished=1 abortedByClient=0   app heap after GC: used 86598K
t= 5.2s  client got   896 KiB   stub: chunksWritten=603 finished=1 abortedByClient=0   app heap after GC: used 86745K
t= 8.5s  client got  1768 KiB   stub: chunksWritten=603 finished=1 abortedByClient=0   app heap after GC: used 109571K
t=12.9s  client got  2967 KiB   stub: chunksWritten=603 finished=1 abortedByClient=0   app heap after GC: used 116491K
-- app log (demand + outcome):
    213 web-side demand request(1)
     35 model-side demand request(6)
      1 model-side demand request(8)
      1 CANCELLED by client after 213 tokens

The stub had written all 603 chunks by 2.8 s here, and the heap is in the same range (output/09-backpressure.txt). delayElements(5 ms) behaved the same. The remedy that did something is a bounded queue with a policy, here onBackpressureBuffer(16, BufferOverflowStrategy.ERROR):

== chat.backpressure=bounded  (stub delay 0 ms, 600 chunks x 32 KiB, client reads 64 KiB every 0.2 s)
t= 1.0s  client got     0 KiB   stub: chunksWritten=0 finished=0 abortedByClient=0   app heap after GC: used 61061K
t= 2.7s  client got   133 KiB   stub: chunksWritten=256 finished=0 abortedByClient=0   app heap after GC: used 58420K
t= 5.1s  client got   787 KiB   stub: chunksWritten=256 finished=0 abortedByClient=1   app heap after GC: used 58420K
t= 8.5s  client got  1659 KiB   stub: chunksWritten=256 finished=0 abortedByClient=1   app heap after GC: used 30251K
t=12.8s  client got  2858 KiB   stub: chunksWritten=256 finished=0 abortedByClient=1   app heap after GC: used 30381K
-- app log (demand + outcome):
    144 web-side demand request(1)
      1 model-side demand request(unbounded)
      1 error reactor.core.Exceptions$OverflowException: The receiver is overrun by more signals than expected (bounded queue...)

After about 8 MB had been absorbed by the socket and the buffer, the queue overflowed, the stream was cancelled (and, thanks to the cancel fix, the stub saw the connection close after 256 chunks), and the heap stayed at 58 MB and then fell to 30 MB. The price is that a client which falls behind is disconnected instead of being buffered, which for a chat page is a reasonable policy and for a download is not.

Intermediate: what the three operators actually did to the demand

Spring MVC’s subscriber requested one element at a time in every mode (213 to 221 request(1) calls before the client left). limitRate(8) sits between the model and that subscriber and turns the model-side demand into request(8) followed by request(6) repeatedly, which is Reactor replenishing at 75 percent. delayElements keeps request(1) on both sides. In none of them did the stub slow down, because Flux.create queues what it cannot deliver. The bounded variant asks the model side for an unbounded amount on purpose and enforces the limit with the queue size instead.

Real tokens are tiny. My stream used 32 KiB chunks so that a slow reader would be visible in seconds. A real model emits tens of bytes per token, so an unbounded queue for one answer is small. The risk is many slow clients at once, or a very long answer, not one chat. I did not measure either.

Going deeper

• Spring AI with Docker Model Runner vs Ollama shows the throughput side of streaming: first-token latency and decode rate on real local engines.
• Reactor reference: backpressure and the overflow strategies of Flux.create / onBackpressureBuffer

Long answers: no 30-second cut-off here

A streamed answer is one long-lived request, and I expected to hit the servlet container’s async timeout (Tomcat’s default is 30 seconds). With no timeout configured anywhere in the module, a 700-token answer at 50 ms per token ran for 38 seconds and completed:

$ curl -sN "localhost:38180/chat/stream?message=count 700"   (700 tokens x 50 ms = 35 s)
<blank>
event:done
data:
<blank>
elapsed: 38 s
turn 1 complete, 700 tokens

Output: output/10-long-stream.txt. I have no explanation for why 30 seconds did not apply, so I would not rely on it: if your answers can run long, set spring.mvc.async.request-timeout explicitly and test with your own proxy in front. Proxies that buffer responses (nginx’s default does) are a separate trap I did not test.

Running it yourself

mvn -DskipTests package
mvn -q dependency:build-classpath -Dmdep.outputFile=target/cp.txt
java -cp "target/classes:$(cat target/cp.txt)" com.ankurm.chatui.stub.OpenAiStubServer 38181 20 &
java -jar target/chat-ui-htmx-1.0.0.jar          # http://localhost:38180/
scripts/run-all.sh                               # regenerates output/ (about five minutes)

These are the commands from the module README. mvn test alone runs seven tests (the chat flow on a random port and the OkHttp reproduction) and rewrites output/01 to 05, 11 and 12; the rest of output/ comes from the scripts and needs Playwright for Python with Chromium.

Should you even do this?

Probably yes for an internal tool or a chat widget in a server-rendered app; probably not for a rich chat product. htmx plus SSE keeps all state on the server and costs no build step, which is the right trade when the page is a form and a transcript. It is the wrong trade if you need optimistic UI, offline use, streamed markdown with partial-render fix-ups, or message editing. Two limits are built in: EventSource issues only GET requests and cannot set headers, so the user’s message rides in the URL (and in your access logs) and a bearer token cannot be sent — if that is a problem, the usual alternative is fetch with a streamed body, which is JavaScript you would write. And everything above was measured against a stub, one browser engine, one HTTP/1.1 connection style and one JVM.

Not tested

• Any real model provider, including its chunk sizes, usage chunks, HTTP/2 or TLS transport, billing after cancel and max_tokens.
• Firefox, Safari and mobile browsers (only headless Chromium 153 was driven).
• A reverse proxy or CDN in front, response buffering, idle timeouts, heartbeats.
• More than one app instance, a durable ChatMemoryRepository, authentication, CSRF protection for the POSTs, two simultaneous turns in one session.
• A WebFlux version of the same controller, and virtual threads.
• Why a few of the plain-OkHttp close runs succeed; whether newer openai-java or Spring AI releases changed the close path.

Further reading

• Companion code: chat-ui-htmx in asmhatre/spring-ai and its captured output
• Spring AI 2.0 in 10 Minutes: ChatClient on Spring Boot 4.1
• Chat Memory in Spring AI 2.0: JDBC, Redis and Windowed Conversations
• Writing Custom Advisors in Spring AI 2.0
• Spring AI with Docker Model Runner vs Ollama
• Server-Sent Events and WebSocket on Spring Boot 4
• Prompt Injection Defense in Spring AI
• Spring AI 1.x to 2.0 migration guide
• htmx: SSE extension
• WHATWG: Server-sent events
• Spring AI reference: ChatClient API
• Spring AI reference: Chat Memory
• Project Reactor reference