AI & Machine Learning

Agentic Patterns in Spring AI: Routing, Orchestrator-Workers, Evaluator-Optimizer and Parallelization

Anthropic’s five agentic workflows (prompt chaining, routing, parallelization, orchestrator-workers, evaluator-optimizer) as plain Spring AI ChatClient code, tested offline with a scripted ChatModel that records every prompt, with the failure modes and the call-count cost of each.

Ankur Nov 6, 2026 36 min read

A single call to a language model is easy to reason about: one prompt in, one answer out. Real features need more. A support bot has to decide whether a message is about billing or a broken integration before it can answer well. A code assistant has to plan work whose shape it cannot know in advance. A commit-message generator has to be told when its first attempt is wrong. Each of those is more than one model call, and the moment there is more than one call, there is control flow — and control flow is where bugs live. Anthropic’s engineering article Building effective agents (19 December 2024) gives that control flow names. It separates workflows, where your code decides the sequence of model calls, from agents, where the model decides. It then lists five workflow patterns: prompt chaining, routing, parallelization (in two variants, sectioning and voting), orchestrator-workers and evaluator-optimizer. The patterns and names are Anthropic’s; Spring AI’s reference has a page on implementing them, Building Effective Agents, with runnable examples in the spring-ai-examples repository. What this article adds is the part those leave out: each pattern written as short plain ChatClient code, tested offline against a scripted model that records every prompt, so you can see the exact sequence of calls, how each pattern fails, and what the extra calls cost.

Versions, and an honest limit. Spring AI spring-ai-client-chat 2.0.2 and the Spring Boot 4.1.2 BOM, both GA in maven-metadata.xml and both dated 22 October 2026 on Maven Central. JDK 25.0.5 (LTS), JUnit 6.1.3. The module overrides the parent pom’s older 2.0.1 / 4.1.1 so that the other modules in the repository keep building against what they were verified with. All the code is in the agentic-patterns module of asmhatre/java-ai-agents, and every console block below is quoted from a file under agentic-patterns/output/, written by a test that asserts the same facts. No real model was called. The scripted replies prove what the control flow does with the answers it gets. They say nothing about how well any real model classifies, plans or judges.

Lead with the failures: what goes wrong in each pattern

The happy path of every pattern is a few lines and every other article has it. The days get lost in the failures, so here they are first, with the guard each gets here and the transcript that shows it.

Pattern	What goes wrong	Guard in the code	Transcript
Prompt chaining	Step 1 returns prose instead of the agreed format; step 2 then builds on garbage	A plain-Java gate stops the chain after call 1	02-chain-gate-failure.txt
Routing	The classifier answers with a sentence, a label you do not have, or nothing	Unparseable label falls back to a general handler and is flagged	04-routing-fallback.txt
Parallelization	One branch throws; or the voters split evenly	A failed branch becomes a marked section; a tie is UNSAFE	06-sectioning-branch-failure.txt, 07-voting.txt
Orchestrator-workers	The plan is not valid JSON, or has too many steps	One retry that names the error, then a direct single call	09-orchestrator-malformed-plan.txt
Evaluator-optimizer	The judge never says PASS, or says something that is not PASS	A round cap; only an exact PASS ends the loop	11-evaluator-max-rounds.txt

A workflow is code you control; an agent is a model that controls the code

Keep one distinction in your head and the rest of the article follows. In a workflow, your Java decides which model call happens next: if, for, a list of tasks. The model fills in text at each step but never chooses the route. In an agent, the model chooses the route: it decides which tool to call and when it is done. Anthropic’s article makes the same split and recommends the simplest thing that works, increasing complexity only when it demonstrably helps. That is why this article builds the five workflows and not an agent. The agent loop with tools is a different tool, covered in Tool Calling in Spring AI 2.0. Everything below uses one object, ChatClient, whose basics are in Spring AI 2.0 in 10 Minutes. Spring AI has no workflow DSL and the code here does not invent one: each pattern is a class of 45 to 75 lines (the five together are 284) in which the control flow is ordinary Java. That is a feature when you want to read, test and debug it, and a cost when you want something declarative.

Test the prompts, not the model: a scripted ChatModel

A pattern is only worth trusting if you can say exactly which prompts it sent. A real model cannot give you that in a test: it is slow, costs money, and answers differently each time. So the module tests every pattern against ScriptedChatModel, an implementation of Spring AI’s ChatModel interface that never touches the network. I confirmed the shape with javap on 2.0.2: the interface has exactly one abstract method that matters, ChatResponse call(Prompt), and a Prompt exposes its messages through getInstructions(). Implement that one method and the whole ChatClient stack runs on top of it. The scripted model picks a reply from rules (the first rule whose text appears in the system prompt wins, and each rule hands out its replies in order), records every prompt it saw split into system and user text, and stamps each call with a logical start and end tick, which later lets the cost test measure how many calls really ran one after another.

The diagram shows what is real and what is fake. Only the box on the right is fake. The pattern classes, the ChatClient, the way system(..) and user(..) become messages, and the JSON parsing behind entity(..) are all the real Spring AI code, which is what makes an assertion about the recorded prompts mean something. The heart of the model is the lookup and the recording (full file in the accordion below, source in ScriptedChatModel.java):

    private Object scripted(Call call) {
        for (Rule rule : rules) {
            boolean hit = rule.systemContains == null ? call.system.isEmpty() : call.system.contains(rule.systemContains);
            if (hit) {
                return rule.next();
            }
        }
        throw new AssertionError("no scripted reply for system=[" + call.system + "] user=[" + call.user + "]");
    }
}

Scripted does not mean meaningless, and it does not mean complete. A test that says “call 2 received exactly the text call 1 returned” is a real fact about your code and cannot flake. A test that says “the router picks BILLING” because you scripted BILLING proves only that the router parses and dispatches. Whether a real model says BILLING for a real complaint needs an evaluation set against the real model, as in Testing LLM Apps in Java.

Full listing: ScriptedChatModel.java

Source: ScriptedChatModel.java. It is about 180 lines because of the measuring, not the scripting. Rules are checked in order; whenNoSystem matches the direct-fallback call that has no system message; a scripted reply that is a RuntimeException is thrown instead of returned; rendezvousOf(n) blocks each call until n calls are inside call() at once, which is how the parallel tests prove real overlap without timing anything.

package com.ankurm.agentic;

import org.springframework.ai.chat.messages.AssistantMessage;
import org.springframework.ai.chat.messages.Message;
import org.springframework.ai.chat.messages.MessageType;
import org.springframework.ai.chat.model.ChatModel;
import org.springframework.ai.chat.model.ChatResponse;
import org.springframework.ai.chat.model.Generation;
import org.springframework.ai.chat.prompt.Prompt;

import java.util.ArrayDeque;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Deque;
import java.util.List;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicLong;

/**
 * A ChatModel that never touches the network. Replies come from rules: the first rule whose text appears in the
 * system prompt wins, and each rule hands out its scripted replies in order (the last one repeats). A scripted
 * reply may be a RuntimeException, which is thrown instead of returned. Every prompt the model sees is recorded
 * with a logical start and end tick, so tests can assert the exact prompt sequence and measure how many calls
 * really ran one after another.
 */
final class ScriptedChatModel implements ChatModel {

    /** One call as the model saw it. Ticks come from one shared counter, so they order events across threads. */
    static final class Call {
        final int index;
        final String system;
        final String user;
        final long start;
        volatile long end;

        Call(int index, String system, String user, long start) {
            this.index = index;
            this.system = system;
            this.user = user;
            this.start = start;
        }

        int chars() {
            return system.length() + user.length();
        }
    }

    private static final class Rule {
        final String systemContains;          // null means: matches only calls that have no system message
        final Deque<Object> replies = new ArrayDeque<>();
        Object last;

        Rule(String systemContains, Object[] scripted) {
            this.systemContains = systemContains;
            replies.addAll(List.of(scripted));
        }

        synchronized Object next() {
            if (!replies.isEmpty()) {
                last = replies.poll();
            }
            return last;
        }
    }

    private final List<Rule> rules = new ArrayList<>();
    private final List<Call> calls = Collections.synchronizedList(new ArrayList<>());
    private final AtomicLong tick = new AtomicLong();
    private final AtomicInteger inFlight = new AtomicInteger();
    private final AtomicInteger maxInFlight = new AtomicInteger();
    private long latencyMs;
    private CountDownLatch rendezvous;

    ScriptedChatModel when(String systemContains, Object... replies) {
        rules.add(new Rule(systemContains, replies));
        return this;
    }

    ScriptedChatModel whenNoSystem(Object... replies) {
        rules.add(new Rule(null, replies));
        return this;
    }

    /** Every call sleeps this long between its start tick and its end tick, so parallel calls overlap. */
    ScriptedChatModel latency(long ms) {
        this.latencyMs = ms;
        return this;
    }

    /** Every call blocks until n callers are inside call() at once. Sequential code would time out. */
    ScriptedChatModel rendezvousOf(int n) {
        this.rendezvous = new CountDownLatch(n);
        return this;
    }

    List<Call> calls() {
        synchronized (calls) {
            return List.copyOf(calls);
        }
    }

    int maxInFlight() {
        return maxInFlight.get();
    }

    /** Longest chain of calls where each one started after the previous one ended: the sequential depth. */
    int stages() {
        List<Call> sorted = new ArrayList<>(calls());
        sorted.sort((a, b) -> Long.compare(a.start, b.start));
        int[] depth = new int[sorted.size()];
        int best = 0;
        for (int i = 0; i < sorted.size(); i++) {
            depth[i] = 1;
            for (int j = 0; j < i; j++) {
                if (sorted.get(j).end < sorted.get(i).start) {
                    depth[i] = Math.max(depth[i], depth[j] + 1);
                }
            }
            best = Math.max(best, depth[i]);
        }
        return best;
    }

    int totalChars() {
        return calls().stream().mapToInt(Call::chars).sum();
    }

    @Override
    public ChatResponse call(Prompt prompt) {
        StringBuilder system = new StringBuilder();
        StringBuilder user = new StringBuilder();
        for (Message m : prompt.getInstructions()) {
            if (m.getMessageType() == MessageType.SYSTEM) {
                system.append(m.getText());
            } else if (m.getMessageType() == MessageType.USER) {
                user.append(m.getText());
            }
        }
        Call call;
        synchronized (calls) {
            call = new Call(calls.size() + 1, system.toString(), user.toString(), tick.incrementAndGet());
            calls.add(call);
        }
        int now = inFlight.incrementAndGet();
        maxInFlight.accumulateAndGet(now, Math::max);
        try {
            if (rendezvous != null) {
                rendezvous.countDown();
                if (!rendezvous.await(5, TimeUnit.SECONDS)) {
                    throw new IllegalStateException("calls did not overlap: this code ran them one after another");
                }
            }
            if (latencyMs > 0) {
                Thread.sleep(latencyMs);
            }
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            throw new IllegalStateException(e);
        } finally {
            inFlight.decrementAndGet();
            call.end = tick.incrementAndGet();
        }
        Object reply = scripted(call);
        if (reply instanceof RuntimeException e) {
            throw e;
        }
        return new ChatResponse(List.of(new Generation(new AssistantMessage((String) reply))));
    }

    private Object scripted(Call call) {
        for (Rule rule : rules) {
            boolean hit = rule.systemContains == null ? call.system.isEmpty() : call.system.contains(rule.systemContains);
            if (hit) {
                return rule.next();
            }
        }
        throw new AssertionError("no scripted reply for system=[" + call.system + "] user=[" + call.user + "]");
    }
}

Prompt chaining: fixed steps, with a gate between them

Prompt chaining is the simplest workflow: call the model, take its answer, put it in the next prompt. Anthropic’s description fits tasks that decompose cleanly into fixed subtasks, trading some latency for accuracy because each call has an easier job. The example here turns a git log into release notes in two steps: extract user-visible bullets, then write a two-sentence summary from them. The part that matters is the gate between the steps. It is ordinary Java that checks step 1’s output before step 2 spends a call on it.

The diagram’s argument is the red dashed arrow: when the gate fails, the second call never happens. The code is the whole pattern (source: PromptChain.java):

    public Result run(String gitLog) {
        String bullets = chat.prompt().system(EXTRACT).user(gitLog).call().content();
        gate(bullets);                                   // programmatic check, no model involved
        String summary = chat.prompt().system(WRITE).user(bullets).call().content();
        return new Result(bullets, summary);
    }

    static void gate(String bullets) {
        if (bullets == null || bullets.isBlank()) {
            throw new GateFailedException("step 1 returned nothing");
        }
        for (String line : bullets.split("\n")) {
            if (!line.isBlank() && !line.startsWith("- ")) {
                throw new GateFailedException("step 1 line is not a bullet: " + line);
            }
        }
    }
}

The test asserts the exact prompt sequence: two calls, the first system prompt is EXTRACT and its user text is the git log, the second system prompt is WRITE and its user text is exactly what call 1 returned (source: PromptChainTest.java). The transcript it writes:

# prompt chain: git log -> bullets -> gate -> summary
call 1 | system: You turn a git log into release-note bullets. Reply with... | user: abc123 add --json flag to the export command [+2 lines]
call 2 | system: You write release notes. Turn the bullets into a two-sen... | user: - Added a --json flag to the export command [+1 lines]
calls: 2
bullets: - Added a --json flag to the export command / - Fixed a crash on empty input files
summary: Export can now write JSON with --json. Empty input files no longer crash it.

Transcript: 01-prompt-chain.txt. Prompts are cut to one short line each; the test compares the full strings.

Now the failure. Suppose the model answers step 1 with a friendly preamble, “Sure! Here are the changes:”, before the bullets. A chain without a gate would pass that whole string to step 2 and produce a summary that quotes the preamble. With the gate, the chain stops after one call (02-chain-gate-failure.txt):

# prompt chain: step 1 breaks the bullet format
call 1 | system: You turn a git log into release-note bullets. Reply with... | user: abc123 add --json flag to the export command [+2 lines]
gate: step 1 line is not a bullet: Sure! Here are the changes:
calls: 1 (the summary call was not made)

A gate is cheaper than a retry and cheaper than a judge. It costs no model call at all. Put every check you can express in code — format, length, required fields, a number in range — in a gate, and keep model-based checking for the things code cannot express. The evaluator-optimizer pattern below is what you reach for when the check itself needs a model.

Going deeper: what to do when the gate fails

This module throws GateFailedException and lets the caller decide. The options are: retry step 1 once with the failure reason appended (the orchestrator below does exactly that for its plan), fall back to a single direct call, or surface the error. For a background job, fail loudly and count; for a chat reply, a direct call usually beats an error.

Step 1’s output is untrusted input to step 2; see prompt injection defense.
Typed gates: replace the string with a record via entity(..), covered in Structured Output in Spring AI 2.0.

Routing: classify first, then answer with the right prompt

Routing sends each input to a prompt written for its category. A billing question and a broken-integration report deserve different instructions, and one prompt that tries to serve both usually serves neither well. The cost is one cheap classification call in front of the real one, so routing always takes exactly two calls here.

The solid arrows are the normal case. The dashed arrow is the guard: the classifier’s reply is parsed into an enum, and anything that is not a known label goes to the general handler and is flagged, rather than being guessed at (source: Router.java).

    public Routed handle(String message) {
        String raw = chat.prompt().system(CLASSIFY).user(message).call().content();
        Category category;
        boolean fellBack = false;
        try {
            category = Category.valueOf(raw.trim().toUpperCase(Locale.ROOT).replaceAll("[.!\\s]+$", ""));
        } catch (RuntimeException notALabel) {           // IllegalArgumentException or NullPointerException
            category = Category.GENERAL;
            fellBack = true;
        }
        String answer = chat.prompt().system(HANDLERS.get(category)).user(message).call().content();
        return new Routed(category, fellBack, raw, answer);
    }
}

Three messages, three classifier answers, three different second-call system prompts. The test asserts that the second call’s system prompt is the handler for the category (03-routing.txt):

message: I was charged twice this month
call 2 | system: You are a billing specialist. Answer in two sentences an... | user: I was charged twice this month
category: BILLING, fell back: false
message: The connector returns 401 since Monday
call 2 | system: You are a support engineer. Answer in two sentences and ... | user: The connector returns 401 since Monday
category: TECHNICAL, fell back: false
message: Do you have an office in Pune?
call 2 | system: You are a friendly support agent. Answer in two sentence... | user: Do you have an office in Pune?
category: GENERAL, fell back: false

Now the classifier misbehaves. The test feeds the router four different replies to the same message about a 401 error (04-routing-fallback.txt):

classifier said [Technical.] -> TECHNICAL, fell back: false
classifier said [I think this is probably a billing question] -> GENERAL, fell back: true
classifier said [REFUNDS] -> GENERAL, fell back: true
classifier said [] -> GENERAL, fell back: true

Read the second line carefully. “I think this is probably a billing question” contains the word billing, and the router still refuses it. That is deliberate: matching a word inside prose is how a message that says “this is not a billing issue” ends up with the billing agent. The cost is that a chatty model sends more traffic to the general handler, which is why the fallback is flagged, so you can count it.

The fallback catches unparseable labels, not wrong ones. If the model confidently answers BILLING for a technical problem, the code has no way to know. The guards for that are outside this module: log the label with the message, sample and review the routes, and keep an evaluation set. A fallback that fires for 30% of traffic is also a finding — usually that the classification prompt needs a worked example, or that the categories overlap.

Going deeper: routing with a typed result instead of a word

Parsing a single word by hand is fine for three categories. For anything with more fields, ask for a record instead: ChatClient‘s entity(Class) builds a JSON schema from the type, appends instructions to the prompt, and parses the reply. Structured Output in Spring AI 2.0 covers that and the validation advisor that retries on a bad shape.

Routing between models instead of prompts is the same pattern: see Jev with Spring AI.

Parallelization: sectioning and voting on virtual threads

Parallelization runs several model calls at once and combines the answers. Anthropic names two variants. Sectioning splits a task into independent subtasks, each with its own prompt: here, one reviewer each for security, performance and readability, all looking at the same code. Voting runs the same prompt several times and takes a majority, which is useful where one wrong answer is expensive: here, three voters decide whether a text is safe to publish.

Both halves of the diagram cost three calls and one stage. Sectioning merges in plain Java, with no fourth call; voting counts ballots. The code uses CompletableFuture on a virtual-thread executor, which is stable on JDK 25 (source: Parallelizer.java).

    public List<Section> sectioning(String code) {
        try (var pool = Executors.newVirtualThreadPerTaskExecutor()) {
            List<CompletableFuture<Section>> futures = ASPECTS.stream()
                    .map(aspect -> CompletableFuture.supplyAsync(() -> {
                        try {
                            String text = chat.prompt().system(aspectPrompt(aspect)).user(code).call().content();
                            return new Section(aspect, text, false);
                        } catch (RuntimeException e) {
                            return new Section(aspect, "unavailable: " + e.getMessage(), true);
                        }
                    }, pool))
                    .toList();
            return futures.stream().map(CompletableFuture::join).toList();   // ASPECTS order, not finish order
        }
    }

Proving that calls really overlapped is the part a timing assertion gets wrong, because timing makes tests flaky. The scripted model has a rendezvous: each call blocks until three callers are inside call() at the same time, and gives up after five seconds. Sequential code would time out and fail the test; parallel code passes instantly. The test also asserts the three system prompts in any order, because arrival order is not deterministic (05-sectioning.txt):

# sectioning: three prompts, same input, merged in plain Java
call | system: You review Java code for performance only. Reply with on... | user: String q = "select * from users where id=" + id;
call | system: You review Java code for readability only. Reply with on... | user: String q = "select * from users where id=" + id;
call | system: You review Java code for security only. Reply with one s... | user: String q = "select * from users where id=" + id;
calls: 3, most calls in flight at once: 3
security: SQL is built by string concatenation, which allows injection.
performance: select * pulls every column when one row is needed.
readability: The query string should be a named constant.

The merged result comes back in aspect order (security, performance, readability), not finish order, because the code joins the futures in the order it created them. Now one branch fails. The performance reviewer throws a rate-limit error; the other two are unaffected, and the failed branch is marked rather than silently dropped (06-sectioning-branch-failure.txt):

# sectioning: the performance branch throws
security SQL is built by string concatenation, which allows injection.
performance [FAILED] unavailable: 429 rate limited
readability The query string should be a named constant.

Voting has its own failure: the voters disagree, or one of them does not answer with either word. Three runs, with the scripted ballots shown on each line (07-voting.txt):

ballots SAFE | UNSAFE | UNSAFE -> UNSAFE (safe=1, unsafe=2, spoiled=0)
ballots SAFE | SAFE | Probably fine! -> SAFE (safe=2, unsafe=0, spoiled=1)
ballots SAFE | UNSAFE | Hmm -> UNSAFE (safe=1, unsafe=1, spoiled=1)

The third line is a tie, one SAFE against one UNSAFE, with a third ballot that was neither. The rule here is that a tie is UNSAFE, and an unparseable ballot is not a vote. Both are policy choices you should make deliberately for your own use: for a safety check, ties go to caution; for a “is this spam?” filter that must not drop real mail, you might choose the opposite.

Voting multiplies cost to buy confidence, not speed. Three voters are three times the spend for one stage of waiting. It helps when the individual votes are independent and sometimes wrong. If the model is wrong the same way every time — the usual failure with a deterministic prompt at low temperature — three identical wrong votes are still wrong, and you paid triple.

Going deeper: StructuredTaskScope is still a preview API on JDK 25

The natural tool for “start these tasks together, and cancel the rest if one fails” is StructuredTaskScope, and the brief for this module was to use it if it was stable. It is not stable on JDK 25. I checked by compiling a small probe (ScopeProbe.java) with and without the flag, and the script that captured it is probe-structured-task-scope.sh. Without --enable-preview the compiler refuses; with it, the code compiles, class files are marked as preview (version 69.65535) and only run with the flag again (14-structured-task-scope-preview.txt):

# JDK: openjdk version "25.0.5" 2026-10-20 LTS
$ javac --release 25 ScopeProbe.java
ScopeProbe.java:1: error: StructuredTaskScope is a preview API and is disabled by default.
import java.util.concurrent.StructuredTaskScope;
                           ^
  (use --enable-preview to enable preview APIs)
ScopeProbe.java:6: error: StructuredTaskScope is a preview API and is disabled by default.
        try (var scope = StructuredTaskScope.open()) {
                         ^
  (use --enable-preview to enable preview APIs)
2 errors
$ javac --release 25 --enable-preview ScopeProbe.java
Note: ScopeProbe.java uses preview features of Java SE 25.
Note: Recompile with -Xlint:preview for details.
$ java -cp preview-classes ScopeProbe
Error: LinkageError occurred while loading main class ScopeProbe
	java.lang.UnsupportedClassVersionError: Preview features are not enabled for ScopeProbe (class file version 69.65535). Try running with '--enable-preview'
$ java --enable-preview -cp preview-classes ScopeProbe
security: ok | performance: ok

That is why the module uses CompletableFuture and virtual threads: both are final, and the module builds with no flags. The cost is that CompletableFuture does not cancel sibling tasks when one fails; the sectioning code here deliberately lets every branch finish and marks the failed one, and the voting code waits for all ballots. If you need cancel-on-failure, that is the reason to adopt the preview API and accept the flags. The status of the API across releases, including that its API changed between previews, is laid out in Structured Concurrency Is Final in JDK 28; I did not re-verify the JDK 28 claims there, only the JDK 25 behaviour above.

One more note: a real provider has rate limits. Three parallel calls per request become three hundred under load, so cap concurrency with a semaphore before production.

Orchestrator-workers: the model writes the plan, your code runs it

Parallelization splits work you can describe in advance. Orchestrator-workers is for work you cannot: “add a --json flag to this CLI” touches however many files it touches. One call, the orchestrator, breaks the task into steps at run time. One call per step does the work. A final call synthesises the results. The number of calls is therefore not known until the plan arrives.

The diagram shows the variable fan-out and, in the dashed red box, the guard for a bad plan. The plan is a typed record, parsed by entity(Plan.class), so a malformed reply is an exception, not a string you have to inspect (source: Orchestrator.java):

    public Result run(String task) {
        List<String> errors = new ArrayList<>();
        String userPrompt = task;
        for (int attempt = 1; attempt <= 2; attempt++) {
            try {
                Plan plan = chat.prompt().system(PLANNER).user(userPrompt).call().entity(Plan.class);
                validate(plan);
                return execute(task, plan, attempt, errors);
            } catch (RuntimeException e) {
                errors.add(e.getClass().getSimpleName());
                userPrompt = task + "\n\nYour previous reply could not be used (" + e.getClass().getSimpleName()
                        + "). Reply with valid JSON only.";
            }
        }
        // Two bad plans: stop orchestrating and answer directly. Worse, but it is an answer.
        String direct = chat.prompt().user(task).call().content();
        return new Result(null, 2, List.of(), direct, true, errors);
    }

    static void validate(Plan plan) {
        if (plan == null || plan.steps() == null || plan.steps().isEmpty() || plan.steps().size() > MAX_STEPS) {
            throw new IllegalStateException("plan must have 1.." + MAX_STEPS + " steps");
        }
    }

What does entity(Plan.class) actually send? The test records the planner call and writes it out. The system message is just the planner instruction; the user message is the task followed by the format instructions and JSON schema that Spring AI generated from the record (13-planner-prompt-as-sent.txt):

# system message the planner call carried
You are a planner. Break the task into 1 to 5 independent steps. Each step has a title and an instruction.
# user message the planner call carried: the task, then what entity(Plan.class) appended
Add a --json output flag to the export CLI
Your response should be in JSON format.
Do not include any explanations, only provide a RFC8259 compliant JSON response following this format without deviation.
Do not include markdown code blocks in your response.
Remove the ```json markdown from the output.
Here is the JSON Schema instance your output must adhere to:
```{
  "$schema" : "https://json-schema.org/draft/2020-12/schema",
  "type" : "object",

That is the first 12 lines of 13-planner-prompt-as-sent.txt; the rest is the schema for steps, title and instruction. Because it is appended to the user message, the prompt in the transcript below shows [+29 lines] after the task.

The happy path is five calls for a three-step plan: one plan, three workers, one synthesis (08-orchestrator-workers.txt):

plan steps: [Parse the flag, Serialise rows, Document it]
calls: 5 (1 plan + 3 workers + 1 synthesis)

Each worker sees the task, its own step title and instruction, and nothing else; the test asserts the first worker’s exact prompt. Now the failure this pattern is famous for: the plan is wrong. The test runs three scenarios (09-orchestrator-malformed-plan.txt):

# (a) prose first, valid JSON on the retry
plan errors: [StreamReadException], attempts: 2, calls: 6
# (b) seven steps twice: over the limit of 5
plan errors: [IllegalStateException, IllegalStateException], fell back: true, answer: Just do it: one direct answer.
# (c) prose twice
plan errors: [StreamReadException, StreamReadException], fell back: true

In (a) the model answers in prose first. entity(..) throws a Jackson 3 StreamReadException (the name is recorded from the real exception), the orchestrator retries once with a message saying the previous reply could not be used, and the second reply parses. That cost six calls instead of five. In (b) the JSON is valid but has seven steps, over the limit of five: the code validates the plan itself, since a schema cannot say “at most five” the way the prompt does, and rejects it. In (c) the model never produces JSON. After two failed plans, (b) and (c) fall back to a single direct call with the original task, which is three calls for what a single call would have done.

An unvalidated plan is an unbounded loop. The line that decides how many workers run is plan.steps().size(), which the model wrote. Cap it, as validate does, and reject plans outside the cap before any worker is paid for. Without the cap, one over-eager plan of 40 steps is 40 calls before anyone notices. The retry doubles planner cost in the bad case too, which is why it is exactly one.

Going deeper: the worker and synthesis calls

The remaining half of the class (source: Orchestrator.java). Workers run one after another here, which keeps the prompt order deterministic for the test. They are independent, so they could run through the same virtual-thread fan-out as sectioning; by construction that would cut the stages from five to three. I did not run that variant, so the table below reports what was measured: five stages.

    private Result execute(String task, Plan plan, int attempts, List<String> errors) {
        List<String> results = new ArrayList<>();
        for (Step step : plan.steps()) {
            results.add(chat.prompt().system(WORKER)
                    .user("Task: " + task + "\nStep: " + step.title() + "\nInstruction: " + step.instruction())
                    .call().content());
        }
        StringBuilder joined = new StringBuilder("Task: " + task + "\n");
        for (int i = 0; i < results.size(); i++) {
            joined.append("\n## ").append(plan.steps().get(i).title()).append('\n').append(results.get(i)).append('\n');
        }
        String answer = chat.prompt().system(SYNTHESISER).user(joined.toString()).call().content();
        return new Result(plan, attempts, results, answer, false, errors);
    }
}

The retry message names the exception class and nothing else, so the model’s bad reply is not fed back in. Whether naming the error helps a real model produce valid JSON is not something scripted tests can show.

Evaluator-optimizer: generate, judge, revise, and stop

Evaluator-optimizer is the loop. One call writes a draft; a second call judges it against criteria and returns either PASS or feedback; the feedback goes into the next draft. It earns its cost only when there are clear criteria and a revision demonstrably improves the result. The example generates a git commit message under three rules: imperative mood, at most 50 characters, no trailing period.

Two exits, and the second is the one that matters: PASS returns the draft, and the round cap returns the best draft so far, flagged as not converged. Without the cap, a judge that never passes spends two calls a round indefinitely (source: EvaluatorOptimizer.java).

    public Result run(String task, int maxRounds) {
        List<String> drafts = new ArrayList<>();
        List<String> feedback = new ArrayList<>();
        String draft = null;
        for (int round = 1; round <= maxRounds; round++) {
            String user = draft == null ? task
                    : task + "\nPrevious draft: " + draft + "\nFeedback: " + feedback.getLast() + "\nRevise the draft.";
            draft = chat.prompt().system(GENERATOR).user(user).call().content().trim();
            drafts.add(draft);
            String verdict = chat.prompt().system(EVALUATOR).user(draft).call().content().trim();
            if (verdict.equalsIgnoreCase("PASS")) {
                return new Result(draft, true, round, drafts, feedback);
            }
            // Anything that is not exactly PASS counts as a failure, so a rambling judge cannot pass a draft by accident.
            feedback.add(verdict.regionMatches(true, 0, "FAIL:", 0, 5) ? verdict.substring(5).trim() : verdict);
        }
        return new Result(draft, false, maxRounds, drafts, feedback);   // best effort, flagged as not converged
    }
}

First the loop that works: draft 1 is in the past tense, draft 2 has a trailing period, draft 3 passes. The test asserts all six prompts in order, including that the revision prompt carries the previous draft and the judge’s feedback (10-evaluator-converges.txt):

# evaluator-optimizer: three rounds, PASS on the third
converged: true, rounds: 3, calls: 6
best: Add --json output flag
feedback: [Use the imperative mood., Remove the trailing period.]

Now the loop that does not converge. Every scripted draft is over 50 characters (the test asserts that), and the judge says FAIL, then something that is not PASS, then FAIL again, with a cap of three rounds (11-evaluator-max-rounds.txt):

# evaluator-optimizer: the judge never says PASS, maxRounds = 3
converged: false, rounds: 3, calls: 6
best so far (returned, flagged): Add a --json output flag to the export command, CLI (51 chars)
feedback: [Longer than 50 characters., Looks good to me!, Still longer than 50 characters.]

The cap, not the judge, ended the loop: six calls, converged: false, and the caller gets the best draft so far plus a flag. The middle reply, “Looks good to me!”, was not accepted as a pass: only the exact word PASS ends the loop, so a chatty judge cannot approve a draft by accident. Its text was recorded as feedback, which is a small hazard of its own, because the next draft is told to act on “Looks good to me!”.

The judge is part of the product, and it can be wrong too. A judge that approves bad drafts makes the loop a more expensive single call. A judge that rejects good ones burns rounds until the cap. For a rule like “at most 50 characters” you do not need a model at all — a gate in code is exact and free. Use a model judge for criteria code cannot check, and measure it, as in Testing LLM Apps in Java, before trusting it to steer a loop.

Going deeper: making the judge cheaper and the loop safer

Run the code-checkable rules first and call the model judge only on drafts that pass them; the example above would then reject the over-length drafts without a call.
Return a typed verdict (a record with pass and feedback) with entity(..) instead of parsing PASS by hand; see Structured Output in Spring AI 2.0.
Log the round count: a rising average is the early warning that generator and judge disagree.

What it costs: calls, stages and characters

Every pattern above buys something with extra calls. Two numbers matter and they are different. Calls multiply money (each call is billed). Stages multiply waiting: the length of the longest chain of calls where each starts after the previous one ended. Parallel calls add to the first and not the second. The cost test runs every happy path against a scripted model with 100 ms of fake latency per call and measures both from the logical start and end ticks of each recorded call, rather than counting them by hand (12-cost-latency.txt):

# calls, sequential stages and prompt characters per pattern (scripted model, 100 ms fake latency per call)
pattern                            calls  stages in flight prompt chars
single call (baseline)                 1       1         1          130
prompt chain                           2       2         1          430
routing                                2       2         1          251
sectioning x3                          3       1         3          342
voting x3                              3       1         3          261
orchestrator, 3 steps                  5       5         1         1929
evaluator, passes in round 3           6       6         1         1293
evaluator, hits the 3-round cap        6       6         1         1372

The chart draws the same numbers: blue bars are calls and amber bars are stages. The two parallel rows are the only ones where blue is longer than amber. The table below divides each row by the single-call baseline, which is the honest comparison, because it is what you would do instead.

Setup	Calls	Money vs one call	Stages	Waiting vs one call	Note
Prompt chain	2	2×	2	2×	a failed gate stops at 1 call
Routing	2	2×	2	2×	always two
Sectioning	3	3×	1	1×	three at once; merge is free
Voting	3	3×	1	1×	three at once; confidence, not speed
Orchestrator, 3 steps	5	5×	5	5×	1 + N + 1; a bad plan adds a call, see above
Evaluator, passes in round 3	6	6×	6	6×	2 per round
Evaluator, hits the cap of 3	6	6×	6	6×	the cap bounds it at 2 × rounds

The multipliers are call counts, not dollars: they assume every call costs the same, and it does not. Prompt chars is the summed length of every system and user message, a rough proxy for input size and not tokens; the rows use different inputs, so read it only as a scale (the orchestrator’s synthesis call re-sends every worker’s result, hence the largest figure). Latency per call, token prices and output length are not measured, because there is no real model. Multiply the counts by your own measured latency and price; the observability article shows where those numbers come from in a running Spring AI app.

Going deeper: how stages is measured

Each recorded call gets a start tick and an end tick from one shared counter, and the model sleeps for the fake latency between the two. A call b is “after” a when a’s end tick is lower than b’s start tick. Stages is the longest chain under that relation (ScriptedChatModel.stages(), ScriptedChatModel.java). Parallel calls start before any of them ends, so they form no chain and count once. It depends on the 100 ms being long compared with thread start-up; raise it on a heavily loaded CI machine.

Spring AI and LangChain4j for the same shapes

This site already has the other framework’s version of this story: Agentic Workflows with LangChain4j: Sequential, Parallel, Loop and Supervisor Agents. It builds a travel planner from small agents wired by LangChain4j’s agentic module. That module is a beta (1.22.0-beta32 in that article), so everything in it may change. The patterns line up like this, and the table is my reading of the two articles, not a feature-for-feature claim:

Anthropic pattern	Closest LangChain4j shape in that article	Here
Prompt chaining	Sequence of agents, sharing a scope by name	Two calls; the answer is passed as a string
Routing	Not covered in that article	An enum and a map of prompts
Parallelization	Parallel agents	`CompletableFuture` on virtual threads
Orchestrator-workers	Supervisor, where the model chooses the order	A typed plan the code validates and executes
Evaluator-optimizer	Loop until a condition holds	A `for` loop with a cap and a judge call

The difference in approach is the real comparison. LangChain4j gives you declarations: you describe agents and the framework runs the shape, including the shared scratchpad. Spring AI gives you the client and nothing in between, so each pattern here is code you can read top to bottom and set a breakpoint in, and you write the retries, the caps and the fallbacks yourself. If you want the framework to own the control flow, look at the other article; if you want tests that name the exact prompts, writing the flow yourself made that easy.

Should you even build this?

Usually not yet. Each pattern above multiplies calls and adds failure modes, and the table shows by how much. Anthropic’s own advice in the article these patterns come from is to find the simplest solution possible and add complexity only when it demonstrably improves outcomes. If a single well-written prompt with the right context answers your inputs correctly, stop there. Add a chain when the steps are fixed and one prompt keeps getting one wrong; routing when a dedicated prompt measurably beats a shared one; parallelization when latency is the problem or a wrong answer is worth paying for votes; orchestrator-workers only when you cannot list the subtasks in advance; evaluator-optimizer only when you can state the criterion and show the second draft is better. None of these is an agent: your code, not the model, decides what happens next, and letting the model choose its own steps brings its own cost and safety questions. Build the evaluation first. A pattern you cannot measure is just a more expensive way to be wrong.

Run it yourself

JDK 25 and Maven on the path, no API key, from the repository root:

$ mvn test -pl agentic-patterns
$ bash agentic-patterns/scripts/run-all.sh

The first command runs 12 tests and rewrites output/01 to 13; the script also regenerates output/14. Both commands are in the module README (README.md) and the script is run-all.sh. The module does not depend on Spring Boot at runtime: Boot appears only as a BOM for dependency versions and for the test starter.

No Comments yet!

Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.