spring-boot-demo/db-migrations-expand-contract/docs/11-the-load-generator.md

# 11. The load generator

[← 10. What happens if you drop too soon](10-what-happens-if-you-drop-too-soon.md) · [Next: 12. The AUTO_SERVER trap →](12-the-auto-server-trap.md)

Every other chapter so far proves a property of the technique with a JUnit test
against a shared `JdbcClient` — real code, real SQL, no mocks, but also no real HTTP,
no real process restarts, no real timing pressure. `scripts/run-all.sh` is where all
three of those show up: two real Spring Boot processes, a real standalone database, a
real rolling restart between each stage, and
[`LoadGenerator`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/main/java/com/ankurm/expandcontract/loadgen/LoadGenerator.java)
sending continuous HTTP traffic through the whole sequence.

## What it does

Eight worker threads, each running the same loop: pick a healthy backend, then create
a customer (50% of the time), read one it already knows about (30%), or update one
(20%). Every read and update verifies the response body matches what the load
generator itself expects — not just the HTTP status code:

```java
JsonNode node = MAPPER.readTree(resp.body());
String returnedEmail = node.get("email").asString();
if (!target.email().equals(returnedEmail)) {
    c.recordError("read-consistency-mismatch");
    return;
}
```

A 200 with the wrong email in the body would be a much worse bug than a 500, and a
naive load test that only checks status codes would never catch it.

## Health-checked traffic, not a raw hose

`LoadGenerator` polls `/actuator/health` on both ports every 300ms and only sends
traffic to backends it currently believes are up:

```java
private static final int UNHEALTHY_THRESHOLD = 2;
```

A backend needs two consecutive failed checks before it's removed from rotation —
this exists because this whole sequence runs two JVMs on a shared, small sandbox, and
one replica's cold-start CPU burst can make its *sibling* miss a single health check
without actually being down. Requiring two consecutive failures is the same debounce
a real load balancer's health check threshold gives you, and skipping it turned
transient slowness into false `no-healthy-backend` errors in an earlier version of
this test.

That fix mattered enough to also show up in
[`start-instance.sh`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/scripts/start-instance.sh),
on the other side of the same problem — cutting each replica's own startup CPU cost
so it's less likely to starve its sibling in the first place:

```bash
nohup java -XX:TieredStopAtLevel=1 -XX:+UseSerialGC -Xms128m -Xmx256m \
    -jar "$JAR" --server.port="$PORT" --app.stage="$STAGE" \
```

## Per-thread state, not shared state

Each worker thread owns a private `ArrayDeque` of customer ids it has created — never
shared with the other seven threads:

```java
workers.submit(() -> {
    try {
        workerLoop(new ArrayDeque<>());
```

An earlier version shared one pool across all eight threads, and produced
`read-consistency-mismatch` errors that had nothing to do with the server at all: two
threads racing to update the *same* shared id could leave the pool holding a stale
expected value, so a perfectly correct server response looked like a bug. Giving each
thread exclusive ownership of the rows it creates removes that entire class of false
positive while still hammering both replicas concurrently.

## The result

```
Load generator summary
=======================
Total requests: 30911
Successful:     30905
Errors:         6

By phase:
  04a-stage4-soak              ok=2130     error=0
  04b-contract-migration       ok=336      error=0
  04a-deploy-stage4-rollout    ok=4147     error=0
  03-stage3-soak               ok=2116     error=0
  03-deploy-stage3-rollout     ok=4645     error=0
  00-baseline-soak             ok=1893     error=0
  05-final-soak                ok=7118     error=4
      - read-http-404            2
      - update-http-404          2
  02-stage2-soak               ok=2068     error=0
  01-expand-migration          ok=1773     error=1
      - read-http-404            1
  02-deploy-stage2-rollout     ok=4679     error=1
      - update-http-404          1
```

Full transcript:
[`docs/output/12-load-generator-summary.txt`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/docs/output/12-load-generator-summary.txt),
and the phase-by-phase deploy log this run came from:
[`docs/output/11-live-deploy-sequence.txt`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/docs/output/11-live-deploy-sequence.txt).

30,905 of 30,911 requests succeeded — 99.98%, across a real rolling restart through
all four deploys. The six errors are all `404`s, not `500`s: a request for a customer
id that genuinely wasn't found, not a crash. Every one of them traces to the same
root cause, and it's the most interesting finding in this whole module — see
[chapter 14](14-the-ddl-lock-window.md).

## Going deeper

- The graceful-shutdown-and-drain sequence that gets the *rolling restart* portion of
  this run down to zero client-visible errors on its own is
  [chapter 13](13-graceful-shutdown-vs-kill-9.md) — the six remaining errors above
  have a different cause entirely, isolated in chapter 14.
- Every deploy's schema state during this exact run, captured live:
  [`docs/output/13-schema-diagnostics-timeline.txt`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/docs/output/13-schema-diagnostics-timeline.txt),
  via [`SchemaDiagnosticsController`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/main/java/com/ankurm/expandcontract/diag/SchemaDiagnosticsController.java).

[← 10. What happens if you drop too soon](10-what-happens-if-you-drop-too-soon.md) · [Next: 12. The AUTO_SERVER trap →](12-the-auto-server-trap.md)