# db-migrations-expand-contract

Companion code for **[Zero-Downtime Database Migrations: Expand-Contract in Practice
with Spring Boot](https://ankurm.com/zero-downtime-database-migrations-expand-contract-spring-boot/)**
— every claim in that post traces to a test in this module, a real 4-deploy live run,
or a transcript in [`docs/output/`](docs/output).

## Versions

| Component | Version |
|---|---|
| Spring Boot | 4.1.1 |
| Flyway (via `spring-boot-starter-flyway`) | 12.4.0 |
| Jackson (via `spring-boot-starter-jackson`) | `tools.jackson` 3.1.5 |
| Database | H2 2.4.240, standalone TCP server mode |
| JDK | 25 (LTS) |

## Quickstart

```bash
mvn -DskipTests package        # builds target/db-migrations-expand-contract-1.0.0.jar
mvn test                       # regenerates every unit-test transcript in docs/output/
./scripts/run-all.sh           # the live exhibit: real 4-deploy rolling sequence + load generator
```

`run-all.sh` starts a standalone H2 TCP server, brings up two replicas on Stage 1,
runs all four deploys of the expand-contract sequence as an actual rolling deploy —
schema-only, code-only, schema-only, in the right order — and keeps a load generator
sending real HTTP traffic through the entire thing. It regenerates:

- [`docs/output/11-live-deploy-sequence.txt`](docs/output/11-live-deploy-sequence.txt) — the deploy log, phase by phase
- [`docs/output/12-load-generator-summary.txt`](docs/output/12-load-generator-summary.txt) — total/ok/error counts, by phase
- [`docs/output/13-schema-diagnostics-timeline.txt`](docs/output/13-schema-diagnostics-timeline.txt) — `/diag/schema` after each deploy

## The four deploys (Spring profile: `app.stage`)

| Stage | What it does | Deploy kind |
|---|---|---|
| 1 | Reads and writes only `email` | *(baseline, before this article starts)* |
| 2 | Writes both `email` and `email_address`; still reads `email` | Deploy 2 — code |
| 3 | Writes both columns; reads `COALESCE(email_address, email)` | Deploy 3 — code |
| 4 | Reads and writes only `email_address` | Deploy 4a — code |

Between stages, two schema-only migrations run with **zero application restarts**:
Deploy 1 (EXPAND, `V2__add_email_address_column.sql`) before Stage 2 ships, and
Deploy 4b (CONTRACT, `V3__drop_email_column.sql`) after every replica is confirmed on
Stage 4. See [`docs/01-the-problem-and-the-plan.md`](docs/01-the-problem-and-the-plan.md).

## Endpoints

| Endpoint | Purpose |
|---|---|
| `POST /customers`, `GET /customers/{id}`, `PUT /customers/{id}/email` | The API contract that never changes across all four stages — see [`CustomerController`](src/main/java/com/ankurm/expandcontract/customer/CustomerController.java) |
| `GET /diag/schema` | Live column list + row counts — the exhibit that shows the schema actually expanding and contracting. **Delete before shipping** (see [chapter 15](docs/15-production-checklist.md)) |
| `POST /admin/drain` | Publishes `ReadinessState.REFUSING_TRAFFIC` before a graceful shutdown — the `preStop`-hook pattern. **Unauthenticated in this demo — lock down before shipping** |
| `GET /actuator/health`, `GET /actuator/info` | Boot's own actuator endpoints |

## Documentation

1. [The problem and the plan](docs/01-the-problem-and-the-plan.md)
2. [The expand migration](docs/02-the-expand-migration.md)
3. [Why migrations run outside the app](docs/03-why-migrations-run-outside-the-app.md)
4. [The dual write](docs/04-the-dual-write.md)
5. [The read switch](docs/05-the-read-switch.md)
6. [The NOT NULL trap](docs/06-the-not-null-trap.md)
7. [The backfill window bug](docs/07-the-backfill-window-bug.md)
8. [The rolling-window proof](docs/08-the-rolling-window-proof.md)
9. [The contract migration](docs/09-the-contract-migration.md)
10. [What happens if you drop too soon](docs/10-what-happens-if-you-drop-too-soon.md)
11. [The load generator](docs/11-the-load-generator.md)
12. [The AUTO_SERVER trap](docs/12-the-auto-server-trap.md)
13. [Graceful shutdown vs. kill -9](docs/13-graceful-shutdown-vs-kill-9.md)
14. [The DDL lock window](docs/14-the-ddl-lock-window.md) — the module's central finding
15. [Production checklist](docs/15-production-checklist.md)

## Captured output

Every number quoted in the post and in the chapters above comes from a committed
transcript in [`docs/output/`](docs/output). Files `02`–`10` and `14` are written by a
`Transcript` helper while the JUnit test that produced them asserts the same numbers —
a transcript going stale fails the build. Files `11`–`13` come from the live
`run-all.sh` sequence.

| File | Source |
|---|---|
| [`02-expand-backward-compatible.txt`](docs/output/02-expand-backward-compatible.txt) | [`ExpandMigrationBackwardCompatibleTest`](src/test/java/com/ankurm/expandcontract/ExpandMigrationBackwardCompatibleTest.java) |
| [`04-dual-write-consistency.txt`](docs/output/04-dual-write-consistency.txt) | [`DualWriteConsistencyTest`](src/test/java/com/ankurm/expandcontract/DualWriteConsistencyTest.java) |
| [`06-not-null-trap.txt`](docs/output/06-not-null-trap.txt) | [`NotNullConstraintTrapTest`](src/test/java/com/ankurm/expandcontract/NotNullConstraintTrapTest.java) |
| [`07-backfill-window-bug.txt`](docs/output/07-backfill-window-bug.txt) | [`BackfillWindowBugTest`](src/test/java/com/ankurm/expandcontract/BackfillWindowBugTest.java) |
| [`08-mixed-stage-rolling-window.txt`](docs/output/08-mixed-stage-rolling-window.txt) | [`MixedStageRollingWindowTest`](src/test/java/com/ankurm/expandcontract/MixedStageRollingWindowTest.java) |
| [`09-contract-safety.txt`](docs/output/09-contract-safety.txt) | [`ContractSafetyTest`](src/test/java/com/ankurm/expandcontract/ContractSafetyTest.java) (happy path) |
| [`10-drop-too-soon.txt`](docs/output/10-drop-too-soon.txt) | [`ContractSafetyTest`](src/test/java/com/ankurm/expandcontract/ContractSafetyTest.java) (dropped too soon) |
| [`11-live-deploy-sequence.txt`](docs/output/11-live-deploy-sequence.txt) | [`scripts/run-all.sh`](scripts/run-all.sh) |
| [`12-load-generator-summary.txt`](docs/output/12-load-generator-summary.txt) | [`LoadGenerator`](src/main/java/com/ankurm/expandcontract/loadgen/LoadGenerator.java), via `run-all.sh` |
| [`13-schema-diagnostics-timeline.txt`](docs/output/13-schema-diagnostics-timeline.txt) | [`SchemaDiagnosticsController`](src/main/java/com/ankurm/expandcontract/diag/SchemaDiagnosticsController.java), via `run-all.sh` |
| [`14-ddl-collision-exception.txt`](docs/output/14-ddl-collision-exception.txt) | [`DdlCollisionExceptionTest`](src/test/java/com/ankurm/expandcontract/DdlCollisionExceptionTest.java) |
| [`14-ddl-silent-data-loss.txt`](docs/output/14-ddl-silent-data-loss.txt) | [`DdlSilentDataLossTest`](src/test/java/com/ankurm/expandcontract/DdlSilentDataLossTest.java) |

## Findings worth the trip

- H2's `AUTO_SERVER=TRUE` "shared embedded database" mode makes the first connecting
  process the de facto server for every other connection — killing that one replica
  during a rolling restart broke the *other* replica's "embedded" database entirely.
  Fixed by running H2 as an independent standalone TCP server ([chapter 12](docs/12-the-auto-server-trap.md)).
- Expanding a column without relaxing the retired column's `NOT NULL` constraint
  fails every Stage 4 `INSERT` from the first request onward — a genuine bug in this
  module's own first draft, reproduced and fixed in the same migration
  ([chapter 6](docs/06-the-not-null-trap.md)).
- A naive Stage 3 read of `email_address` alone returns `NULL` for any row a
  still-live Stage 1 replica wrote during Deploy 2's own rollout window — fixed with
  `COALESCE(email_address, email)` ([chapter 7](docs/07-the-backfill-window-bug.md)).
- H2 implements `ALTER TABLE ADD COLUMN` and `DROP COLUMN` by rebuilding the table,
  which can silently discard a row committed by a concurrent `INSERT` while the
  rebuild is mid-scan — with no exception thrown to the inserting connection.
  Reproduced directly, and confirmed to be an H2-specific behavior rather than a
  property of the expand-contract technique itself, since PostgreSQL documents both
  operations as metadata-only ([chapter 14](docs/14-the-ddl-lock-window.md) — the
  module's central finding).
- A real rolling restart, with graceful shutdown alone, still produces
  `ConnectException` bursts: the health-checked pool needs time to notice a draining
  instance before that instance is actually killed. A `/admin/drain` endpoint that
  publishes `ReadinessState.REFUSING_TRAFFIC` before `SIGTERM` closes that gap
  ([chapter 13](docs/13-graceful-shutdown-vs-kill-9.md)).
- The full live sequence — real HTTP traffic through a real 4-deploy rolling restart —
  finished at 99.98% success (30,905 of 30,911 requests), with every one of the six
  residual errors traced to a single, explained root cause rather than left as an
  unexplained miss ([chapter 11](docs/11-the-load-generator.md)).