Add db-migrations-expand-contract: zero-downtime schema migrations proven with a real 4-deploy rolling run
Companion code for Zero-Downtime Database Migrations: Expand-Contract in Practice with Spring Boot: a full expand/migrate-writes/migrate-reads/contract sequence run as an actual rolling deploy across two live replicas, with a load generator sending continuous HTTP traffic through all four deploys (99.98% success, every residual error traced to a root cause rather than left unexplained). Findings include a real NOT NULL constraint trap in the expand migration, a backfill-window bug in the read switch, H2's AUTO_SERVER=TRUE single-point-of-failure behavior under a rolling restart, the drain-before-SIGTERM fix needed to close a health-check gap during graceful shutdown, and H2 silently discarding a concurrently committed INSERT during an ALTER TABLE ADD/DROP COLUMN rebuild - confirmed, by primary source, to be an H2-specific behavior rather than a property of the technique itself. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_019Fb7vW8vLyLKngBc4R3huA
This commit is contained in:
Claude Sonnet 5
@@ -0,0 +1,126 @@
|
||||
# db-migrations-expand-contract
|
||||
|
||||
Companion code for **[Zero-Downtime Database Migrations: Expand-Contract in Practice
|
||||
with Spring Boot](https://ankurm.com/zero-downtime-database-migrations-expand-contract-spring-boot/)**
|
||||
— every claim in that post traces to a test in this module, a real 4-deploy live run,
|
||||
or a transcript in [`docs/output/`](docs/output).
|
||||
|
||||
## Versions
|
||||
|
||||
| Component | Version |
|
||||
|---|---|
|
||||
| Spring Boot | 4.1.1 |
|
||||
| Flyway (via `spring-boot-starter-flyway`) | 12.4.0 |
|
||||
| Jackson (via `spring-boot-starter-jackson`) | `tools.jackson` 3.1.5 |
|
||||
| Database | H2 2.4.240, standalone TCP server mode |
|
||||
| JDK | 25 (LTS) |
|
||||
|
||||
## Quickstart
|
||||
|
||||
```bash
|
||||
mvn -DskipTests package # builds target/db-migrations-expand-contract-1.0.0.jar
|
||||
mvn test # regenerates every unit-test transcript in docs/output/
|
||||
./scripts/run-all.sh # the live exhibit: real 4-deploy rolling sequence + load generator
|
||||
```
|
||||
|
||||
`run-all.sh` starts a standalone H2 TCP server, brings up two replicas on Stage 1,
|
||||
runs all four deploys of the expand-contract sequence as an actual rolling deploy —
|
||||
schema-only, code-only, schema-only, in the right order — and keeps a load generator
|
||||
sending real HTTP traffic through the entire thing. It regenerates:
|
||||
|
||||
- [`docs/output/11-live-deploy-sequence.txt`](docs/output/11-live-deploy-sequence.txt) — the deploy log, phase by phase
|
||||
- [`docs/output/12-load-generator-summary.txt`](docs/output/12-load-generator-summary.txt) — total/ok/error counts, by phase
|
||||
- [`docs/output/13-schema-diagnostics-timeline.txt`](docs/output/13-schema-diagnostics-timeline.txt) — `/diag/schema` after each deploy
|
||||
|
||||
## The four deploys (Spring profile: `app.stage`)
|
||||
|
||||
| Stage | What it does | Deploy kind |
|
||||
|---|---|---|
|
||||
| 1 | Reads and writes only `email` | *(baseline, before this article starts)* |
|
||||
| 2 | Writes both `email` and `email_address`; still reads `email` | Deploy 2 — code |
|
||||
| 3 | Writes both columns; reads `COALESCE(email_address, email)` | Deploy 3 — code |
|
||||
| 4 | Reads and writes only `email_address` | Deploy 4a — code |
|
||||
|
||||
Between stages, two schema-only migrations run with **zero application restarts**:
|
||||
Deploy 1 (EXPAND, `V2__add_email_address_column.sql`) before Stage 2 ships, and
|
||||
Deploy 4b (CONTRACT, `V3__drop_email_column.sql`) after every replica is confirmed on
|
||||
Stage 4. See [`docs/01-the-problem-and-the-plan.md`](docs/01-the-problem-and-the-plan.md).
|
||||
|
||||
## Endpoints
|
||||
|
||||
| Endpoint | Purpose |
|
||||
|---|---|
|
||||
| `POST /customers`, `GET /customers/{id}`, `PUT /customers/{id}/email` | The API contract that never changes across all four stages — see [`CustomerController`](src/main/java/com/ankurm/expandcontract/customer/CustomerController.java) |
|
||||
| `GET /diag/schema` | Live column list + row counts — the exhibit that shows the schema actually expanding and contracting. **Delete before shipping** (see [chapter 15](docs/15-production-checklist.md)) |
|
||||
| `POST /admin/drain` | Publishes `ReadinessState.REFUSING_TRAFFIC` before a graceful shutdown — the `preStop`-hook pattern. **Unauthenticated in this demo — lock down before shipping** |
|
||||
| `GET /actuator/health`, `GET /actuator/info` | Boot's own actuator endpoints |
|
||||
|
||||
## Documentation
|
||||
|
||||
1. [The problem and the plan](docs/01-the-problem-and-the-plan.md)
|
||||
2. [The expand migration](docs/02-the-expand-migration.md)
|
||||
3. [Why migrations run outside the app](docs/03-why-migrations-run-outside-the-app.md)
|
||||
4. [The dual write](docs/04-the-dual-write.md)
|
||||
5. [The read switch](docs/05-the-read-switch.md)
|
||||
6. [The NOT NULL trap](docs/06-the-not-null-trap.md)
|
||||
7. [The backfill window bug](docs/07-the-backfill-window-bug.md)
|
||||
8. [The rolling-window proof](docs/08-the-rolling-window-proof.md)
|
||||
9. [The contract migration](docs/09-the-contract-migration.md)
|
||||
10. [What happens if you drop too soon](docs/10-what-happens-if-you-drop-too-soon.md)
|
||||
11. [The load generator](docs/11-the-load-generator.md)
|
||||
12. [The AUTO_SERVER trap](docs/12-the-auto-server-trap.md)
|
||||
13. [Graceful shutdown vs. kill -9](docs/13-graceful-shutdown-vs-kill-9.md)
|
||||
14. [The DDL lock window](docs/14-the-ddl-lock-window.md) — the module's central finding
|
||||
15. [Production checklist](docs/15-production-checklist.md)
|
||||
|
||||
## Captured output
|
||||
|
||||
Every number quoted in the post and in the chapters above comes from a committed
|
||||
transcript in [`docs/output/`](docs/output). Files `02`–`10` and `14` are written by a
|
||||
`Transcript` helper while the JUnit test that produced them asserts the same numbers —
|
||||
a transcript going stale fails the build. Files `11`–`13` come from the live
|
||||
`run-all.sh` sequence.
|
||||
|
||||
| File | Source |
|
||||
|---|---|
|
||||
| [`02-expand-backward-compatible.txt`](docs/output/02-expand-backward-compatible.txt) | [`ExpandMigrationBackwardCompatibleTest`](src/test/java/com/ankurm/expandcontract/ExpandMigrationBackwardCompatibleTest.java) |
|
||||
| [`04-dual-write-consistency.txt`](docs/output/04-dual-write-consistency.txt) | [`DualWriteConsistencyTest`](src/test/java/com/ankurm/expandcontract/DualWriteConsistencyTest.java) |
|
||||
| [`06-not-null-trap.txt`](docs/output/06-not-null-trap.txt) | [`NotNullConstraintTrapTest`](src/test/java/com/ankurm/expandcontract/NotNullConstraintTrapTest.java) |
|
||||
| [`07-backfill-window-bug.txt`](docs/output/07-backfill-window-bug.txt) | [`BackfillWindowBugTest`](src/test/java/com/ankurm/expandcontract/BackfillWindowBugTest.java) |
|
||||
| [`08-mixed-stage-rolling-window.txt`](docs/output/08-mixed-stage-rolling-window.txt) | [`MixedStageRollingWindowTest`](src/test/java/com/ankurm/expandcontract/MixedStageRollingWindowTest.java) |
|
||||
| [`09-contract-safety.txt`](docs/output/09-contract-safety.txt) | [`ContractSafetyTest`](src/test/java/com/ankurm/expandcontract/ContractSafetyTest.java) (happy path) |
|
||||
| [`10-drop-too-soon.txt`](docs/output/10-drop-too-soon.txt) | [`ContractSafetyTest`](src/test/java/com/ankurm/expandcontract/ContractSafetyTest.java) (dropped too soon) |
|
||||
| [`11-live-deploy-sequence.txt`](docs/output/11-live-deploy-sequence.txt) | [`scripts/run-all.sh`](scripts/run-all.sh) |
|
||||
| [`12-load-generator-summary.txt`](docs/output/12-load-generator-summary.txt) | [`LoadGenerator`](src/main/java/com/ankurm/expandcontract/loadgen/LoadGenerator.java), via `run-all.sh` |
|
||||
| [`13-schema-diagnostics-timeline.txt`](docs/output/13-schema-diagnostics-timeline.txt) | [`SchemaDiagnosticsController`](src/main/java/com/ankurm/expandcontract/diag/SchemaDiagnosticsController.java), via `run-all.sh` |
|
||||
| [`14-ddl-silent-data-loss.txt`](docs/output/14-ddl-silent-data-loss.txt) | [`DdlSilentDataLossTest`](src/test/java/com/ankurm/expandcontract/DdlSilentDataLossTest.java) |
|
||||
|
||||
## Findings worth the trip
|
||||
|
||||
- H2's `AUTO_SERVER=TRUE` "shared embedded database" mode makes the first connecting
|
||||
process the de facto server for every other connection — killing that one replica
|
||||
during a rolling restart broke the *other* replica's "embedded" database entirely.
|
||||
Fixed by running H2 as an independent standalone TCP server ([chapter 12](docs/12-the-auto-server-trap.md)).
|
||||
- Expanding a column without relaxing the retired column's `NOT NULL` constraint
|
||||
fails every Stage 4 `INSERT` from the first request onward — a genuine bug in this
|
||||
module's own first draft, reproduced and fixed in the same migration
|
||||
([chapter 6](docs/06-the-not-null-trap.md)).
|
||||
- A naive Stage 3 read of `email_address` alone returns `NULL` for any row a
|
||||
still-live Stage 1 replica wrote during Deploy 2's own rollout window — fixed with
|
||||
`COALESCE(email_address, email)` ([chapter 7](docs/07-the-backfill-window-bug.md)).
|
||||
- H2 implements `ALTER TABLE ADD COLUMN` and `DROP COLUMN` by rebuilding the table,
|
||||
which can silently discard a row committed by a concurrent `INSERT` while the
|
||||
rebuild is mid-scan — with no exception thrown to the inserting connection.
|
||||
Reproduced directly, and confirmed to be an H2-specific behavior rather than a
|
||||
property of the expand-contract technique itself, since PostgreSQL documents both
|
||||
operations as metadata-only ([chapter 14](docs/14-the-ddl-lock-window.md) — the
|
||||
module's central finding).
|
||||
- A real rolling restart, with graceful shutdown alone, still produces
|
||||
`ConnectException` bursts: the health-checked pool needs time to notice a draining
|
||||
instance before that instance is actually killed. A `/admin/drain` endpoint that
|
||||
publishes `ReadinessState.REFUSING_TRAFFIC` before `SIGTERM` closes that gap
|
||||
([chapter 13](docs/13-graceful-shutdown-vs-kill-9.md)).
|
||||
- The full live sequence — real HTTP traffic through a real 4-deploy rolling restart —
|
||||
finished at 99.98% success (30,905 of 30,911 requests), with every one of the six
|
||||
residual errors traced to a single, explained root cause rather than left as an
|
||||
unexplained miss ([chapter 11](docs/11-the-load-generator.md)).
|
||||
Reference in New Issue
Block a user