asmhatre/spring-boot-demo
1
0
Fork 0
Code Activity
Files
e478eafda3aa6b2c0e309967478dae2b1a0c8f4f
spring-boot-demo/db-migrations-expand-contract/README.md
T
asmhatreandClaude Sonnet 5 e478eafda3 Add db-migrations-expand-contract: zero-downtime schema migrations proven with a real 4-deploy rolling run 
Companion code for Zero-Downtime Database Migrations: Expand-Contract in Practice
with Spring Boot: a full expand/migrate-writes/migrate-reads/contract sequence run
as an actual rolling deploy across two live replicas, with a load generator sending
continuous HTTP traffic through all four deploys (99.98% success, every residual
error traced to a root cause rather than left unexplained). Findings include a real
NOT NULL constraint trap in the expand migration, a backfill-window bug in the read
switch, H2's AUTO_SERVER=TRUE single-point-of-failure behavior under a rolling
restart, the drain-before-SIGTERM fix needed to close a health-check gap during
graceful shutdown, and H2 silently discarding a concurrently committed INSERT during
an ALTER TABLE ADD/DROP COLUMN rebuild - confirmed, by primary source, to be an
H2-specific behavior rather than a property of the technique itself.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_019Fb7vW8vLyLKngBc4R3huA
2026-09-16 19:22:48 +00:00

127 lines
8.5 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# db-migrations-expand-contract
Companion code for **[Zero-Downtime Database Migrations: Expand-Contract in Practice
with Spring Boot](https://ankurm.com/zero-downtime-database-migrations-expand-contract-spring-boot/)**
— every claim in that post traces to a test in this module, a real 4-deploy live run,
or a transcript in [`docs/output/`](docs/output).
## Versions
| Component | Version |
|---|---|
| Spring Boot | 4.1.1 |
| Flyway (via `spring-boot-starter-flyway`) | 12.4.0 |
| Jackson (via `spring-boot-starter-jackson`) | `tools.jackson` 3.1.5 |
| Database | H2 2.4.240, standalone TCP server mode |
| JDK | 25 (LTS) |
## Quickstart
```bash
mvn -DskipTests package # builds target/db-migrations-expand-contract-1.0.0.jar
mvn test # regenerates every unit-test transcript in docs/output/
./scripts/run-all.sh # the live exhibit: real 4-deploy rolling sequence + load generator
```
`run-all.sh` starts a standalone H2 TCP server, brings up two replicas on Stage 1,
runs all four deploys of the expand-contract sequence as an actual rolling deploy —
schema-only, code-only, schema-only, in the right order — and keeps a load generator
sending real HTTP traffic through the entire thing. It regenerates:
- [`docs/output/11-live-deploy-sequence.txt`](docs/output/11-live-deploy-sequence.txt) — the deploy log, phase by phase
- [`docs/output/12-load-generator-summary.txt`](docs/output/12-load-generator-summary.txt) — total/ok/error counts, by phase
- [`docs/output/13-schema-diagnostics-timeline.txt`](docs/output/13-schema-diagnostics-timeline.txt) — `/diag/schema` after each deploy
## The four deploys (Spring profile: `app.stage`)
| Stage | What it does | Deploy kind |
|---|---|---|
| 1 | Reads and writes only `email` | *(baseline, before this article starts)* |
| 2 | Writes both `email` and `email_address`; still reads `email` | Deploy 2 — code |
| 3 | Writes both columns; reads `COALESCE(email_address, email)` | Deploy 3 — code |
| 4 | Reads and writes only `email_address` | Deploy 4a — code |
Between stages, two schema-only migrations run with **zero application restarts**:
Deploy 1 (EXPAND, `V2__add_email_address_column.sql`) before Stage 2 ships, and
Deploy 4b (CONTRACT, `V3__drop_email_column.sql`) after every replica is confirmed on
Stage 4. See [`docs/01-the-problem-and-the-plan.md`](docs/01-the-problem-and-the-plan.md).
## Endpoints
| Endpoint | Purpose |
|---|---|
| `POST /customers`, `GET /customers/{id}`, `PUT /customers/{id}/email` | The API contract that never changes across all four stages — see [`CustomerController`](src/main/java/com/ankurm/expandcontract/customer/CustomerController.java) |
| `GET /diag/schema` | Live column list + row counts — the exhibit that shows the schema actually expanding and contracting. **Delete before shipping** (see [chapter 15](docs/15-production-checklist.md)) |
| `POST /admin/drain` | Publishes `ReadinessState.REFUSING_TRAFFIC` before a graceful shutdown — the `preStop`-hook pattern. **Unauthenticated in this demo — lock down before shipping** |
| `GET /actuator/health`, `GET /actuator/info` | Boot's own actuator endpoints |
## Documentation
1. [The problem and the plan](docs/01-the-problem-and-the-plan.md)
2. [The expand migration](docs/02-the-expand-migration.md)
3. [Why migrations run outside the app](docs/03-why-migrations-run-outside-the-app.md)
4. [The dual write](docs/04-the-dual-write.md)
5. [The read switch](docs/05-the-read-switch.md)
6. [The NOT NULL trap](docs/06-the-not-null-trap.md)
7. [The backfill window bug](docs/07-the-backfill-window-bug.md)
8. [The rolling-window proof](docs/08-the-rolling-window-proof.md)
9. [The contract migration](docs/09-the-contract-migration.md)
10. [What happens if you drop too soon](docs/10-what-happens-if-you-drop-too-soon.md)
11. [The load generator](docs/11-the-load-generator.md)
12. [The AUTO_SERVER trap](docs/12-the-auto-server-trap.md)
13. [Graceful shutdown vs. kill -9](docs/13-graceful-shutdown-vs-kill-9.md)
14. [The DDL lock window](docs/14-the-ddl-lock-window.md) — the module's central finding
15. [Production checklist](docs/15-production-checklist.md)
## Captured output
Every number quoted in the post and in the chapters above comes from a committed
transcript in [`docs/output/`](docs/output). Files `02`–`10` and `14` are written by a
`Transcript` helper while the JUnit test that produced them asserts the same numbers —
a transcript going stale fails the build. Files `11`–`13` come from the live
`run-all.sh` sequence.
| File | Source |
|---|---|
| [`02-expand-backward-compatible.txt`](docs/output/02-expand-backward-compatible.txt) | [`ExpandMigrationBackwardCompatibleTest`](src/test/java/com/ankurm/expandcontract/ExpandMigrationBackwardCompatibleTest.java) |
| [`04-dual-write-consistency.txt`](docs/output/04-dual-write-consistency.txt) | [`DualWriteConsistencyTest`](src/test/java/com/ankurm/expandcontract/DualWriteConsistencyTest.java) |
| [`06-not-null-trap.txt`](docs/output/06-not-null-trap.txt) | [`NotNullConstraintTrapTest`](src/test/java/com/ankurm/expandcontract/NotNullConstraintTrapTest.java) |
| [`07-backfill-window-bug.txt`](docs/output/07-backfill-window-bug.txt) | [`BackfillWindowBugTest`](src/test/java/com/ankurm/expandcontract/BackfillWindowBugTest.java) |
| [`08-mixed-stage-rolling-window.txt`](docs/output/08-mixed-stage-rolling-window.txt) | [`MixedStageRollingWindowTest`](src/test/java/com/ankurm/expandcontract/MixedStageRollingWindowTest.java) |
| [`09-contract-safety.txt`](docs/output/09-contract-safety.txt) | [`ContractSafetyTest`](src/test/java/com/ankurm/expandcontract/ContractSafetyTest.java) (happy path) |
| [`10-drop-too-soon.txt`](docs/output/10-drop-too-soon.txt) | [`ContractSafetyTest`](src/test/java/com/ankurm/expandcontract/ContractSafetyTest.java) (dropped too soon) |
| [`11-live-deploy-sequence.txt`](docs/output/11-live-deploy-sequence.txt) | [`scripts/run-all.sh`](scripts/run-all.sh) |
| [`12-load-generator-summary.txt`](docs/output/12-load-generator-summary.txt) | [`LoadGenerator`](src/main/java/com/ankurm/expandcontract/loadgen/LoadGenerator.java), via `run-all.sh` |
| [`13-schema-diagnostics-timeline.txt`](docs/output/13-schema-diagnostics-timeline.txt) | [`SchemaDiagnosticsController`](src/main/java/com/ankurm/expandcontract/diag/SchemaDiagnosticsController.java), via `run-all.sh` |
| [`14-ddl-silent-data-loss.txt`](docs/output/14-ddl-silent-data-loss.txt) | [`DdlSilentDataLossTest`](src/test/java/com/ankurm/expandcontract/DdlSilentDataLossTest.java) |
## Findings worth the trip
- H2's `AUTO_SERVER=TRUE` "shared embedded database" mode makes the first connecting
process the de facto server for every other connection — killing that one replica
during a rolling restart broke the *other* replica's "embedded" database entirely.
Fixed by running H2 as an independent standalone TCP server ([chapter 12](docs/12-the-auto-server-trap.md)).
- Expanding a column without relaxing the retired column's `NOT NULL` constraint
fails every Stage 4 `INSERT` from the first request onward — a genuine bug in this
module's own first draft, reproduced and fixed in the same migration
([chapter 6](docs/06-the-not-null-trap.md)).
- A naive Stage 3 read of `email_address` alone returns `NULL` for any row a
still-live Stage 1 replica wrote during Deploy 2's own rollout window — fixed with
`COALESCE(email_address, email)` ([chapter 7](docs/07-the-backfill-window-bug.md)).
- H2 implements `ALTER TABLE ADD COLUMN` and `DROP COLUMN` by rebuilding the table,
which can silently discard a row committed by a concurrent `INSERT` while the
rebuild is mid-scan — with no exception thrown to the inserting connection.
Reproduced directly, and confirmed to be an H2-specific behavior rather than a
property of the expand-contract technique itself, since PostgreSQL documents both
operations as metadata-only ([chapter 14](docs/14-the-ddl-lock-window.md) — the
module's central finding).
- A real rolling restart, with graceful shutdown alone, still produces
`ConnectException` bursts: the health-checked pool needs time to notice a draining
instance before that instance is actually killed. A `/admin/drain` endpoint that
publishes `ReadinessState.REFUSING_TRAFFIC` before `SIGTERM` closes that gap
([chapter 13](docs/13-graceful-shutdown-vs-kill-9.md)).
- The full live sequence — real HTTP traffic through a real 4-deploy rolling restart —
finished at 99.98% success (30,905 of 30,911 requests), with every one of the six
residual errors traced to a single, explained root cause rather than left as an
unexplained miss ([chapter 11](docs/11-the-load-generator.md)).
View Git Blame Copy Permalink