asmhatre/spring-boot-demo
1
0
Fork 0
Code Activity
Files
a875bea55ab960987a68aa0a20f12a9894b54ce2
spring-boot-demo/db-migrations-expand-contract/README.md
T
asmhatreandClaude Sonnet 5 a875bea55a db-migrations-expand-contract: add a dedicated reproduction for the DDL-collision exception 
While writing the article, chapter 14's first failure mode (a concurrent statement
seeing "Table CUSTOMERS not found" while DROP COLUMN runs) was described from the
live load-generator run but had no dedicated, committed reproduction of its own -
CustomerService's own retry would silently absorb it if triggered through the
service layer. DdlCollisionExceptionTest reproduces it directly at the raw JDBC
level, and the chapter and README now link to its captured transcript.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_019Fb7vW8vLyLKngBc4R3huA
2026-09-16 19:33:31 +00:00

128 lines
8.7 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# db-migrations-expand-contract
Companion code for **[Zero-Downtime Database Migrations: Expand-Contract in Practice
with Spring Boot](https://ankurm.com/zero-downtime-database-migrations-expand-contract-spring-boot/)**
— every claim in that post traces to a test in this module, a real 4-deploy live run,
or a transcript in [`docs/output/`](docs/output).
## Versions
| Component | Version |
|---|---|
| Spring Boot | 4.1.1 |
| Flyway (via `spring-boot-starter-flyway`) | 12.4.0 |
| Jackson (via `spring-boot-starter-jackson`) | `tools.jackson` 3.1.5 |
| Database | H2 2.4.240, standalone TCP server mode |
| JDK | 25 (LTS) |
## Quickstart
```bash
mvn -DskipTests package # builds target/db-migrations-expand-contract-1.0.0.jar
mvn test # regenerates every unit-test transcript in docs/output/
./scripts/run-all.sh # the live exhibit: real 4-deploy rolling sequence + load generator
```
`run-all.sh` starts a standalone H2 TCP server, brings up two replicas on Stage 1,
runs all four deploys of the expand-contract sequence as an actual rolling deploy —
schema-only, code-only, schema-only, in the right order — and keeps a load generator
sending real HTTP traffic through the entire thing. It regenerates:
- [`docs/output/11-live-deploy-sequence.txt`](docs/output/11-live-deploy-sequence.txt) — the deploy log, phase by phase
- [`docs/output/12-load-generator-summary.txt`](docs/output/12-load-generator-summary.txt) — total/ok/error counts, by phase
- [`docs/output/13-schema-diagnostics-timeline.txt`](docs/output/13-schema-diagnostics-timeline.txt) — `/diag/schema` after each deploy
## The four deploys (Spring profile: `app.stage`)
| Stage | What it does | Deploy kind |
|---|---|---|
| 1 | Reads and writes only `email` | *(baseline, before this article starts)* |
| 2 | Writes both `email` and `email_address`; still reads `email` | Deploy 2 — code |
| 3 | Writes both columns; reads `COALESCE(email_address, email)` | Deploy 3 — code |
| 4 | Reads and writes only `email_address` | Deploy 4a — code |
Between stages, two schema-only migrations run with **zero application restarts**:
Deploy 1 (EXPAND, `V2__add_email_address_column.sql`) before Stage 2 ships, and
Deploy 4b (CONTRACT, `V3__drop_email_column.sql`) after every replica is confirmed on
Stage 4. See [`docs/01-the-problem-and-the-plan.md`](docs/01-the-problem-and-the-plan.md).
## Endpoints
| Endpoint | Purpose |
|---|---|
| `POST /customers`, `GET /customers/{id}`, `PUT /customers/{id}/email` | The API contract that never changes across all four stages — see [`CustomerController`](src/main/java/com/ankurm/expandcontract/customer/CustomerController.java) |
| `GET /diag/schema` | Live column list + row counts — the exhibit that shows the schema actually expanding and contracting. **Delete before shipping** (see [chapter 15](docs/15-production-checklist.md)) |
| `POST /admin/drain` | Publishes `ReadinessState.REFUSING_TRAFFIC` before a graceful shutdown — the `preStop`-hook pattern. **Unauthenticated in this demo — lock down before shipping** |
| `GET /actuator/health`, `GET /actuator/info` | Boot's own actuator endpoints |
## Documentation
1. [The problem and the plan](docs/01-the-problem-and-the-plan.md)
2. [The expand migration](docs/02-the-expand-migration.md)
3. [Why migrations run outside the app](docs/03-why-migrations-run-outside-the-app.md)
4. [The dual write](docs/04-the-dual-write.md)
5. [The read switch](docs/05-the-read-switch.md)
6. [The NOT NULL trap](docs/06-the-not-null-trap.md)
7. [The backfill window bug](docs/07-the-backfill-window-bug.md)
8. [The rolling-window proof](docs/08-the-rolling-window-proof.md)
9. [The contract migration](docs/09-the-contract-migration.md)
10. [What happens if you drop too soon](docs/10-what-happens-if-you-drop-too-soon.md)
11. [The load generator](docs/11-the-load-generator.md)
12. [The AUTO_SERVER trap](docs/12-the-auto-server-trap.md)
13. [Graceful shutdown vs. kill -9](docs/13-graceful-shutdown-vs-kill-9.md)
14. [The DDL lock window](docs/14-the-ddl-lock-window.md) — the module's central finding
15. [Production checklist](docs/15-production-checklist.md)
## Captured output
Every number quoted in the post and in the chapters above comes from a committed
transcript in [`docs/output/`](docs/output). Files `02`–`10` and `14` are written by a
`Transcript` helper while the JUnit test that produced them asserts the same numbers —
a transcript going stale fails the build. Files `11`–`13` come from the live
`run-all.sh` sequence.
| File | Source |
|---|---|
| [`02-expand-backward-compatible.txt`](docs/output/02-expand-backward-compatible.txt) | [`ExpandMigrationBackwardCompatibleTest`](src/test/java/com/ankurm/expandcontract/ExpandMigrationBackwardCompatibleTest.java) |
| [`04-dual-write-consistency.txt`](docs/output/04-dual-write-consistency.txt) | [`DualWriteConsistencyTest`](src/test/java/com/ankurm/expandcontract/DualWriteConsistencyTest.java) |
| [`06-not-null-trap.txt`](docs/output/06-not-null-trap.txt) | [`NotNullConstraintTrapTest`](src/test/java/com/ankurm/expandcontract/NotNullConstraintTrapTest.java) |
| [`07-backfill-window-bug.txt`](docs/output/07-backfill-window-bug.txt) | [`BackfillWindowBugTest`](src/test/java/com/ankurm/expandcontract/BackfillWindowBugTest.java) |
| [`08-mixed-stage-rolling-window.txt`](docs/output/08-mixed-stage-rolling-window.txt) | [`MixedStageRollingWindowTest`](src/test/java/com/ankurm/expandcontract/MixedStageRollingWindowTest.java) |
| [`09-contract-safety.txt`](docs/output/09-contract-safety.txt) | [`ContractSafetyTest`](src/test/java/com/ankurm/expandcontract/ContractSafetyTest.java) (happy path) |
| [`10-drop-too-soon.txt`](docs/output/10-drop-too-soon.txt) | [`ContractSafetyTest`](src/test/java/com/ankurm/expandcontract/ContractSafetyTest.java) (dropped too soon) |
| [`11-live-deploy-sequence.txt`](docs/output/11-live-deploy-sequence.txt) | [`scripts/run-all.sh`](scripts/run-all.sh) |
| [`12-load-generator-summary.txt`](docs/output/12-load-generator-summary.txt) | [`LoadGenerator`](src/main/java/com/ankurm/expandcontract/loadgen/LoadGenerator.java), via `run-all.sh` |
| [`13-schema-diagnostics-timeline.txt`](docs/output/13-schema-diagnostics-timeline.txt) | [`SchemaDiagnosticsController`](src/main/java/com/ankurm/expandcontract/diag/SchemaDiagnosticsController.java), via `run-all.sh` |
| [`14-ddl-collision-exception.txt`](docs/output/14-ddl-collision-exception.txt) | [`DdlCollisionExceptionTest`](src/test/java/com/ankurm/expandcontract/DdlCollisionExceptionTest.java) |
| [`14-ddl-silent-data-loss.txt`](docs/output/14-ddl-silent-data-loss.txt) | [`DdlSilentDataLossTest`](src/test/java/com/ankurm/expandcontract/DdlSilentDataLossTest.java) |
## Findings worth the trip
- H2's `AUTO_SERVER=TRUE` "shared embedded database" mode makes the first connecting
process the de facto server for every other connection — killing that one replica
during a rolling restart broke the *other* replica's "embedded" database entirely.
Fixed by running H2 as an independent standalone TCP server ([chapter 12](docs/12-the-auto-server-trap.md)).
- Expanding a column without relaxing the retired column's `NOT NULL` constraint
fails every Stage 4 `INSERT` from the first request onward — a genuine bug in this
module's own first draft, reproduced and fixed in the same migration
([chapter 6](docs/06-the-not-null-trap.md)).
- A naive Stage 3 read of `email_address` alone returns `NULL` for any row a
still-live Stage 1 replica wrote during Deploy 2's own rollout window — fixed with
`COALESCE(email_address, email)` ([chapter 7](docs/07-the-backfill-window-bug.md)).
- H2 implements `ALTER TABLE ADD COLUMN` and `DROP COLUMN` by rebuilding the table,
which can silently discard a row committed by a concurrent `INSERT` while the
rebuild is mid-scan — with no exception thrown to the inserting connection.
Reproduced directly, and confirmed to be an H2-specific behavior rather than a
property of the expand-contract technique itself, since PostgreSQL documents both
operations as metadata-only ([chapter 14](docs/14-the-ddl-lock-window.md) — the
module's central finding).
- A real rolling restart, with graceful shutdown alone, still produces
`ConnectException` bursts: the health-checked pool needs time to notice a draining
instance before that instance is actually killed. A `/admin/drain` endpoint that
publishes `ReadinessState.REFUSING_TRAFFIC` before `SIGTERM` closes that gap
([chapter 13](docs/13-graceful-shutdown-vs-kill-9.md)).
- The full live sequence — real HTTP traffic through a real 4-deploy rolling restart —
finished at 99.98% success (30,905 of 30,911 requests), with every one of the six
residual errors traced to a single, explained root cause rather than left as an
unexplained miss ([chapter 11](docs/11-the-load-generator.md)).
View Git Blame Copy Permalink