spring-boot-demo/db-migrations-expand-contract/README.md

db-migrations-expand-contract

Companion code for Zero-Downtime Database Migrations: Expand-Contract in Practice with Spring Boot — every claim in that post traces to a test in this module, a real 4-deploy live run, or a transcript in docs/output/.

Versions

Component	Version
Spring Boot	4.1.1
Flyway (via spring-boot-starter-flyway)	12.4.0
Jackson (via spring-boot-starter-jackson)	tools.jackson 3.1.5
Database	H2 2.4.240, standalone TCP server mode
JDK	25 (LTS)

Quickstart

mvn -DskipTests package        # builds target/db-migrations-expand-contract-1.0.0.jar
mvn test                       # regenerates every unit-test transcript in docs/output/
./scripts/run-all.sh           # the live exhibit: real 4-deploy rolling sequence + load generator

run-all.sh starts a standalone H2 TCP server, brings up two replicas on Stage 1, runs all four deploys of the expand-contract sequence as an actual rolling deploy — schema-only, code-only, schema-only, in the right order — and keeps a load generator sending real HTTP traffic through the entire thing. It regenerates:

• docs/output/11-live-deploy-sequence.txt — the deploy log, phase by phase
• docs/output/12-load-generator-summary.txt — total/ok/error counts, by phase
• docs/output/13-schema-diagnostics-timeline.txt — /diag/schema after each deploy

The four deploys (Spring profile: app.stage)

Stage	What it does	Deploy kind
1	Reads and writes only email	(baseline, before this article starts)
2	Writes both email and email_address; still reads email	Deploy 2 — code
3	Writes both columns; reads COALESCE(email_address, email)	Deploy 3 — code
4	Reads and writes only email_address	Deploy 4a — code

Between stages, two schema-only migrations run with zero application restarts: Deploy 1 (EXPAND, V2__add_email_address_column.sql) before Stage 2 ships, and Deploy 4b (CONTRACT, V3__drop_email_column.sql) after every replica is confirmed on Stage 4. See docs/01-the-problem-and-the-plan.md.

Endpoints

Endpoint	Purpose
POST /customers, GET /customers/{id}, PUT /customers/{id}/email	The API contract that never changes across all four stages — see CustomerController
GET /diag/schema	Live column list + row counts — the exhibit that shows the schema actually expanding and contracting. Delete before shipping (see chapter 15)
POST /admin/drain	Publishes ReadinessState.REFUSING_TRAFFIC before a graceful shutdown — the preStop-hook pattern. Unauthenticated in this demo — lock down before shipping
GET /actuator/health, GET /actuator/info	Boot's own actuator endpoints

Documentation

1. The problem and the plan
2. The expand migration
3. Why migrations run outside the app
4. The dual write
5. The read switch
6. The NOT NULL trap
7. The backfill window bug
8. The rolling-window proof
9. The contract migration
10. What happens if you drop too soon
11. The load generator
12. The AUTO_SERVER trap
13. Graceful shutdown vs. kill -9
14. The DDL lock window — the module's central finding
15. Production checklist

Captured output

Every number quoted in the post and in the chapters above comes from a committed transcript in docs/output/. Files 02–10 and 14 are written by a Transcript helper while the JUnit test that produced them asserts the same numbers — a transcript going stale fails the build. Files 11–13 come from the live run-all.sh sequence.

File	Source
02-expand-backward-compatible.txt	ExpandMigrationBackwardCompatibleTest
04-dual-write-consistency.txt	DualWriteConsistencyTest
06-not-null-trap.txt	NotNullConstraintTrapTest
07-backfill-window-bug.txt	BackfillWindowBugTest
08-mixed-stage-rolling-window.txt	MixedStageRollingWindowTest
09-contract-safety.txt	ContractSafetyTest (happy path)
10-drop-too-soon.txt	ContractSafetyTest (dropped too soon)
11-live-deploy-sequence.txt	scripts/run-all.sh
12-load-generator-summary.txt	LoadGenerator, via run-all.sh
13-schema-diagnostics-timeline.txt	SchemaDiagnosticsController, via run-all.sh
14-ddl-collision-exception.txt	DdlCollisionExceptionTest
14-ddl-silent-data-loss.txt	DdlSilentDataLossTest

Findings worth the trip

• H2's AUTO_SERVER=TRUE "shared embedded database" mode makes the first connecting process the de facto server for every other connection — killing that one replica during a rolling restart broke the other replica's "embedded" database entirely. Fixed by running H2 as an independent standalone TCP server (chapter 12).
• Expanding a column without relaxing the retired column's NOT NULL constraint fails every Stage 4 INSERT from the first request onward — a genuine bug in this module's own first draft, reproduced and fixed in the same migration (chapter 6).
• A naive Stage 3 read of email_address alone returns NULL for any row a still-live Stage 1 replica wrote during Deploy 2's own rollout window — fixed with COALESCE(email_address, email) (chapter 7).
• H2 implements ALTER TABLE ADD COLUMN and DROP COLUMN by rebuilding the table, which can silently discard a row committed by a concurrent INSERT while the rebuild is mid-scan — with no exception thrown to the inserting connection. Reproduced directly, and confirmed to be an H2-specific behavior rather than a property of the expand-contract technique itself, since PostgreSQL documents both operations as metadata-only (chapter 14 — the module's central finding).
• A real rolling restart, with graceful shutdown alone, still produces ConnectException bursts: the health-checked pool needs time to notice a draining instance before that instance is actually killed. A /admin/drain endpoint that publishes ReadinessState.REFUSING_TRAFFIC before SIGTERM closes that gap (chapter 13).
• The full live sequence — real HTTP traffic through a real 4-deploy rolling restart — finished at 99.98% success (30,905 of 30,911 requests), with every one of the six residual errors traced to a single, explained root cause rather than left as an unexplained miss (chapter 11).