asmhatreandClaude Sonnet 5
e478eafda3
Companion code for Zero-Downtime Database Migrations: Expand-Contract in Practice with Spring Boot: a full expand/migrate-writes/migrate-reads/contract sequence run as an actual rolling deploy across two live replicas, with a load generator sending continuous HTTP traffic through all four deploys (99.98% success, every residual error traced to a root cause rather than left unexplained). Findings include a real NOT NULL constraint trap in the expand migration, a backfill-window bug in the read switch, H2's AUTO_SERVER=TRUE single-point-of-failure behavior under a rolling restart, the drain-before-SIGTERM fix needed to close a health-check gap during graceful shutdown, and H2 silently discarding a concurrently committed INSERT during an ALTER TABLE ADD/DROP COLUMN rebuild - confirmed, by primary source, to be an H2-specific behavior rather than a property of the technique itself. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_019Fb7vW8vLyLKngBc4R3huA
66 lines
3.5 KiB
Markdown
66 lines
3.5 KiB
Markdown
# 2. The expand migration
|
|
|
|
[← 1. The problem and the plan](01-the-problem-and-the-plan.md) · [Next: 3. Why migrations run outside the app →](03-why-migrations-run-outside-the-app.md)
|
|
|
|
Deploy 1 is schema-only. No application code changes, no replica restarts. The
|
|
migration is
|
|
[`V2__add_email_address_column.sql`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/main/resources/db/migration/V2__add_email_address_column.sql):
|
|
|
|
```sql
|
|
ALTER TABLE customers ADD COLUMN email_address VARCHAR(320);
|
|
UPDATE customers SET email_address = email WHERE email_address IS NULL;
|
|
ALTER TABLE customers ALTER COLUMN email DROP NOT NULL;
|
|
```
|
|
|
|
Three statements, three separate jobs:
|
|
|
|
- **Add the column, nullable.** Nullable is what makes it additive: no existing
|
|
`INSERT` statement mentions `email_address`, so none of them break. A column added
|
|
`NOT NULL` with no default would fail immediately for any code still running the old
|
|
`INSERT INTO customers(name, email) VALUES (?, ?)`.
|
|
- **Backfill it** for every row that already exists. This is a one-time pass over
|
|
whatever data predates Deploy 1.
|
|
- **Relax the old column's constraint.** This third line is the one that's easy to
|
|
skip, and skipping it is a real, reproducible outage — see
|
|
[chapter 6](06-the-not-null-trap.md).
|
|
|
|
[`ExpandMigrationBackwardCompatibleTest`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/test/java/com/ankurm/expandcontract/ExpandMigrationBackwardCompatibleTest.java)
|
|
checks the two things Deploy 1 promises: every pre-existing row gets backfilled, and
|
|
Stage 1's original `INSERT` — unmodified, unaware `email_address` exists — still
|
|
works after the migration runs:
|
|
|
|
```
|
|
-- schema before Deploy 1 --
|
|
[COLUMN_NAME=ID, ...][COLUMN_NAME=NAME, ...][COLUMN_NAME=EMAIL, ...][COLUMN_NAME=CREATED_AT, ...]
|
|
|
|
-- schema after Deploy 1 (email_address added) --
|
|
[COLUMN_NAME=ID, ...][COLUMN_NAME=NAME, ...][COLUMN_NAME=EMAIL, ...][COLUMN_NAME=CREATED_AT, ...][COLUMN_NAME=EMAIL_ADDRESS, ...]
|
|
|
|
-- Ada's row was backfilled by the migration itself --
|
|
[NAME=Ada Lovelace, [email protected], [email protected]]
|
|
|
|
-- Stage 1's original INSERT still works, unmodified, after the migration --
|
|
[NAME=Grace Hopper, [email protected], EMAIL_ADDRESS=NULL]
|
|
```
|
|
|
|
Full transcript:
|
|
[`docs/output/02-expand-backward-compatible.txt`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/docs/output/02-expand-backward-compatible.txt).
|
|
|
|
Grace Hopper's row is the important one: `email_address` is `NULL` for it, because
|
|
Stage 1 never wrote to it, and that's *correct* — Deploy 1 hasn't shipped any code
|
|
that would. That gap is exactly what Deploy 2 exists to close, and it reopens itself
|
|
on a smaller scale during Deploy 2's own rollout — see
|
|
[chapter 7](07-the-backfill-window-bug.md).
|
|
|
|
## Going deeper
|
|
|
|
- The migration runs against the live database with no application restart —
|
|
[chapter 3](03-why-migrations-run-outside-the-app.md) covers how and why that's a
|
|
separate mechanism from the app's own deploy.
|
|
- H2's specific behavior for `ALTER TABLE ADD COLUMN` under concurrent traffic —
|
|
not just "is it additive" but "is it safe to run while inserts are in flight" — is
|
|
covered in [chapter 14](14-the-ddl-lock-window.md), and it's the most surprising
|
|
finding in this whole module.
|
|
|
|
[← 1. The problem and the plan](01-the-problem-and-the-plan.md) · [Next: 3. Why migrations run outside the app →](03-why-migrations-run-outside-the-app.md)
|