asmhatreandClaude Sonnet 5
e478eafda3
Companion code for Zero-Downtime Database Migrations: Expand-Contract in Practice with Spring Boot: a full expand/migrate-writes/migrate-reads/contract sequence run as an actual rolling deploy across two live replicas, with a load generator sending continuous HTTP traffic through all four deploys (99.98% success, every residual error traced to a root cause rather than left unexplained). Findings include a real NOT NULL constraint trap in the expand migration, a backfill-window bug in the read switch, H2's AUTO_SERVER=TRUE single-point-of-failure behavior under a rolling restart, the drain-before-SIGTERM fix needed to close a health-check gap during graceful shutdown, and H2 silently discarding a concurrently committed INSERT during an ALTER TABLE ADD/DROP COLUMN rebuild - confirmed, by primary source, to be an H2-specific behavior rather than a property of the technique itself. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_019Fb7vW8vLyLKngBc4R3huA
3.5 KiB
2. The expand migration
← 1. The problem and the plan · Next: 3. Why migrations run outside the app →
Deploy 1 is schema-only. No application code changes, no replica restarts. The
migration is
V2__add_email_address_column.sql:
ALTER TABLE customers ADD COLUMN email_address VARCHAR(320);
UPDATE customers SET email_address = email WHERE email_address IS NULL;
ALTER TABLE customers ALTER COLUMN email DROP NOT NULL;
Three statements, three separate jobs:
- Add the column, nullable. Nullable is what makes it additive: no existing
INSERTstatement mentionsemail_address, so none of them break. A column addedNOT NULLwith no default would fail immediately for any code still running the oldINSERT INTO customers(name, email) VALUES (?, ?). - Backfill it for every row that already exists. This is a one-time pass over whatever data predates Deploy 1.
- Relax the old column's constraint. This third line is the one that's easy to skip, and skipping it is a real, reproducible outage — see chapter 6.
ExpandMigrationBackwardCompatibleTest
checks the two things Deploy 1 promises: every pre-existing row gets backfilled, and
Stage 1's original INSERT — unmodified, unaware email_address exists — still
works after the migration runs:
-- schema before Deploy 1 --
[COLUMN_NAME=ID, ...][COLUMN_NAME=NAME, ...][COLUMN_NAME=EMAIL, ...][COLUMN_NAME=CREATED_AT, ...]
-- schema after Deploy 1 (email_address added) --
[COLUMN_NAME=ID, ...][COLUMN_NAME=NAME, ...][COLUMN_NAME=EMAIL, ...][COLUMN_NAME=CREATED_AT, ...][COLUMN_NAME=EMAIL_ADDRESS, ...]
-- Ada's row was backfilled by the migration itself --
[NAME=Ada Lovelace, [email protected], [email protected]]
-- Stage 1's original INSERT still works, unmodified, after the migration --
[NAME=Grace Hopper, [email protected], EMAIL_ADDRESS=NULL]
Full transcript:
docs/output/02-expand-backward-compatible.txt.
Grace Hopper's row is the important one: email_address is NULL for it, because
Stage 1 never wrote to it, and that's correct — Deploy 1 hasn't shipped any code
that would. That gap is exactly what Deploy 2 exists to close, and it reopens itself
on a smaller scale during Deploy 2's own rollout — see
chapter 7.
Going deeper
- The migration runs against the live database with no application restart — chapter 3 covers how and why that's a separate mechanism from the app's own deploy.
- H2's specific behavior for
ALTER TABLE ADD COLUMNunder concurrent traffic — not just "is it additive" but "is it safe to run while inserts are in flight" — is covered in chapter 14, and it's the most surprising finding in this whole module.
← 1. The problem and the plan · Next: 3. Why migrations run outside the app →