Add db-migrations-expand-contract: zero-downtime schema migrations proven with a real 4-deploy rolling run

Companion code for Zero-Downtime Database Migrations: Expand-Contract in Practice
with Spring Boot: a full expand/migrate-writes/migrate-reads/contract sequence run
as an actual rolling deploy across two live replicas, with a load generator sending
continuous HTTP traffic through all four deploys (99.98% success, every residual
error traced to a root cause rather than left unexplained). Findings include a real
NOT NULL constraint trap in the expand migration, a backfill-window bug in the read
switch, H2's AUTO_SERVER=TRUE single-point-of-failure behavior under a rolling
restart, the drain-before-SIGTERM fix needed to close a health-check gap during
graceful shutdown, and H2 silently discarding a concurrently committed INSERT during
an ALTER TABLE ADD/DROP COLUMN rebuild - confirmed, by primary source, to be an
H2-specific behavior rather than a property of the technique itself.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_019Fb7vW8vLyLKngBc4R3huA

db-migrations-expand-contract/docs/02-the-expand-migration.md

@@ -0,0 +1,65 @@
+# 2. The expand migration
+
+[← 1. The problem and the plan](01-the-problem-and-the-plan.md) · [Next: 3. Why migrations run outside the app →](03-why-migrations-run-outside-the-app.md)
+
+Deploy 1 is schema-only. No application code changes, no replica restarts. The
+migration is
+[`V2__add_email_address_column.sql`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/main/resources/db/migration/V2__add_email_address_column.sql):
+
+```sql
+ALTER TABLE customers ADD COLUMN email_address VARCHAR(320);
+UPDATE customers SET email_address = email WHERE email_address IS NULL;
+ALTER TABLE customers ALTER COLUMN email DROP NOT NULL;
+```
+
+Three statements, three separate jobs:
+
+- **Add the column, nullable.** Nullable is what makes it additive: no existing
+  `INSERT` statement mentions `email_address`, so none of them break. A column added
+  `NOT NULL` with no default would fail immediately for any code still running the old
+  `INSERT INTO customers(name, email) VALUES (?, ?)`.
+- **Backfill it** for every row that already exists. This is a one-time pass over
+  whatever data predates Deploy 1.
+- **Relax the old column's constraint.** This third line is the one that's easy to
+  skip, and skipping it is a real, reproducible outage — see
+  [chapter 6](06-the-not-null-trap.md).
+
+[`ExpandMigrationBackwardCompatibleTest`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/test/java/com/ankurm/expandcontract/ExpandMigrationBackwardCompatibleTest.java)
+checks the two things Deploy 1 promises: every pre-existing row gets backfilled, and
+Stage 1's original `INSERT` — unmodified, unaware `email_address` exists — still
+works after the migration runs:
+
+```
+-- schema before Deploy 1 --
+[COLUMN_NAME=ID, ...][COLUMN_NAME=NAME, ...][COLUMN_NAME=EMAIL, ...][COLUMN_NAME=CREATED_AT, ...]
+
+-- schema after Deploy 1 (email_address added) --
+[COLUMN_NAME=ID, ...][COLUMN_NAME=NAME, ...][COLUMN_NAME=EMAIL, ...][COLUMN_NAME=CREATED_AT, ...][COLUMN_NAME=EMAIL_ADDRESS, ...]
+
+-- Ada's row was backfilled by the migration itself --
+[NAME=Ada Lovelace, [email protected], [email protected]]
+
+-- Stage 1's original INSERT still works, unmodified, after the migration --
+[NAME=Grace Hopper, [email protected], EMAIL_ADDRESS=NULL]
+```
+
+Full transcript:
+[`docs/output/02-expand-backward-compatible.txt`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/docs/output/02-expand-backward-compatible.txt).
+
+Grace Hopper's row is the important one: `email_address` is `NULL` for it, because
+Stage 1 never wrote to it, and that's *correct* — Deploy 1 hasn't shipped any code
+that would. That gap is exactly what Deploy 2 exists to close, and it reopens itself
+on a smaller scale during Deploy 2's own rollout — see
+[chapter 7](07-the-backfill-window-bug.md).
+
+## Going deeper
+
+- The migration runs against the live database with no application restart —
+  [chapter 3](03-why-migrations-run-outside-the-app.md) covers how and why that's a
+  separate mechanism from the app's own deploy.
+- H2's specific behavior for `ALTER TABLE ADD COLUMN` under concurrent traffic —
+  not just "is it additive" but "is it safe to run while inserts are in flight" — is
+  covered in [chapter 14](14-the-ddl-lock-window.md), and it's the most surprising
+  finding in this whole module.
+
+[← 1. The problem and the plan](01-the-problem-and-the-plan.md) · [Next: 3. Why migrations run outside the app →](03-why-migrations-run-outside-the-app.md)