Add db-migrations-expand-contract: zero-downtime schema migrations proven with a real 4-deploy rolling run

Companion code for Zero-Downtime Database Migrations: Expand-Contract in Practice
with Spring Boot: a full expand/migrate-writes/migrate-reads/contract sequence run
as an actual rolling deploy across two live replicas, with a load generator sending
continuous HTTP traffic through all four deploys (99.98% success, every residual
error traced to a root cause rather than left unexplained). Findings include a real
NOT NULL constraint trap in the expand migration, a backfill-window bug in the read
switch, H2's AUTO_SERVER=TRUE single-point-of-failure behavior under a rolling
restart, the drain-before-SIGTERM fix needed to close a health-check gap during
graceful shutdown, and H2 silently discarding a concurrently committed INSERT during
an ALTER TABLE ADD/DROP COLUMN rebuild - confirmed, by primary source, to be an
H2-specific behavior rather than a property of the technique itself.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_019Fb7vW8vLyLKngBc4R3huA

db-migrations-expand-contract/docs/06-the-not-null-trap.md

@@ -0,0 +1,66 @@
+# 6. The NOT NULL trap
+
+[← 5. The read switch](05-the-read-switch.md) · [Next: 7. The backfill window bug →](07-the-backfill-window-bug.md)
+
+This module's own first draft of the expand migration shipped without one line, and
+the bug it produced is worth showing exactly as it happened, because "add a nullable
+column" reads like the entire expand step and it isn't.
+
+`customers.email` was declared `NOT NULL` back in
+[`V1__create_customer.sql`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/main/resources/db/migration/V1__create_customer.sql).
+Deploy 1 adds `email_address`, nullable — that part is fine. But Stage 4's `create()`
+never writes `email` at all:
+
+```java
+case 4 -> jdbc.sql("INSERT INTO customers(name, email_address) VALUES (?, ?)")
+        .param(name).param(email)
+        .update(keyHolder, "id");
+```
+
+If `email` is still mandatory when Stage 4 code starts running, that `INSERT` omits a
+`NOT NULL` column with no default. Every single create fails, in production, from the
+first request the first Stage 4 replica handles.
+
+[`NotNullConstraintTrapTest`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/test/java/com/ankurm/expandcontract/NotNullConstraintTrapTest.java)
+reproduces this against the naive migration (expand plus backfill, nothing else) and
+then shows the one-line fix working:
+
+```
+-- Stage 4 create() against the NAIVE migration (no DROP NOT NULL) --
+org.springframework.dao.DataIntegrityViolationException: PreparedStatementCallback; SQL [INSERT INTO customers(name, email_address) VALUES (?, ?)]; NULL not allowed for column "EMAIL"; SQL statement:
+INSERT INTO customers(name, email_address) VALUES (?, ?) [23502-240]
+root cause: org.h2.jdbc.JdbcSQLIntegrityConstraintViolationException: NULL not allowed for column "EMAIL"; SQL statement:
+INSERT INTO customers(name, email_address) VALUES (?, ?) [23502-240]
+
+-- Stage 4 create() against the SHIPPED V2 migration (DROP NOT NULL included) --
+Customer[id=1, name=On Time, [email protected]]
+```
+
+Full transcript:
+[`docs/output/06-not-null-trap.txt`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/docs/output/06-not-null-trap.txt).
+
+The fix is the third statement in
+[`V2__add_email_address_column.sql`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/main/resources/db/migration/V2__add_email_address_column.sql):
+
+```sql
+ALTER TABLE customers ALTER COLUMN email DROP NOT NULL;
+```
+
+It belongs in the *same* migration as the add — Deploy 1 — not a later one. Stage 1
+and Stage 2 code both still write `email` on every insert, so relaxing its
+constraint changes nothing observable for them. But by the time Stage 4 ships, the
+constraint has to already be gone, and Stage 4 doesn't run a migration of its own —
+Deploy 4a is a pure code deploy (see [chapter 9](09-the-contract-migration.md)).
+Retrofitting the `DROP NOT NULL` later means adding a second schema change in the
+middle of what was supposed to be a code-only step.
+
+## Going deeper
+
+- Every column being retired in an expand-contract migration is worth auditing for
+  `NOT NULL`, `UNIQUE`, and foreign-key constraints the *new* write path won't
+  satisfy — this module only had one such constraint, but a wider table can have
+  several.
+- H2's constraint-violation exception hierarchy:
+  [`JdbcSQLIntegrityConstraintViolationException`](https://www.h2database.com/javadoc/org/h2/api/ErrorCode.html) (nofollow).
+
+[← 5. The read switch](05-the-read-switch.md) · [Next: 7. The backfill window bug →](07-the-backfill-window-bug.md)