Add db-migrations-expand-contract: zero-downtime schema migrations proven with a real 4-deploy rolling run

Companion code for Zero-Downtime Database Migrations: Expand-Contract in Practice with Spring Boot: a full expand/migrate-writes/migrate-reads/contract sequence run as an actual rolling deploy across two live replicas, with a load generator sending continuous HTTP traffic through all four deploys (99.98% success, every residual error traced to a root cause rather than left unexplained). Findings include a real NOT NULL constraint trap in the expand migration, a backfill-window bug in the read switch, H2's AUTO_SERVER=TRUE single-point-of-failure behavior under a rolling restart, the drain-before-SIGTERM fix needed to close a health-check gap during graceful shutdown, and H2 silently discarding a concurrently committed INSERT during an ALTER TABLE ADD/DROP COLUMN rebuild - confirmed, by primary source, to be an H2-specific behavior rather than a property of the technique itself. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_019Fb7vW8vLyLKngBc4R3huA
2026-09-16 19:22:48 +00:00
parent 22c30d4a5b
commit e478eafda3
60 changed files with 3561 additions and 0 deletions
@@ -0,0 +1,14 @@
+==============================================================================
+The failure the retry cannot catch: a committed INSERT that ALTER TABLE loses silently
+==============================================================================
+captured: 2026-09-16T19:20:58.574465432Z
+
+attempts needed to reproduce the race: 2 of 20
+customer creates that returned a generated id with no error: 50
+customer creates that got the already-documented, already-fixed DDL-collision error: 1
+of the ids that came back with no error, missing from the table once V2 finished: 8
+example missing ids: [41, 42, 43, 44, 45]
+
+This is why the retry in CustomerService cannot be the whole fix: these inserts
+never threw anything to retry. The row was committed, then discarded when the
+ADD COLUMN rebuild swapped in a new table that had already been scanned.