asmhatreandClaude Sonnet 5
e478eafda3
Companion code for Zero-Downtime Database Migrations: Expand-Contract in Practice with Spring Boot: a full expand/migrate-writes/migrate-reads/contract sequence run as an actual rolling deploy across two live replicas, with a load generator sending continuous HTTP traffic through all four deploys (99.98% success, every residual error traced to a root cause rather than left unexplained). Findings include a real NOT NULL constraint trap in the expand migration, a backfill-window bug in the read switch, H2's AUTO_SERVER=TRUE single-point-of-failure behavior under a rolling restart, the drain-before-SIGTERM fix needed to close a health-check gap during graceful shutdown, and H2 silently discarding a concurrently committed INSERT during an ALTER TABLE ADD/DROP COLUMN rebuild - confirmed, by primary source, to be an H2-specific behavior rather than a property of the technique itself. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_019Fb7vW8vLyLKngBc4R3huA
3.6 KiB
3.6 KiB
15. Production checklist
Everything in this module is a demonstration, deliberately built so its failure modes are reachable and its output is captured. Taking the technique — not this exact code — to a real service means addressing what this module intentionally left uncovered:
- Gate the contract migration on confirmed fleet state, not a timer. Chapter 10 shows what a lingering old-stage instance sees the moment the drop runs — a real deploy pipeline should check every instance's reported version/health before running Deploy 4b, not just wait "long enough".
- Know your database's
ALTER TABLEsemantics before you run this against a real table. Chapter 14 is specific to H2's rebuild-based implementation ofADD COLUMN/DROP COLUMN— verify what your actual production database does under concurrent writes for the specific change you're making, and test it, the same way that chapter's test does, against your own engine and table size before trusting either "it's always safe" or "it always errors loudly". - Remove or lock down the diagnostic endpoints.
SchemaDiagnosticsController's/diag/schemaandDrainController's/admin/drainhave no authentication and are wired for a demo where anyone can curl them freely./admin/drainin particular can take a real instance out of a real load balancer's rotation with a single unauthenticatedPOST— restrict it to the same internal network your orchestrator'spreStophook runs from, or replace it with your platform's native lifecycle hook. - Run the real load test against your real database and your real fleet size,
not just this module's two-replica sandbox. The
AUTO_SERVERtrap (chapter 12) and the DDL lock window (chapter 14) were both found by running actual concurrent traffic through actual restarts — reading about the technique would not have surfaced either one. - Budget for at least two separate code deploys plus two schema changes, not one
deploy. Teams estimating "rename a column" as a single-PR, single-deploy task are
the ones most likely to reach for the
RENAME COLUMNshortcut this article opened with — see chapter 1. - Decide what "zero errors" means for your own load test before you run it. This article's own final number is 99.98%, not literally zero, and the six-in-30,911 residual is explained down to its root mechanism rather than hand-waved — see the honest accounting in chapter 14. A number you can fully explain is more useful, and more trustworthy, than one you cannot account for at all.
The whole sequence, one command
./scripts/run-all.sh
regenerates every transcript this article and these chapters quote, end to end: schema-only Deploy 1, rolling Deploy 2, rolling Deploy 3, rolling Deploy 4a, schema-only Deploy 4b, with the load generator running continuously throughout. See the module README for the full script index and version table.