Companion code for Zero-Downtime Database Migrations: Expand-Contract in Practice with Spring Boot: a full expand/migrate-writes/migrate-reads/contract sequence run as an actual rolling deploy across two live replicas, with a load generator sending continuous HTTP traffic through all four deploys (99.98% success, every residual error traced to a root cause rather than left unexplained). Findings include a real NOT NULL constraint trap in the expand migration, a backfill-window bug in the read switch, H2's AUTO_SERVER=TRUE single-point-of-failure behavior under a rolling restart, the drain-before-SIGTERM fix needed to close a health-check gap during graceful shutdown, and H2 silently discarding a concurrently committed INSERT during an ALTER TABLE ADD/DROP COLUMN rebuild - confirmed, by primary source, to be an H2-specific behavior rather than a property of the technique itself. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_019Fb7vW8vLyLKngBc4R3huA
2.3 KiB
5. The read switch
← 4. The dual write · Next: 6. The NOT NULL trap →
Deploy 3 changes exactly one thing from Deploy 2: where reads come from. In
CustomerService.findById:
String sql = switch (stage) {
case 1, 2 -> "SELECT id, name, email AS email FROM customers WHERE id = ?";
case 3 -> "SELECT id, name, COALESCE(email_address, email) AS email FROM customers WHERE id = ?";
case 4 -> "SELECT id, name, email_address AS email FROM customers WHERE id = ?";
default -> throw new IllegalStateException();
};
Stage 1 and Stage 2 read the same column: only writes changed between them. Stage 3
is the read switch, and it's a separate deploy from Stage 2 for a reason that isn't
obvious until you say it out loud: the write switch and the read switch cannot be
the same deploy, because a rolling deploy is not instantaneous. Deploy 2's own
rollout has a window — anywhere from seconds to minutes, depending on fleet size —
where some replicas are still Stage 1, writing only email. If Deploy 3's read
switch were bundled into that same deploy, a Stage 3 replica reading
email_address alone would return NULL for every row a still-live Stage 1
replica had just written. That's not a hypothetical: it's
chapter 7, reproduced and fixed.
The fix already visible above is COALESCE(email_address, email) rather than
email_address alone — Stage 3 falls back to the old column for exactly the rows
that predate full dual-write coverage. Stage 4 doesn't need the fallback anymore: by
the time Deploy 4 starts, two full deploys' worth of dual-writing (Stage 2 and Stage
3, run back to back) have guaranteed every row has both columns populated.