asmhatreandClaude Sonnet 5
e478eafda3
Companion code for Zero-Downtime Database Migrations: Expand-Contract in Practice with Spring Boot: a full expand/migrate-writes/migrate-reads/contract sequence run as an actual rolling deploy across two live replicas, with a load generator sending continuous HTTP traffic through all four deploys (99.98% success, every residual error traced to a root cause rather than left unexplained). Findings include a real NOT NULL constraint trap in the expand migration, a backfill-window bug in the read switch, H2's AUTO_SERVER=TRUE single-point-of-failure behavior under a rolling restart, the drain-before-SIGTERM fix needed to close a health-check gap during graceful shutdown, and H2 silently discarding a concurrently committed INSERT during an ALTER TABLE ADD/DROP COLUMN rebuild - confirmed, by primary source, to be an H2-specific behavior rather than a property of the technique itself. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_019Fb7vW8vLyLKngBc4R3huA
3.6 KiB
6. The NOT NULL trap
← 5. The read switch · Next: 7. The backfill window bug →
This module's own first draft of the expand migration shipped without one line, and the bug it produced is worth showing exactly as it happened, because "add a nullable column" reads like the entire expand step and it isn't.
customers.email was declared NOT NULL back in
V1__create_customer.sql.
Deploy 1 adds email_address, nullable — that part is fine. But Stage 4's create()
never writes email at all:
case 4 -> jdbc.sql("INSERT INTO customers(name, email_address) VALUES (?, ?)")
.param(name).param(email)
.update(keyHolder, "id");
If email is still mandatory when Stage 4 code starts running, that INSERT omits a
NOT NULL column with no default. Every single create fails, in production, from the
first request the first Stage 4 replica handles.
NotNullConstraintTrapTest
reproduces this against the naive migration (expand plus backfill, nothing else) and
then shows the one-line fix working:
-- Stage 4 create() against the NAIVE migration (no DROP NOT NULL) --
org.springframework.dao.DataIntegrityViolationException: PreparedStatementCallback; SQL [INSERT INTO customers(name, email_address) VALUES (?, ?)]; NULL not allowed for column "EMAIL"; SQL statement:
INSERT INTO customers(name, email_address) VALUES (?, ?) [23502-240]
root cause: org.h2.jdbc.JdbcSQLIntegrityConstraintViolationException: NULL not allowed for column "EMAIL"; SQL statement:
INSERT INTO customers(name, email_address) VALUES (?, ?) [23502-240]
-- Stage 4 create() against the SHIPPED V2 migration (DROP NOT NULL included) --
Customer[id=1, name=On Time, [email protected]]
Full transcript:
docs/output/06-not-null-trap.txt.
The fix is the third statement in
V2__add_email_address_column.sql:
ALTER TABLE customers ALTER COLUMN email DROP NOT NULL;
It belongs in the same migration as the add — Deploy 1 — not a later one. Stage 1
and Stage 2 code both still write email on every insert, so relaxing its
constraint changes nothing observable for them. But by the time Stage 4 ships, the
constraint has to already be gone, and Stage 4 doesn't run a migration of its own —
Deploy 4a is a pure code deploy (see chapter 9).
Retrofitting the DROP NOT NULL later means adding a second schema change in the
middle of what was supposed to be a code-only step.
Going deeper
- Every column being retired in an expand-contract migration is worth auditing for
NOT NULL,UNIQUE, and foreign-key constraints the new write path won't satisfy — this module only had one such constraint, but a wider table can have several. - H2's constraint-violation exception hierarchy:
JdbcSQLIntegrityConstraintViolationException(nofollow).