asmhatreandClaude Sonnet 5
e478eafda3
Companion code for Zero-Downtime Database Migrations: Expand-Contract in Practice with Spring Boot: a full expand/migrate-writes/migrate-reads/contract sequence run as an actual rolling deploy across two live replicas, with a load generator sending continuous HTTP traffic through all four deploys (99.98% success, every residual error traced to a root cause rather than left unexplained). Findings include a real NOT NULL constraint trap in the expand migration, a backfill-window bug in the read switch, H2's AUTO_SERVER=TRUE single-point-of-failure behavior under a rolling restart, the drain-before-SIGTERM fix needed to close a health-check gap during graceful shutdown, and H2 silently discarding a concurrently committed INSERT during an ALTER TABLE ADD/DROP COLUMN rebuild - confirmed, by primary source, to be an H2-specific behavior rather than a property of the technique itself. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_019Fb7vW8vLyLKngBc4R3huA
6.0 KiB
1. The problem and the plan
Next: 2. The expand migration →
The problem
customers has a column called email. You want it called email_address — maybe
because a second contact_email table is coming and the naming needs to be
consistent, maybe because "email" collided with a reserved word in a tool you just
adopted. The reason doesn't matter. What matters is that this table has rows in it,
right now, in production, and something is reading and writing that column while you
work.
The naive fix is one migration:
ALTER TABLE customers RENAME COLUMN email TO email_address;
That statement is correct and it is also a production outage. The instant it commits,
every currently-running copy of your application — the ones you have not redeployed
yet, because a rolling deploy takes minutes, not zero seconds — starts issuing SQL
against a column that no longer exists. INSERT INTO customers(name, email) VALUES (?, ?) becomes a 500 on every single request, on every replica that hasn't restarted
yet, until the rollout finishes. You have coupled a schema change to a code
deploy, and the two of them do not happen atomically across a fleet.
The plan: expand, migrate, contract
Expand-contract (sometimes "parallel change") solves this by never letting the schema and the code disagree about what's safe. Instead of one migration and one deploy, it's four:
- Expand — add the new column, alongside the old one. Nothing reads it yet. Nothing that's running has to change.
- Migrate writes — deploy code that writes to both columns. Every row created or updated from this point on is consistent in both places.
- Migrate reads — deploy code that reads from the new column instead of the old one. This is a separate deploy from step 2, and the gap between them matters more than it looks like it should — see chapter 5.
- Contract — once every replica in the fleet is confirmed running the Stage 4 code from step 3, drop the old column. Nothing is reading or writing it anymore, so dropping it is safe.
The diagram's bottom half is the fact the rest of this article keeps coming back to: at every point during a rolling deploy, two adjacent stages are running against the same table at the same time. Deploy 2's rollout has Stage 1 and Stage 2 replicas live together for however long the rollout takes; Deploy 3's rollout has Stage 2 and Stage 3 together; Deploy 4a's has Stage 3 and Stage 4 together. Each of those overlaps is a window where the "old" code and the "new" code both have to produce correct answers against a schema neither one fully owns. Chapter 8 is the test that checks every one of those six write/read combinations directly, and the article's live load-generator run reproduces the same overlaps under real HTTP traffic and real timing.
What never changes
The four deploys change exactly one thing about how the database is used. They change nothing about the API:
public record Customer(long id, String name, String email) {
}
Customer.java
never mentions email_address — a caller of this API cannot tell which stage
answered a given request just by looking at the response shape, and neither can the
article's own load generator. That's deliberate: expand-contract is a technique for
changing storage without changing the contract clients depend on.
Going deeper
- The companion module for this article is
db-migrations-expand-contractinspring-boot-demo— every chapter from here on links to a real file or a real captured transcript in it. - Martin Fowler's ParallelChange is the canonical name and description of this pattern outside a specific database or framework.