asmhatreandClaude Sonnet 5
e478eafda3
Companion code for Zero-Downtime Database Migrations: Expand-Contract in Practice with Spring Boot: a full expand/migrate-writes/migrate-reads/contract sequence run as an actual rolling deploy across two live replicas, with a load generator sending continuous HTTP traffic through all four deploys (99.98% success, every residual error traced to a root cause rather than left unexplained). Findings include a real NOT NULL constraint trap in the expand migration, a backfill-window bug in the read switch, H2's AUTO_SERVER=TRUE single-point-of-failure behavior under a rolling restart, the drain-before-SIGTERM fix needed to close a health-check gap during graceful shutdown, and H2 silently discarding a concurrently committed INSERT during an ALTER TABLE ADD/DROP COLUMN rebuild - confirmed, by primary source, to be an H2-specific behavior rather than a property of the technique itself. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_019Fb7vW8vLyLKngBc4R3huA
117 lines
6.0 KiB
Markdown
117 lines
6.0 KiB
Markdown
# 1. The problem and the plan
|
|
|
|
[Next: 2. The expand migration →](02-the-expand-migration.md)
|
|
|
|
## The problem
|
|
|
|
`customers` has a column called `email`. You want it called `email_address` — maybe
|
|
because a second `contact_email` table is coming and the naming needs to be
|
|
consistent, maybe because "email" collided with a reserved word in a tool you just
|
|
adopted. The reason doesn't matter. What matters is that this table has rows in it,
|
|
right now, in production, and something is reading and writing that column while you
|
|
work.
|
|
|
|
The naive fix is one migration:
|
|
|
|
```sql
|
|
ALTER TABLE customers RENAME COLUMN email TO email_address;
|
|
```
|
|
|
|
That statement is correct and it is also a production outage. The instant it commits,
|
|
every currently-running copy of your application — the ones you have not redeployed
|
|
yet, because a rolling deploy takes minutes, not zero seconds — starts issuing SQL
|
|
against a column that no longer exists. `INSERT INTO customers(name, email) VALUES
|
|
(?, ?)` becomes a 500 on every single request, on every replica that hasn't restarted
|
|
yet, until the rollout finishes. You have coupled a **schema change** to a **code
|
|
deploy**, and the two of them do not happen atomically across a fleet.
|
|
|
|
## The plan: expand, migrate, contract
|
|
|
|
Expand-contract (sometimes "parallel change") solves this by never letting the schema
|
|
and the code disagree about what's safe. Instead of one migration and one deploy, it's
|
|
four:
|
|
|
|
1. **Expand** — add the new column, alongside the old one. Nothing reads it yet.
|
|
Nothing that's running has to change.
|
|
2. **Migrate writes** — deploy code that writes to *both* columns. Every row created
|
|
or updated from this point on is consistent in both places.
|
|
3. **Migrate reads** — deploy code that reads from the new column instead of the old
|
|
one. This is a *separate* deploy from step 2, and the gap between them matters more
|
|
than it looks like it should — see [chapter 5](05-the-read-switch.md).
|
|
4. **Contract** — once every replica in the fleet is confirmed running the Stage 4
|
|
code from step 3, drop the old column. Nothing is reading or writing it anymore, so
|
|
dropping it is safe.
|
|
|
|
<svg viewBox="0 0 760 230" xmlns="http://www.w3.org/2000/svg" role="img" aria-label="Four deploys, each one changing exactly one thing">
|
|
<style>
|
|
text{font-family:-apple-system,Segoe UI,Helvetica,Arial,sans-serif;font-size:13px;fill:#1a1a1a}
|
|
.hdr{font-weight:600;font-size:13px}
|
|
.box{fill:#eef4fc;stroke:#3b6fb0;stroke-width:1.4;rx:6}
|
|
.dim{fill:#f4f4f4;stroke:#999;stroke-width:1;rx:6}
|
|
.arrow{stroke:#555;stroke-width:1.6;marker-end:url(#a)}
|
|
</style>
|
|
<defs><marker id="a" markerWidth="8" markerHeight="8" refX="6" refY="3" orient="auto"><path d="M0,0 L6,3 L0,6 z" fill="#555"/></marker></defs>
|
|
<rect x="10" y="10" width="170" height="70" class="box"/>
|
|
<text x="20" y="30" class="hdr">1. EXPAND</text>
|
|
<text x="20" y="48">add email_address</text>
|
|
<text x="20" y="64">schema only, no deploy</text>
|
|
<rect x="200" y="10" width="170" height="70" class="box"/>
|
|
<text x="210" y="30" class="hdr">2. MIGRATE WRITES</text>
|
|
<text x="210" y="48">write both columns</text>
|
|
<text x="210" y="64">code deploy</text>
|
|
<rect x="390" y="10" width="170" height="70" class="box"/>
|
|
<text x="400" y="30" class="hdr">3. MIGRATE READS</text>
|
|
<text x="400" y="48">read email_address</text>
|
|
<text x="400" y="64">code deploy</text>
|
|
<rect x="580" y="10" width="170" height="70" class="box"/>
|
|
<text x="590" y="30" class="hdr">4. CONTRACT</text>
|
|
<text x="590" y="48">drop email</text>
|
|
<text x="590" y="64">schema, then cleanup</text>
|
|
<line x1="180" y1="45" x2="198" y2="45" class="arrow"/>
|
|
<line x1="370" y1="45" x2="388" y2="45" class="arrow"/>
|
|
<line x1="560" y1="45" x2="578" y2="45" class="arrow"/>
|
|
<rect x="10" y="110" width="740" height="100" class="dim"/>
|
|
<text x="20" y="132" class="hdr">Same database, the whole time</text>
|
|
<text x="20" y="152">customers.email [always present until step 4]</text>
|
|
<text x="20" y="172">customers.email_address [present from step 1 onward, populated from step 2 onward]</text>
|
|
<text x="20" y="194">Two adjacent stages serve real traffic against this table at once during every rollout above.</text>
|
|
</svg>
|
|
|
|
The diagram's bottom half is the fact the rest of this article keeps coming back to:
|
|
at every point during a rolling deploy, two adjacent stages are running against the
|
|
same table at the same time. Deploy 2's rollout has Stage 1 and Stage 2 replicas live
|
|
together for however long the rollout takes; Deploy 3's rollout has Stage 2 and Stage
|
|
3 together; Deploy 4a's has Stage 3 and Stage 4 together. Each of those overlaps is a
|
|
window where the "old" code and the "new" code both have to produce correct answers
|
|
against a schema neither one fully owns. [Chapter 8](08-the-rolling-window-proof.md)
|
|
is the test that checks every one of those six write/read combinations directly, and
|
|
the article's live load-generator run reproduces the same overlaps under real HTTP
|
|
traffic and real timing.
|
|
|
|
## What never changes
|
|
|
|
The four deploys change exactly one thing about how the *database* is used. They
|
|
change nothing about the *API*:
|
|
|
|
```java
|
|
public record Customer(long id, String name, String email) {
|
|
}
|
|
```
|
|
|
|
[`Customer.java`](https://ankurm.com/git.app/asmhatre/spring-boot-demo/src/branch/main/db-migrations-expand-contract/src/main/java/com/ankurm/expandcontract/customer/Customer.java)
|
|
never mentions `email_address` — a caller of this API cannot tell which stage
|
|
answered a given request just by looking at the response shape, and neither can the
|
|
article's own load generator. That's deliberate: expand-contract is a technique for
|
|
changing storage without changing the contract clients depend on.
|
|
|
|
## Going deeper
|
|
|
|
- The companion module for this article is `db-migrations-expand-contract` in
|
|
[`spring-boot-demo`](https://ankurm.com/git.app/asmhatre/spring-boot-demo) — every
|
|
chapter from here on links to a real file or a real captured transcript in it.
|
|
- Martin Fowler's [ParallelChange](https://martinfowler.com/bliki/ParallelChange.html)
|
|
is the canonical name and description of this pattern outside a specific database or
|
|
framework.
|
|
|
|
[Next: 2. The expand migration →](02-the-expand-migration.md)
|