Zero-Downtime Database Migrations: Expand-Contract in Practice with Spring Boot

A teammate asks you to rename a column. customers.email needs to become customers.email_address — a naming cleanup, nothing more. You write the obvious migration — ALTER TABLE customers RENAME COLUMN email TO email_address; — the one-line fix this entire article argues you should not run against a live table, and the reason it never appears as a file in the companion repository.

It runs in eleven milliseconds against your staging database. Then you run it against production, and for the next four minutes — however long your rolling deploy takes to finish — every replica that hasn’t restarted yet throws a SQL error on every single request that touches this table. You didn’t break the database. You broke the assumption that a schema change and a code deploy happen at the same instant, and they don’t.

This article builds the fix — a technique called expand-contract — as a real, running thing: two live Spring Boot replicas, a shared database, and a load generator that keeps sending real HTTP traffic through an actual four-deploy rolling sequence while the schema changes underneath it. Along the way it finds two genuine bugs in its own first draft, and one surprising, verified limitation in the database itself that no amount of correct application code would have fixed.

Versions used in this article. Spring Boot 4.1.1 (GA 2026-08-20), Flyway 12.4.0 (pulled in transitively by spring-boot-starter-flyway), H2 2.4.240, JDK 25 LTS (GA 2025-09-16). Every version here was confirmed against Maven Central’s metadata and the project’s own release notes, not copied from a tutorial — see the companion repository for the exact pom.xml.

The full runnable code for everything below lives in db-migrations-expand-contract, one module inside the spring-boot-demo repository. Every code block and every line of output below links to a real file in it.

Why “just rename the column” is a production outage

The problem isn’t the SQL. ALTER TABLE ... RENAME COLUMN is correct, fast, and exactly what you’d want if the whole fleet updated atomically. It doesn’t. A rolling deploy replaces your application’s replicas one at a time, specifically so that the service never goes fully down — and that’s precisely what turns an instant, all-at-once schema rename into a multi-minute window where old code is still running against a column that no longer exists.

The diagram’s argument is the whole problem in one line: the schema change is a point in time, but the deploy is a duration, and “old code, new schema” is real and unavoidable for the length of that duration. Expand-contract’s answer isn’t a smarter migration — it’s refusing to let any single deploy require the schema and the code to agree.

Going deeper

Martin Fowler’s ParallelChange is the name and origin of this pattern outside any specific framework. The companion repo’s chapter 1 covers the same ground with the article’s own running example already in place.

The fix is four deploys, not one

Instead of one migration, expand-contract runs the change as four separate deploys, each changing exactly one thing:

Deploy	What changes	Kind
1 — Expand	Add email_address, nullable, alongside email	Schema only, zero app restarts
2 — Migrate writes	Code writes to both columns	Code deploy
3 — Migrate reads	Code reads from email_address instead	Code deploy
4 — Contract	Drop email, once every replica is confirmed on Deploy 3’s code	Code deploy, then schema only

Every one of those four steps is individually safe for a fleet that hasn’t fully rolled out yet, because at no point does a step require code and schema to change in the same instant. The two schema-only steps (1 and the second half of 4) run against the live database with zero application restarts — both replicas keep serving traffic on whatever code they’re already running while the ALTER TABLE executes.

Plant this now, it pays off later. “Schema-only, zero app restarts” sounds like it means zero effect on anything. It means zero effect on which code is running. It says nothing about what a concurrently-executing ALTER TABLE statement does to other queries hitting the same table at the same instant — and on at least one popular database, the honest answer turns out to be more interesting than “nothing”. Hold onto that; it’s the last and biggest section of this article.

Going deeper

The full four-deploy sequence, its rationale, and the two-stages-live-at-once property every rolling deploy has, are covered start to finish in the repo’s chapter 1.

Deploy 1: the expand migration

The whole technique starts with a schema change that changes nothing about how the app currently behaves. Here’s Deploy 1’s actual migration, in full:

Source: V2__add_email_address_column.sql

ALTER TABLE customers ADD COLUMN email_address VARCHAR(320);
UPDATE customers SET email_address = email WHERE email_address IS NULL;
ALTER TABLE customers ALTER COLUMN email DROP NOT NULL;

Three statements, three jobs: add the column nullable (so no existing INSERT breaks), backfill it for rows that already exist, and relax the old column’s constraint (the third line, and the one that’s easy to forget — more on that in a moment). Running this against a database seeded with one pre-existing row, then inserting a brand-new row with code that has never heard of email_address, produces exactly what you’d hope:

Output: docs/output/02-expand-backward-compatible.txt

-- Ada's row was backfilled by the migration itself --
NAME         | EMAIL            | EMAIL_ADDRESS
-------------+------------------+-----------------
Ada Lovelace | [email protected] | [email protected]
(1 row)

-- Stage 1's original INSERT still works, unmodified, after the migration --
NAME         | EMAIL              | EMAIL_ADDRESS
-------------+--------------------+--------------
Grace Hopper | [email protected] | NULL
(1 row)

Grace Hopper’s row is the interesting one: EMAIL_ADDRESS is NULL, and that’s correct — nothing has shipped yet that would populate it for a brand-new insert. That gap is exactly what Deploy 2 exists to close.

The bug this module’s own first draft shipped. Forget that third line — relaxing the old column’s NOT NULL constraint — and Deploy 4’s code, which never writes email at all, fails every single customer creation from the first request onward, because it’s omitting a column the database still requires. This is a real bug this article’s own draft migration had.

Deploy 1’s migration runs through a standalone entry point, MigrationCli, deliberately kept outside the Spring Boot application — spring.flyway.enabled=false in this module’s application.yml. If Flyway ran on every application boot instead, an unrelated restart (an OOM, a routine redeploy) could silently trigger a schema change nobody asked for at that moment. Running migrations from their own process, invoked deliberately, keeps “when does the schema change” and “when does a replica restart” as two answerable, independently-controlled questions.

One to two paragraphs of intermediate depth: the same discipline that keeps a migration job as its own CI/CD pipeline step rather than folded into application startup. See chapter 3 for the full argument and how MigrationCli‘s --target flag lets a “deploy” migrate exactly as far as that step requires and no further.

Going deeper

• The exact NOT NULL failure this migration’s naive first draft produced, captured live: chapter 6.
• Why Flyway runs from its own process instead of on application boot: chapter 3.
• What this specific ALTER TABLE actually does to concurrent traffic while it runs — not just “does it change the API”: chapter 14.

Deploy 2: writing to both columns

Deploy 2 is the first code deploy, and it changes exactly one thing: what create() and updateEmail() write. Here’s the shipped write path in CustomerService:

case 2, 3 -> jdbc.sql("INSERT INTO customers(name, email, email_address) VALUES (?, ?, ?)")
        .param(name).param(email).param(email)
        .update(keyHolder, "id");

case 2, 3 -> jdbc.sql("UPDATE customers SET email = ?, email_address = ? WHERE id = ?")
        .param(newEmail).param(newEmail).param(id).update();

Stage 2 and Stage 3 share this exact write path — the difference between them, covered next, is only what they read. Creating a customer and then updating their email produces the same value in both columns at every step, never a stale value left behind in one of them:

Output: docs/output/04-dual-write-consistency.txt

-- after updateEmail() - the old value is gone from BOTH columns, not just one --
NAME              | EMAIL                   | EMAIL_ADDRESS
------------------+-------------------------+------------------------
Margaret Hamilton | [email protected] | [email protected]

The “not just one” in that heading is there on purpose: a dual write that inserts into both columns but only updates one is a much easier bug to ship than it sounds, because the update code path is usually written separately, by someone who has already stopped thinking about the second column.

Going deeper

• The full write-path implementation across all four stages: chapter 4.
• Why writes stay dual for two whole deploys (2 and 3) rather than one: chapter 5.

The bug that happens if the read switch moves too early

Here’s the failure worth leading with, because it’s the one a naive implementation of “step 3: switch the read” walks straight into. Deploy 2’s own rollout is not instantaneous — some replicas are still Stage 1 for however long that rollout takes. A Stage 1 replica, unaware anything has changed, keeps inserting rows the old way: email only, email_address left NULL. If Deploy 3’s read switch were the very next deploy after Deploy 2 started rolling out, a Stage 3 replica reading email_address alone would return NULL for every row a still-live Stage 1 replica had just written.

Reproducing this directly, with a row written the way a lingering Stage 1 instance actually would:

Output: docs/output/07-backfill-window-bug.txt

-- a NAIVE Stage 3 read (email_address alone) - the bug --
naive Stage 3 email column value: null

-- the SHIPPED Stage 3 read (CustomerService, COALESCE) - the fix --
CustomerService (stage 3) result: Optional[Customer[id=1, name=Katherine Johnson, [email protected]]]

The shipped fix, from CustomerService.findById:

case 3 -> "SELECT id, name, COALESCE(email_address, email) AS email FROM customers WHERE id = ?";

Stage 2 never needs this fallback — it still reads the original email column, which every stage keeps populated the entire time. Only Stage 3, the first stage to prefer the new column, needs to tolerate rows that haven’t been dual-written yet. This is why the write switch (Deploy 2) and the read switch (Deploy 3) have to be separate deploys: bundling them would mean Deploy 3’s read code starts running before Deploy 2’s write code has finished rolling out everywhere, and the gap between “started rolling out” and “100% rolled out” is precisely this bug’s habitat.

Going deeper

• Why the write switch and the read switch can’t be the same deploy, spelled out directly: chapter 5.
• The full backfill-window reproduction, including the naive read that fails first: chapter 7.

Proving it for real: every stage pair, cross-checked

Every rolling deploy in this sequence — Deploy 2, Deploy 3, Deploy 4a — has a window where two adjacent stages serve real traffic against the same database at the same time. Rather than trust that by argument, this module checks all six write/read direction pairs directly: Stage 1↔2, Stage 2↔3, and Stage 3↔4, both directions each.

Output: docs/output/08-mixed-stage-rolling-window.txt

-- Stage 1 writes, Stage 2 reads --
Customer[id=1, name=Radia Perlman, [email protected]]

-- Stage 3 writes, Stage 4 reads --
Customer[id=5, name=Adele Goldberg, [email protected]]

-- Stage 4 writes, Stage 3 reads --
Customer[id=6, name=Karen Sparck Jones, [email protected]]

All six pairs pass. If any single one of them failed, the technique wouldn’t be zero-downtime for that rollout — it would just be a race against however long the rollout takes, with correctness depending on luck rather than design.

Going deeper

The full six-pair test and its rationale: chapter 8.

Deploy 4: contracting, and what happens if you drop too soon

Deploy 4 splits the same way Deploy 1 did — 4a is a pure code deploy to Stage 4 (which never touches email), and only once every replica is confirmed on that code does 4b run the actual drop, again with zero app restarts:

Source: V3__drop_email_column.sql

ALTER TABLE customers DROP COLUMN email;

Reads and writes both keep working after the drop, for a row that predates it and one created entirely after:

Output: docs/output/09-contract-safety.txt

-- Stage 4 create + read, entirely after the drop --
Customer[id=2, name=Mary Allen Wilkes, [email protected]]

Why the fleet has to be confirmed first, not just “probably done”. Run 4b before every replica has actually reached Stage 4, and a lingering old instance gets exactly what it should: a loud, immediate failure. Captured directly:

org.h2.jdbc.JdbcSQLSyntaxErrorException: Column "EMAIL" not found

This is the correct outcome, not a bug to work around — it’s why a real pipeline gates Deploy 4b on every instance’s confirmed version, not a fixed timer.

Going deeper

• The contract migration’s design and why it’s split into 4a/4b: chapter 9.
• The drop-too-soon failure, and why it’s a feature rather than a bug: chapter 10.

Making it real: two live replicas and continuous traffic

Every section so far has proven a property with a JUnit test against a shared database connection — real code, real SQL, but no real HTTP, no real process restarts, no real timing pressure. scripts/run-all.sh is where all three show up: two actual Spring Boot processes, a standalone database, a real rolling restart between every stage, and a load generator hammering both replicas the entire time.

The load generator (LoadGenerator) only sends traffic to backends its own health checks currently believe are up, and it checks the content of every response, not just the status code:

JsonNode node = MAPPER.readTree(resp.body());
String returnedEmail = node.get("email").asString();
if (!target.email().equals(returnedEmail)) {
    c.recordError("read-consistency-mismatch");
    return;
}

Running all four deploys with the load generator active the whole time produced this:

Output: docs/output/12-load-generator-summary.txt

Load generator summary
=======================
Total requests: 30911
Successful:     30905
Errors:         6

By phase:
  04a-stage4-soak              ok=2130     error=0
  04b-contract-migration       ok=336      error=0
  03-deploy-stage3-rollout     ok=4645     error=0
  05-final-soak                ok=7118     error=4
      - read-http-404            2
      - update-http-404          2
  01-expand-migration          ok=1773     error=1
      - read-http-404            1
  02-deploy-stage2-rollout     ok=4679     error=1
      - update-http-404          1

30,905 of 30,911 real HTTP requests succeeded — 99.98%, through a genuine rolling restart across all four deploys. Getting there required fixing two real infrastructure bugs first, and the six errors that remain have a single, fully-explained cause covered in the next section.

H2’s AUTO_SERVER=TRUE is a single point of failure, not a shared database. This module’s first draft pointed both replicas at one H2 file with AUTO_SERVER=TRUE, the mode most tutorials show for sharing an embedded file between JVMs. It makes the first connecting process the de facto server for every other connection — kill that one replica during a routine restart, and every other replica’s “embedded” database breaks with it. This was the single biggest source of load-generator errors before it was found and fixed, cutting the error rate by roughly 90% on its own. The fix: run H2 as an actual standalone TCP server process that neither replica owns — see chapter 12.

The second infrastructure fix closed the gap between “graceful shutdown” and “the load balancer noticed”. server.shutdown: graceful only changes behavior on SIGTERM — it stops accepting new connections but finishes in-flight ones. That’s necessary but not sufficient: the load generator’s health checker polls every 300ms, and the process can already be refusing connections before the pool has noticed and rerouted around it. The fix is a /admin/drain endpoint (DrainController) that publishes Spring’s ReadinessState.REFUSING_TRAFFIC — the same event a Kubernetes preStop hook fires — before SIGTERM is sent at all, giving the health check a couple of poll cycles’ head start:

curl -s -X POST "http://localhost:$PORT/admin/drain" -o /dev/null || true
sleep 1.5
kill -15 "$PID"

Together, drain-then-SIGTERM took the rolling-restart portion of this run’s errors to zero — every phase above except the four listed shows error=0.

Going deeper

• The full load generator design — health-check debouncing, per-thread state isolation, the startup race it used to have: chapter 11.
• The AUTO_SERVER trap in full, with H2’s own automatic mixed mode documentation: chapter 12.
• Graceful shutdown vs. kill -9, and Spring’s Application Availability mechanism this drain endpoint uses: chapter 13.

The finding: ALTER TABLE can silently lose a row nobody was told about

This is the section the whole article has been building toward, and it’s the reason “zero errors” in the summary above is 99.98%, not 100%. Getting there took two separate fixes for two separate failure modes, both triggered by the same cause: the schema-only migrations (Deploy 1’s ADD COLUMN, Deploy 4b’s DROP COLUMN) running while real traffic keeps hitting the table.

Failure mode 1: a statement that collides with the DDL, and says so

While Deploy 4b’s DROP COLUMN executes, a concurrent, otherwise-correct query can briefly see the table disappear out from under it. Reproducing this on demand, at the raw JDBC level, with one thread reading the table in a loop while the drop runs on another connection:

Output: docs/output/14-ddl-collision-exception.txt

successful reads while DROP COLUMN was in flight: 222
reads that collided with the in-flight DROP COLUMN: 1
example: org.h2.jdbc.JdbcSQLSyntaxErrorException: Table "CUSTOMERS" not found; SQL statement:
SELECT COUNT(*) FROM customers [42102-240]

This same condition is what first showed up as create-http-500 / update-http-500 errors in this module’s own live load-generator run, in independent runs, before the fix below — not a bug in the application’s SQL. The fix is a narrowly-scoped single retry in CustomerService that catches specifically this grammar error, waits 50ms — long enough for the in-flight ALTER TABLE to finish — and retries exactly once:

private <T> T withRetryForConcurrentDdl(Supplier<T> operation) {
    try {
        return operation.get();
    } catch (BadSqlGrammarException ex) {
        if (ex.getMessage() != null && ex.getMessage().contains("CUSTOMERS")) {
            try {
                Thread.sleep(50);
            } catch (InterruptedException ie) {
                Thread.currentThread().interrupt();
            }
            return operation.get();
        }
        throw ex;
    }
}

It’s deliberately narrow — anything other than this specific, verified condition still propagates, so a genuine programming error never gets silently swallowed. After adding it, a full re-run of the live sequence produced zero errors of this class during Deploy 4b (04b-contract-migration ok=336 error=0 in the summary above).

Failure mode 2: a statement that succeeds, and is still lost

The retry only helps when the colliding statement throws. It does nothing for the six residual 404s in the summary — a customer id that a 201 response had already confirmed existed, later reported not found by a plain read or update, with no exception anywhere. Reproducing that gap directly is what led to the actual mechanism:

Thread inserter = new Thread(() -> {
    started.countDown();
    int n = 0;
    while (!stop.get()) {
        n++;
        try {
            long id = stage1.create("Concurrent " + n, "concurrent" + n + "@example.test");
            confirmedIds.add(id);
        } catch (Exception ex) {
            insertErrors.add(ex.getClass().getSimpleName());
        }
    }
});
inserter.start();
started.await();

// The DDL runs on its own connection, exactly as Deploy 1 (EXPAND) runs V2 live
TestSupport.migrateTo(db, "2");

Source: DdlSilentDataLossTest

customer creates that returned a generated id with no error: 50
customer creates that got the already-documented, already-fixed DDL-collision error: 1
of the ids that came back with no error, missing from the table once V2 finished: 8
example missing ids: [41, 42, 43, 44, 45]

This is why the retry in CustomerService cannot be the whole fix: these inserts
never threw anything to retry. The row was committed, then discarded when the
ADD COLUMN rebuild swapped in a new table that had already been scanned.

Output: docs/output/14-ddl-silent-data-loss.txt

H2 implements both ALTER TABLE ... ADD COLUMN and DROP COLUMN by rebuilding the table: scanning every row into a new table with the new column layout, then swapping it in. A row inserted on another connection can commit while that scan is mid-flight. Depending on exactly when the scan reaches that row relative to the commit, the row ends up copied into the new table, or left behind in the old one — and when it’s left behind, it disappears the instant the rebuild finishes, with nothing telling the connection that inserted it anything went wrong.

Under a tight, unthrottled loop with no delay between inserts, this run lost 8 of the 50 rows inserted during the migration window — other runs made while writing this test lost anywhere from 1 to a few dozen, depending on exactly how the two threads got scheduled. That’s not a claim about the live article run’s own error rate; the load generator’s eight threads sleep 15–40ms between requests and aren’t hammering the exact millisecond the migration runs, so far fewer of its requests land in the vulnerable window. It’s a claim about the mechanism, and the mechanism is real and reproducible on demand.

This is also why the retry in CustomerService can never be “the whole fix” for this class of error — there is nothing to retry. The failure isn’t a rejected statement; it’s data that existed for a moment and then silently didn’t.

This is a property of H2, not of expand-contract. PostgreSQL’s own reference manual states plainly that ADD COLUMN without a volatile default, and DROP COLUMN, are both metadata-only operations — “in neither case is a rewrite of the table required” — so there’s no scan for a concurrent commit to race against. (Postgres does take a brief ACCESS EXCLUSIVE lock for the metadata change itself, which blocks concurrent statements rather than racing past them — a different, more familiar tradeoff.) The lesson isn’t “H2 is bad” — it’s that “ALTER TABLE ADD/DROP COLUMN is always safe under load” is a claim you should verify against your specific database, the same way this article verified it against this one, rather than assume from a blog post.

Going deeper

• Both failure modes, both dedicated reproduction tests (DdlCollisionExceptionTest and DdlSilentDataLossTest), the full retry implementation, and the complete PostgreSQL/MySQL comparison with primary sources: chapter 14 — this module’s central finding.
• PostgreSQL’s own ALTER TABLE reference, the primary source for the Postgres claim above.
• MySQL/InnoDB’s Online DDL documentation, covering which ALTER TABLE operations get INSTANT/INPLACE treatment versus a full COPY rebuild.

What the defaults don’t cover

This module is a demonstration, built so its failure modes are reachable on purpose. Taking the technique — not this exact code — to a real service means covering what it intentionally left out:

• Gate Deploy 4b on confirmed fleet state, not a timer. “Deploy 2 usually finishes in three minutes” is not the same guarantee as “every replica confirmed Stage 4” — check every instance’s reported version or health before running the contract migration.
• Verify your own database’s ALTER TABLE semantics before trusting either direction. Don’t assume “always safe” or “always errors loudly” — test it against your actual engine and table size the way this article tested it against H2.
• Lock down the diagnostic endpoints. /diag/schema and /admin/drain have no authentication in this demo. /admin/drain in particular can pull a real instance out of a real load balancer’s rotation with one unauthenticated POST.
• Run the load test against your real fleet size and real database, not a two-replica sandbox. Both the AUTO_SERVER trap and the DDL lock window were found by running actual concurrent traffic through actual restarts — reading about the technique wouldn’t have surfaced either one.
• Budget at least two code deploys plus two schema changes, not the single PR a column rename sounds like it should be.

Should you even do this?

Honestly: only if the table is live and the downtime isn’t acceptable. Expand-contract trades a single quick migration for four separate deploys, weeks of code that has to handle two column names at once, and — as this article found the hard way — a genuine need to understand your specific database’s concurrency behavior under DDL. If you can take a five-minute maintenance window, a plain RENAME COLUMN is simpler, safer, and produces less code to delete later. Reach for expand-contract when the table is under continuous production load and an outage window isn’t an option — not as a default habit for every schema change.

Further reading

• The full companion repository, including all 15 documentation chapters and every captured transcript quoted above.
• Flyway vs Liquibase for Spring Boot 4 — the migration tooling underneath this article’s MigrationCli, in more depth.
• Deploying Spring Boot 4 on Kubernetes — the probe groups and graceful-shutdown mechanics this article’s drain endpoint builds on, in a real cluster.
• Martin Fowler, ParallelChange.
• PostgreSQL, ALTER TABLE reference.
• MySQL, InnoDB Online DDL.
• H2 Database, Automatic Mixed Mode.
• Spring Boot reference docs, Application Availability.