Release Rollback & Backward-Compatible Database Changes

Release Rollback and Database Compatibility

Rolling back application code is easy: redeploy the previous artifact. Rolling back a release that also changed the database is where teams get hurt. If version 2 renamed a column, dropped a table, or rewrote data in a new format, version 1 cannot run against the database version 2 left behind — and the "quick rollback" becomes an outage, a restore from backup, or both.

The way out is a discipline rather than a tool: every database change must work with both the old and the new version of the application, because during any rolling deploy, canary, or rollback, both versions run against the same database at the same time. Destructive changes are split into several small, reversible releases, and the irreversible step comes last, once nothing depends on the old shape.

TL;DR

Assume two application versions run at once; every schema must work with the current and the previous version.
Change schemas with expand, migrate, contract: additive changes first, destructive changes last, in a later release.
Never ship a destructive migration in the same release as the code that stops using the old structure.
Run canaries with automated analysis against a baseline, and roll back automatically when they breach thresholds.
Use feature flags as kill switches for risky behavior, so many incidents need no deploy at all.
Agree in advance when to roll back and when to roll forward, and rehearse both.

Quick Example

Renaming users.name to users.full_name without ever breaking a rollback:

At every step the previous release still works against the current schema: release 1 still reads name, which release 2 keeps writing, and release 2 reads full_name, which release 3 keeps writing. Only the final step is irreversible, and by then no running code uses the old column.

Core Concepts

The Mixed-Version Window

Old and new code coexist more often, and for longer, than most teams expect:

During a rolling update, some instances run the old version until the last one is replaced.
During a canary, a small share of traffic runs the new version for minutes or hours.
Blue-green deployments switch traffic between two versions that usually share one database.
Background workers, cron jobs, and queued messages created by one version are processed by another.
Mobile and desktop clients keep calling old API shapes for months.
A rollback returns the old version to a database the new version has already changed.

Which Schema Changes Are Safe in One Step

Some ORMs cache a table's column list and include every column in inserts, so dropping a column breaks running instances even when the code no longer mentions it. Tell the ORM to ignore the column one release before dropping it. See Database Migrations.

Schema Changes Versus Data Changes

Adding a column is reversible: the old version ignores it. Rewriting data — merging two fields, changing units, normalizing values — is not, unless the original data is kept. Write transformed data to new columns or tables and keep the originals until the new version has proven itself.

Canary Analysis and Automated Rollback

A canary sends a small share of traffic to the new version and compares it with the stable version over the same period. Comparing against a live baseline, rather than a fixed threshold, separates "the new version is worse" from "everything is slow right now".

Signals worth comparing: error rate, p95 and p99 latency, saturation, crash and restart counts, new error types in logs, and business metrics such as checkout success. Tools such as Argo Rollouts, Flagger, and Spinnaker automate the analysis and roll back when a metric fails:

Automated rollback only helps if rolling back is safe — which is exactly what backward-compatible migrations guarantee. SLO burn-rate signals make good promotion gates as well; see Implementing Multiwindow Burn-Rate Alerts.

Roll Back or Roll Forward?

Decide the rules before the incident. A common one: mitigate first, debug later — if a release is the likely cause and rollback is safe, roll back within minutes and investigate afterwards. Time-box diagnosis so a team does not spend an hour debugging a problem a two-minute rollback would have ended. When data must be restored, the restore path should already be rehearsed — see Database Restore Drills and Recovery Targets.

Other State That Breaks Rollbacks

The database is the biggest source of rollback failures, but not the only one:

Messages and events written in a new format that old consumers cannot parse — version schemas and keep readers tolerant; see Event Schema Evolution.
Cache entries and sessions serialized in a new shape that old code cannot deserialize.
Background jobs enqueued by the new version with new argument formats.
API responses that clients have already adapted to.
External side effects — emails sent, payments captured, partner webhooks fired — which no rollback reverses.

Each needs the same rule as the schema: write in a format the previous version can read until the new version is safely established.

Best Practices

Deploy Migrations Separately From Code

Run expand migrations before the release that needs them, and contract migrations in a later release after the new code is stable everywhere. Keeping them separate makes each step small, reviewable, and individually reversible.

Lint Migrations in CI

Tools such as Squawk for PostgreSQL and strong_migrations for Rails flag operations that take long locks or break compatibility, such as adding a NOT NULL column without a default or renaming a column. Make them a merge gate — see CI Pipeline Design and Quality Gates.

Test the Rollback, Not Just the Deploy

In a pre-production environment, deploy the new version, run its migrations, then deploy the previous version against the migrated schema and run smoke tests. Automating this "downgrade test" catches most compatibility mistakes before production does.

Keep Previous Artifacts One Command Away

Rollback should use the same pipeline as deployment, with the previous artifact already built and stored. A rollback that requires rebuilding or manual steps is too slow when it matters.

Put Risky Behavior Behind Flags

Feature flags separate deploying code from enabling behavior. Turning off a flag is faster and safer than any rollback, and leaves the database untouched.

Common Mistakes

Dropping a Column in the Same Release That Stops Using It

Relying on Down Migrations

Adding a Required Column in One Step

Canaries Judged by Eye

FAQ

Should we write down migrations?

They are useful in development, but they make a poor production rollback plan: they are rarely tested, and reversing a schema change often destroys data written since. Prefer forward-only migrations that keep the previous application version working, and treat genuine data damage as a restore or a forward fix.

How long should we wait between expand and contract?

Until the new version is fully deployed and stable, every old instance, worker, and queued job is gone, and you no longer intend to roll back past that release — commonly at least one full release cycle. There is rarely a reason to rush the contract step.

Does blue-green deployment avoid these problems?

Not when both environments share a database, which is the usual case. Blue and green still run against one schema, so it must support both versions. Blue-green makes switching traffic fast; it does not make schema changes reversible.

Which metrics should a canary check?

Error rate and latency against the stable baseline, resource saturation, crash or restart counts, and at least one business metric that reflects user success, such as sign-ins or completed checkouts. Purely technical metrics miss releases that work but do the wrong thing.

How do we undo a data migration?

Design it to be undoable: write transformed data to new columns or tables, keep the originals until the new version has proven itself, and make the transformation idempotent. If the original data is already gone, recovery means restoring from backups or point-in-time recovery, which is why destructive data changes come last.