Ask any team whether they can roll back a bad deploy and the answer is yes. Ask them when they last did it under pressure and the answer gets longer. Somewhere in that longer answer is a story about a migration, or a queue full of messages in a new format, or a feature flag that turned out to be one-way.
Redeploying the previous artefact is trivial and every platform does it. That is not rollback. Rollback is returning the system to its previous state, and the system includes a database, a message broker, a cache, other people’s inboxes and a payment processor. The artefact is the easy half.
Four things quietly remove the capability
Schema changes that the old code cannot read. The classic is a dropped or renamed column. The new version does not need it, so the migration removed it, and the old version selects it explicitly. Rolling back the code now produces an immediate and total failure, which is a worse outcome than the bug you were rolling back from.
Data written in a shape the old code cannot handle. Subtler and more common. The schema is compatible, but the new version started writing a new enum value, or a nullable field that the old code assumes is present, or a JSON payload with a nested structure. Roll back and the old code runs into rows it cannot parse. It works for ninety percent of traffic, which is the worst possible failure mode because it looks like a partial outage rather than a rollback problem.
Side effects that already left the building. Emails sent. Webhooks delivered. A charge captured. A row created in a partner’s system. No deploy reverses these. If the bug you are rolling back from was producing bad outbound effects, rollback stops the bleeding and does nothing about the blood.
Configuration that only moves forward. A flag flipped on that triggered a one-time backfill. A queue consumer group whose offset was reset. A cache warmed with entries in a new format that the old code will happily read and misinterpret. Each of these is a small decision that silently converts a reversible deploy into an irreversible one, and none of them appears in the deploy diff.
You do not find out whether you can roll back on a good day. You find out at 02:00, from the option you assumed you had.
Expand and contract, actually separated
The standard answer is expand and contract: add the new thing, migrate to it, then remove the old thing. Most teams know the pattern. Fewer enforce the property that makes it work, which is that expand and contract must be separate deploys with a real interval between them, and the interval has to be longer than your rollback window.
For a column rename, that means four deploys, not one:
deploy 1 add column new_name, nullable. nothing reads it.
ROLLBACK SAFE: old code never saw it.
deploy 2 write both old_name and new_name. read old_name.
ROLLBACK SAFE: old code reads the column it always read.
deploy 3 read new_name. keep writing both.
ROLLBACK SAFE: deploy 2 is still writing old_name.
<-- soak here. days, not minutes.
deploy 4 stop writing old_name, then later drop it.
NOT ROLLBACK SAFE past this point, and that is fine,
because the change has been in production for days.The temptation is to collapse this into two deploys because four feels like ceremony for a rename. The four exist so that at every moment before the last one, the previous artefact is a valid thing to run. That is the entire property you are buying.
Rollback has to be faster than fixing forward
There is a practical threshold here that gets ignored. If rolling back takes twenty minutes of coordination and a fix-forward patch takes fifteen, nobody will roll back. They will fix forward, under pressure, writing code at the worst possible time, because it is the faster path and humans take the faster path during incidents.
So the target is not "rollback is possible". It is "rollback is the obviously cheapest option", which means one command or one button, no approvals in the path, and no need to think about which version to go back to. If the on-call engineer has to look up the last known good tag, you have added a step that will be skipped.
The corollary is that fix-forward is not the enemy. For a small, obvious, well-understood bug it is often correct. The problem is when it becomes the default because rollback was never made cheap, and the team ends up writing production patches at 02:00 as a matter of routine.
Say out loud where reversibility ends
Some changes genuinely cannot be reversed, and pretending otherwise is worse than admitting it. A destructive migration, a partner integration cutover, a pricing change that has already billed someone. These deserve to be labelled at review time, not discovered at incident time.
One line in the change description is enough:
- Reversible. Redeploy the previous artefact and the system is as it was.
- Reversible with a compensating action. Rollback plus a named cleanup step, written before the deploy, not improvised after it.
- One-way. There is no rollback. The mitigation is a forward fix, and the plan for that fix exists before the deploy goes out.
Nothing about this slows down the reversible majority, which is most changes. It puts a small amount of friction exactly where the risk is.
Rehearse it, or it is not a capability
A rollback path that has never been executed is an assumption, and assumptions expire. The rehearsal does not need to be dramatic. Once a month, in business hours, with the team watching, roll a real service back one version in production and roll it forward again.
What this surfaces, every time, is not the rollback mechanism. It is the surrounding assumptions: a metric that alerts on the version change, a health check that takes longer to pass than anyone remembered, a dependency that pinned itself to a schema the old version does not produce. Those findings are the product of the exercise.
Measure one number and publish it: time from decision to traffic served by the previous version. If that number is unknown, rollback is a belief. If it is known and under a few minutes, it is a feature, and it is one of the few features that pays for itself the first time it is used.