The Recoverable Change Principle: Prove the Way Back

By Sergey Nosov

16 October 2026

On 29 May 2025, Google added new quota policy checks to Service Control, the binary that checks requests to Google Cloud APIs. According to Google’s incident report, the new code path never ran during the region-by-region rollout: it needed “a policy change that would trigger the code.” The change shipped with a “red-button” that could turn the path off. It did not ship behind a feature flag; with one, the report says, “the issue would have been caught in staging.”

On 12 June, a policy change with blank fields reached that path. Quota policy is global, so the data “was replicated globally within seconds,” and Service Control went into a crash loop in every region. The red-button rollout was complete within 40 minutes. In us-central1, one of the larger regions, the restarting tasks then overloaded the Spanner table they depend on. Full recovery there took up to about 2 hours and 40 minutes.

The switch existed before 12 June because it shipped with the change. But the change had no flag, the data had no staged rollout, and the restarts had no randomized exponential backoff. The Recoverable Change Principle asks for that preparation on every significant change:

A change must have a proven recovery strategy appropriate to its effects, with recovery objectives defined in measurable terms.

Three phrases carry the weight. Proven means exercised: automated where feasible, and validated by tests or drills. A recovery path that exists only on a wiki page is a hypothesis. Appropriate to its effects means the way back depends on where the effects went. A deployment that touched only your own servers can be made reversible; an email in a customer’s inbox cannot. Measurable means stating, before you ship, how fast service must come back, how much data you can afford to lose, and when the data must be correct again.

The principle has its own entry in my Software Development Principles series. The case for it starts with a number from Google’s SRE book: “SRE has found that roughly 70% of outages are due to changes in a live system.” The book’s answer is automation for three practices: progressive rollouts, fast and accurate detection, and safe rollback. My article on availability shows what recovery time does to the arithmetic of downtime. This article is about the way back itself: whether a change has one, and how to prove it works.

Two Kinds of Doors

In his letter to Amazon shareholders for 2015, Jeff Bezos sorted decisions into two kinds. Some are one-way doors: “If you walk through and don’t like what you see on the other side, you can’t get back to where you were before.” Those must be made slowly, with great care. Most decisions are two-way doors: “You can reopen the door and go back through.”

Reversibility is older than the metaphor. Andrew Hunt and David Thomas gave it its own section in The Pragmatic Programmer in 1999, with the tip “There Are No Final Decisions.” Their diagnosis fits operations as well as design: “The mistake lies in assuming that any decision is cast in stone—and in not preparing for the contingencies that might arise.”

Bezos wrote about decisions; the principle applies his terms to changes. A change is a two-way door when the previous production version can run safely against the current state of every persistent store it uses. Most routine changes should be two-way doors. A one-way door cannot be undone, or should not be:

One-way doors are legitimate, but they need different safeguards. Check more before the change: more testing, a staged rollout, an explicit sign-off. Prepare containment to limit the damage, and a plan to fix forward or compensate. And record why the irreversibility is intentional. The dangerous kind is the accidental one-way door: a change that everyone believed was reversible, until the incident in which they tried to reverse it.

Bezos stressed one mistake: putting reversible decisions through the slow process, which he said makes a company slow, needlessly cautious, and less inventive. He called the opposite mistake, the light process for a one-way decision, “less interesting,” because companies that habitually make it “go extinct before they get large.” An accidental one-way door is that mistake, made one change at a time. So classification pays in both directions. A change you have made reversible earns a light process and a fast release; a change you cannot reverse earns the slow, careful one.

In practice, classification is one line in the change’s description, written before the review: “two-way door,” or “one-way door, because…” That line opens the statement of risk and rollback that my article on code review asks for.

An Accidental One-Way Door

A classic accidental one-way door is a database migration that the previous release cannot survive. Here is a small one. The samples run on Node.js 24 and its built-in SQLite module, so they need no packages. Release 1 keeps email addresses in this table:

CREATE TABLE users (
  id INTEGER PRIMARY KEY,
  email TEXT
);

To keep the samples short, each release is reduced to the three queries it runs against the table. A real test would call the release’s own data-access code.

// The queries each release runs.
export const release = (db, col) => ({
  add: (value) => db.prepare(
    `INSERT INTO users (${col})
     VALUES (?)`).run(value)
    .lastInsertRowid,
  change: (id, value) => db.prepare(
    `UPDATE users SET ${col} = ?
     WHERE id = ?`).run(value, id),
  read: (id) => db.prepare(
    `SELECT ${col} AS value
     FROM users WHERE id = ?`)
    .get(id).value,
});

Release 2 renames the column to contact_email. Then release 2 misbehaves in production, and the team rolls back to release 1:

import { readFileSync } from 'node:fs';
import { DatabaseSync } from 'node:sqlite';
import { release } from './release.mjs';

const db = new DatabaseSync(':memory:');
db.exec(readFileSync('v1-schema.sql', 'utf8'));
const v1 = release(db, 'email');
const id = v1.add('ada@example.com');

// Release 2 ships this migration.
db.exec(
  'ALTER TABLE users RENAME COLUMN email TO contact_email');

// Release 2 misbehaves, so
// release 1 comes back.
try {
  v1.read(id);
} catch (error) {
  console.log(error.message);
}
// Prints: no such column: email

Rolling back the code did not roll back the schema. Release 1 now queries a column that no longer exists, so the rollback turns one incident into two. The trouble starts earlier, too: during a rolling deployment, every release 1 instance still serving traffic fails from the moment the migration runs.

A down migration that renames the column back keeps the data, but it only moves the failure. Every release 2 instance still serving breaks instead, unless you stop them all first and accept the downtime. And a rename is the gentle case: a migration that drops a column takes the data with it.

The fix is to split the change so that every state in between is a two-way door. Danilo Sato’s article “Parallel Change,” published on Martin Fowler’s site in 2014, describes the pattern, which is also called expand and contract: a change that would break existing users goes out in three phases, named expand, migrate, and contract. For a column rename, the expand step adds the new column, keeps both columns in step while either release may be running, and copies the data last:

-- Expand: add the new column.
ALTER TABLE users
  ADD COLUMN contact_email TEXT;

-- Keep the columns in step for
-- writes from either release.
CREATE TRIGGER users_sync_insert
AFTER INSERT ON users
WHEN NEW.email
  IS NOT NEW.contact_email
BEGIN
  UPDATE users SET
    email = coalesce(
      NEW.email,
      NEW.contact_email),
    contact_email = coalesce(
      NEW.contact_email,
      NEW.email)
  WHERE id = NEW.id;
END;

CREATE TRIGGER users_sync_old
AFTER UPDATE OF email ON users
WHEN NEW.email
  IS NOT NEW.contact_email
BEGIN
  UPDATE users
  SET contact_email = NEW.email
  WHERE id = NEW.id;
END;

CREATE TRIGGER users_sync_new
AFTER UPDATE OF contact_email
  ON users
WHEN NEW.contact_email
  IS NOT NEW.email
BEGIN
  UPDATE users
  SET email = NEW.contact_email
  WHERE id = NEW.id;
END;

-- Copy the data last, so no
-- release 1 write slips past.
UPDATE users
  SET contact_email = email;

The triggers follow the Rename Column refactoring in Scott Ambler and Pramod Sadalage’s Refactoring Databases. Its published excerpt calls for a synchronization trigger during the transition period. Because the copy runs after the triggers exist, a release 1 write during the migration is caught by a trigger or by the copy. On a large table, copy in batches and commit each one, so that no single transaction blocks release 1’s writes for long. The WHEN guards keep the triggers from firing each other in a loop, even with SQLite’s recursive triggers turned on.

Unlike SQLite, MySQL does not let a trigger update the table whose write fired it; a BEFORE trigger sets the other column on the row being written instead. For PostgreSQL, the open-source tool pgroll automates expand and contract. It serves the old and new schema versions side by side through views, which is all a rename needs. For other breaking changes to a column, it adds a new column and keeps the two in step with triggers.

Rolling out release 2 starts the migrate step, which lasts as long as any code that uses email might run. Release 2 reads and writes only contact_email; release 1 keeps using email. Whether each can run while the other writes is a question with a test:

import { test } from 'node:test';
import assert from 'node:assert/strict';
import { readFileSync } from 'node:fs';
import { DatabaseSync } from 'node:sqlite';
import { release } from './release.mjs';

// The old schema and one row,
// then the migration under test.
function expandedDatabase() {
  const db = new DatabaseSync(':memory:');
  db.exec(readFileSync('v1-schema.sql', 'utf8'));
  db.exec("INSERT INTO users (email) VALUES ('ada@example.com')");
  db.exec(readFileSync('expand.sql', 'utf8'));
  return db;
}

test('existing rows read the same in both releases', () => {
  const db = expandedDatabase();
  assert.equal(release(db, 'email').read(1), 'ada@example.com');
  assert.equal(release(db, 'contact_email').read(1), 'ada@example.com');
});

test('each release sees what the other one wrote', () => {
  const db = expandedDatabase();
  const v1 = release(db, 'email');
  const v2 = release(db, 'contact_email');

  const a = v1.add('a@example.com');
  assert.equal(v2.read(a), 'a@example.com');
  v2.change(a, 'b@example.com');
  assert.equal(v1.read(a), 'b@example.com');

  const c = v2.add('c@example.com');
  assert.equal(v1.read(c), 'c@example.com');
  v1.change(c, 'd@example.com');
  assert.equal(v2.read(c), 'd@example.com');
});

Run it with node --test, and both tests pass. Remove the three triggers, and the second test fails at its first check without an error from either release: release 2 reads nothing for the user that release 1 added. Delete that check, and the next one fails too: release 1 returns the old address of a user whose email release 2 changed. A rollback that restores service and quietly serves stale data is worse than one that fails loudly.

The contract step comes last, and it is a one-way door by design. Run it only after release 1 has left your rollback window and nothing else, such as a report, a job, or another service, still uses email:

-- Contract: only once no release
-- that uses email can come back.
DROP TRIGGER users_sync_insert;
DROP TRIGGER users_sync_old;
DROP TRIGGER users_sync_new;
ALTER TABLE users DROP COLUMN email;

Expand and contract does not abolish the one-way door. It moves the door to a moment when nobody needs the way back, and it makes the step explicit. But the contract step must actually happen: Sato warns that if it never runs, “you might end up in a worse state than you started.”

Four Ways Back

The principle recognizes four recovery strategies, usually considered in this order. Which ones apply depends on where the change’s effects went.

Containment: Stop the Bleeding

Containment limits the damage without undoing the change. Turn off the flag, shed load, pause the queues, stop writes to the data being damaged, halt the deployment pipeline, or move traffic away from the broken part. It is often the right first move, before anyone understands the failure. The troubleshooting chapter of the SRE book agrees: “Stopping the bleeding should be your first priority.” The same chapter goes further: when a bug may be corrupting data beyond repair, freezing the system may be better than letting it run.

Containment has to exist before the incident, as Google’s red-button did. After the outage, Google committed to putting every change to a critical binary behind a feature flag, off by default. Pete Hodgson’s article on feature toggles explains why such a switch must not need a release: its purpose is “to allow operators to quickly react to production issues.”

Rollback: Return to a Known-Good State

A rollback restores the previous version. It works when the change’s effects stayed inside your system and the previous version can run against the current data, which is what the expand step is for. For a two-way door it is the preferred recovery. It need not be faster than the original deployment, but it must be bounded and predictable: it finishes within a known time, at low risk.

Two facts about rollback are easy to forget. First, a rollback is a new change, made under pressure. AWS CodeDeploy rolls back by redeploying an earlier revision as a new deployment, and its documentation warns that it does not revert anything that scripts did in earlier deployments. In Kubernetes, rolling back a Deployment restores “only the Deployment’s Pod template part,” according to the Kubernetes documentation, so a ConfigMap that the failed release edited in place stays edited.

Second, the old version comes back with everything it contains. My article on deleting dead code tells how Knight Capital’s rollback in 2012 made an incident worse, because the version it restored still carried dormant code.

Restoring a database from a snapshot counts as a rollback only with a warning. It discards every legitimate write made since the snapshot, and it brings back whatever was deleted since, GDPR erasures included. Treat it as a decision about how much data to lose, not as an undo.

Compensation: Counterbalance What Already Happened

Once an effect has left your system, you cannot roll it back. You can only add a new effect that counterbalances it: a refund for a charge, a correction for an email, a cancellation event for a partner.

Hector Garcia-Molina and Kenneth Salem built their 1987 paper, “Sagas,” around compensation, and they were exact about its limits. A compensating transaction undoes a step “from a semantic point of view,” which need not restore the state before the step. And a compensation does nothing about the transactions that saw the step’s results before it ran. By the time a correction goes out, a customer may have read the wrong email, and a partner may have shipped the order.

Compensation needs machinery that exists before the incident:

Stripe’s refund documentation draws the line: canceling a payment before it completes costs nothing, but a refund after it succeeds might incur a fee. A refund can also fail if the customer’s bank cannot process it. So delay the one-way effect. Stripe lets you authorize a payment now and capture the funds later, as a hotel authorizes the full amount before the guest arrives and captures it at checkout. Until the capture, the way back is a cancellation, not a refund. The wait has a limit: Stripe says an online card authorization usually lasts 7 days.

Forward-Fix: Repair by Moving Ahead

Sometimes going back is worse than going forward: a flawed security fix, whose rollback would reopen the hole; an irreversible data migration; a rollback that would lose data written since the release. The recovery is then a new release that fixes the problem, and it is only as good as your pipeline.

If a deployment takes hours, forward-fix is not a recovery strategy; it is a hope. DORA, a research program run by Google Cloud, measures how quickly teams recover from a failed deployment, whichever way they do it. Since 2023 it has called that metric failed deployment recovery time.

The principle is satisfied when at least one appropriate strategy is proven and can be carried out within the recovery objectives. A one-way door does not need a rollback plan. It needs containment and a forward-fix or compensation plan, and both must have been exercised.

Proven Means Exercised

On 31 January 2017, GitLab.com went down after an engineer accidentally removed data from the primary database server. GitLab’s postmortem walks through the recovery procedures it had in place. The pg_dump backups were not there: the job had been running pg_dump 9.2 against a PostgreSQL 9.6 database, and failing. The emails that reported the failures were rejected by the receiving mail server. Azure disk snapshots were not enabled for the database servers, and the replica had been wiped as well.

GitLab restored from an LVM snapshot that an engineer had happened to take about 6 hours earlier, to refresh a staging environment, and lost the database changes made since. Asked why the backup procedure was not tested on a regular basis, the postmortem answers: “Because there was no ownership, as a result nobody was responsible for testing this procedure.”

A recovery path is proven only for the cases you have exercised. On 5 December 2025, Cloudflare used a killswitch to turn off an internal testing tool. According to its postmortem, the killswitch had mitigated incidents before, and the team followed the well-defined procedure for it. But Cloudflare had never before applied a killswitch to a rule whose action was execute.

Applied to that rule, the killswitch hit a bug in the older FL1 proxy, and the change reached the whole network within seconds. For about 25 minutes, sites that FL1 served with the Cloudflare Managed Ruleset answered requests with errors. Those sites accounted for about 28 percent of Cloudflare’s HTTP traffic.

Exercising the way back has to be routine, or it will not happen. Martin Fowler notes that blue-green deployment “allows you to test your disaster-recovery procedure on every release.” Other paths need drills, which turn a written procedure into a capability. Roll back in a staging environment with production-like data, restore a backup and compare it with the source, and run a compensation workflow end to end.

A drill at production scale can also show what the recovery itself costs, such as the herd effect that slowed Google’s recovery in us-central1. And give each recovery path an owner: GitLab’s lesson is that a path nobody owns is a path nobody tests.

Measurable Means Three Numbers

Disaster-recovery planning has two standard objectives, which NIST’s glossary defines from SP 800-34 Rev. 1. The recovery time objective (RTO) is how long a system’s components can stay in recovery “before negatively impacting the organization’s mission or mission/business processes.” The recovery point objective (RPO) is “the point in time to which data must be recovered after an outage.” The principle requires both for each significant change, and it adds a third: a correctness objective, the time by which the data must be right again.

Without the third, a team can satisfy the letter of recoverability by restoring uptime while leaving silent data corruption behind. For the column rename, the three might read: restore service within 15 minutes by redeploying release 1; lose no committed writes; and, by the end of the rollback, find no row whose two columns disagree. That last one is a query, which returns 0 with the triggers in place. Without them, the second test’s four writes, run without its checks, leave two of the three rows mismatched.

SELECT count(*) AS mismatched
FROM users
WHERE email IS NOT contact_email;

The business sets the numbers. Engineering makes them testable, and the change’s description records them.

Recovery Starts with Detection

Recovery without fast detection means recovering late, after the damage has spread. Three things shorten that delay:

A fixed threshold beats a feeling, but it compares against the wrong thing. It misses a regression that stays under the line, and it fires when a shared dependency fails both versions alike. Spinnaker’s canary guide warns against comparing a canary with the current production deployment: “Instead always compare the canary against an equivalent baseline, deployed at the same time.” The baseline runs the production version but otherwise matches the canary. Here is that rule at its simplest:

// Judge the new version against a
// baseline running the old version
// at the same time.
export function shouldRollBack(
  canary, baseline, rule
) {
  // Too few requests: no verdict.
  const few = (s) =>
    s.requests < rule.minRequests;
  if (few(canary) || few(baseline)) {
    return false;
  }
  const rate = (s) =>
    s.errors / s.requests;
  const extra =
    rate(canary) - rate(baseline);
  return extra > rule.maxExtraErrorRate;
}

const rule = {
  minRequests: 1000,
  maxExtraErrorRate: 0.005,
};

// 0.8% against 0.05%: a regression
// that a fixed 1% limit misses.
console.log(shouldRollBack(
  { requests: 4000, errors: 32 },
  { requests: 4000, errors: 2 },
  rule));
// Prints: true

// 4.0% against 3.9%: a dependency
// fails both versions alike.
console.log(shouldRollBack(
  { requests: 4000, errors: 160 },
  { requests: 4000, errors: 156 },
  rule));
// Prints: false

The function leaves statistics out on purpose, and that has a price. When both versions share a 4 percent error rate, chance alone makes it call for a rollback about one time in eight at 4,000 requests each, and more often below that. Requiring three non-overlapping windows in a row cuts false alarms to about 1 in 500 per check, but a rollout makes many checks. And a real regression then takes at least three windows to catch.

A production judge adds the statistics: Spinnaker’s default judge uses the Mann-Whitney U test and flags a metric only when it is 98 percent confident that the difference is real. Argo Rollouts, a Kubernetes controller, can run such an analysis during a canary and abort the rollout when it fails, cutting the canary’s traffic back to zero. Kubernetes itself only reports a stalled Deployment and leaves any rollback to higher-level tools. Run the decision in one place, such as the rollout controller, rather than inside every request, and keep a human override.

Recovery without a Meeting

If recovery needs a coordination bottleneck, it will be slow exactly when it matters most. The test is simple: can recovery proceed without waiting for people to wake up, join calls, or reach consensus?

That does not mean removing every control. Pre-authorize the recovery procedures, so the engineer on call can run them without asking. Keep the guardrails that earn their cost: two-person review for destructive actions, an audit trail for break-glass access, and a limit on how much an automated rollback may touch at once.

Reversibility Debt

Like technical debt, reversibility debt accumulates silently: migrations without forward-compatible schemas, feature flags that became permanent forks, API versions nobody tracks, and compensation workflows nobody has run. Each is a recovery path that exists only in theory.

Pay it down: track it, audit recovery paths on a schedule, and treat an unproven path as an incident worth reviewing before an outage reviews it for you. For flags, Hodgson describes teams that add a removal task to the backlog the day a release toggle is introduced. Others give toggles expiration dates, and some plant “time bombs” that fail a test once a flag outlives its date.

The Plan in the Pull Request

All of this fits in the change’s description. Here is the plan for the column rename:

Change: rename users.email
  to contact_email
Door: two-way until contract
Way back: redeploy release 1
Proven by: mixed-version test,
  rollback drill in staging
Contain: halt the rollout
Trigger: errors 0.5 points over
  baseline in 3 non-overlapping
  windows in a row, each with
  1,000+ requests per version
RTO: 15 minutes
RPO: no committed writes lost
Correct: 0 mismatched rows
  by the end of the rollback
Contract: once release 1 is
  past the rollback window and
  nothing else uses email
Owner: team that owns users
Run by: on-call, no approval

Each line answers a question that an incident would otherwise ask at the worst possible time.

Checklist

Design for recovery, not just success:

Further Reading