Multi-AZ is not a blast radius argument

“We are multi-AZ” is offered as the answer to a resilience question more often than any other sentence in infrastructure, and it answers a narrower question than the person asking usually has in mind. Availability zones protect against one specific class of fault: a facility becoming unavailable. That is a real fault and it does happen. It is also not what took your platform down last time.

The distinction that matters is between independent failure and correlated failure. Zones give you independence in power, cooling, and network fabric. They give you nothing at all in anything the zones share, and in a normal architecture almost everything interesting is shared.

The shared-fate inventory

Three copies of a service in three zones is three copies of the same artefact, deployed by the same pipeline, reading the same configuration, resolving the same DNS names, calling the same authentication service, and holding the same certificate. Every one of those is a single failure domain wearing a distributed costume.

Shared thing	What happens when it is wrong	Does multi-AZ help
Deploy pipeline	The bad build reaches all three zones in the same minute	No. It makes the rollout faster.
Configuration or feature flag store	One flag flip changes behaviour everywhere at once	No
DNS zone	Nothing resolves, in every zone, including your health checks	No
Authentication service	Every request fails authorisation, so every service looks broken	No
Certificate	Expiry takes all zones down simultaneously and on a schedule	No
Infrastructure state file	You cannot change anything while you most need to	No
Facility power, cooling, fabric	One zone stops serving	Yes. This is the case it was built for.

Read that table as a checklist rather than as an argument. The exercise that follows from it is concrete: draw the dependency graph for one user action, signing in, and mark every node that exists exactly once. Count them. That number is your real blast radius, and it is usually between four and nine in a system described internally as highly available.

Redundancy in the thing you copied is not resilience against the thing you shared.

Control plane and data plane fail differently

The most useful frame for this comes from how the providers themselves are built. The data plane is the part that serves requests. The control plane is the part that changes things: launching instances, updating routes, issuing credentials, registering targets.

Control planes are more complex, have more dependencies, and fail more often. They also fail exactly when you want them most, because a large event drives everyone in the region to launch capacity at the same moment. Any recovery plan whose first step is “scale up” has a dependency on the component least likely to be working.

The design property that follows is called static stability: the system keeps working correctly using state it already has, without needing anything to change. Concretely that means pre-provisioning the capacity you would have scaled into, and it means every client of a configuration or credential service holding a last-known-good copy rather than failing when the lookup fails.

# A config client that fails closed is a config client that turns a
# dependency outage into your outage. Serve stale, loudly.

class Config:
    def __init__(self, source, max_stale=timedelta(hours=6)):
        self.source = source
        self.cache = None          # last value that parsed successfully
        self.fetched_at = None
        self.max_stale = max_stale

    def get(self):
        try:
            value = self.source.fetch()
            self.cache, self.fetched_at = value, now()
            metrics.gauge("config.stale_seconds", 0)
            return value
        except Exception as e:
            if self.cache is None:
                raise                      # nothing to fall back to, fail
            age = now() - self.fetched_at
            metrics.gauge("config.stale_seconds", age.total_seconds())
            log.warning("serving stale config", age=age, error=e)
            if age > self.max_stale:
                alert.page("config stale beyond tolerance", age=age)
            return self.cache              # keep serving

Two things about that shape. The staleness is a metric, not a hidden state, so “we are running on cached config” is visible on a dashboard rather than discovered during the postmortem. And there is a tolerance beyond which stale is worse than absent, which is a decision someone has to make per config value rather than a global default.

Correlated load is the second half of the problem

Zones also fail to help against load that arrives everywhere at once, and your own system is usually the thing generating it.

Retries without a budget

Exponential backoff is necessary and not sufficient. Backoff spaces out one client’s attempts; it does nothing about a thousand clients backing off in lockstep and arriving together, which is why jitter matters as much as the exponent. And neither addresses the real failure: when a dependency is degraded, retries multiply the load on the thing that is already struggling, at exactly the wrong moment.

The fix is a retry budget: a cap on retries as a proportion of total requests, enforced across the client fleet rather than per call site. Something like “retries may not exceed ten percent of requests in a rolling window”. Past the cap, calls fail immediately. That converts a self-inflicted amplification into a bounded degradation.

Health checks that cause the outage

A health check that removes an overloaded instance from rotation shifts its load onto the remaining instances, which then also become overloaded and are also removed. The mechanism designed to protect availability drives the fleet to zero, and it does so faster the more aggressive the check is.

Two guards. First, a minimum healthy count below which the load balancer stops removing targets and sends traffic to everything, on the reasoning that a slow response beats no response. Second, distinguish “this instance is broken” from “this instance is busy”, because they need opposite responses and a single latency threshold cannot tell them apart. Shed load at the instance, with a fast rejection, rather than letting the queue grow until the check times out.

What to change on Monday

1. Stagger the pipeline. If a deploy reaches all zones at once, the zones are decoration. One zone, a real bake period, automated rollback on error rate, then the rest.
2. Treat config changes as deploys. Same staging, same rollback, same blast-radius thinking. A flag flip that reaches every host in one second is the fastest outage you can buy.
3. Give every external dependency a last-known-good path, and put its staleness on a dashboard.
4. Remove “scale up” from step one of every runbook. If capacity is the answer, it has to already exist.
5. Put a retry budget in the shared client library, because it will not be added consistently at each call site.
6. Test the correlated case, not the zone case. Killing one zone is the easy drill and the one everyone runs. Try instead: the config store returns errors, DNS is slow, the auth service adds two seconds. Those are the rehearsals that find something.

None of this is an argument against spreading across zones. It is an argument against letting that one property stand in for the whole question, because it answers the failure that is easiest to imagine and cheapest to survive, and leaves untouched the shared, single-instance, change-driven failures that produce most real incidents.