On March 1, an availability zone in AWS's me-central-1 region, mec1-az2, was hit by what AWS described as "objects that struck the data center, creating sparks and fire". About eighteen hours later, a second zone, mec1-az3, lost power too. The third, mec1-az1, stayed up throughout. With two of three zones impaired, S3 and DynamoDB began failing region-wide, and AWS's advice to customers, in its own words on the Health Dashboard, was to "enact their disaster recovery plans and recover from remote backups into alternate AWS Regions, ideally in Europe". Availability zone independence is engineered against power faults, cooling faults, and network faults. In March it met a cause it was never engineered against, and the cause did not care about the engineering.

This is Part 2 of Nothing Fails Alone. Part 1 examined the May us-east-1 incident and argued that the zone boundary held, and it did: a cooling failure in one data hall stayed inside one zone, exactly as designed, and the outage that followed was built by the customer. That argument was correct, and it answered a specific question: does AZ isolation contain the failure modes it is engineered against? The March event in me-central-1 asks a different question. It happened before Part 1 was written, and at the time it was easy to file as a wartime special case, an act-of-war footnote with no lessons for normal architecture. The May incident is what made the general shape visible: Part 1 was about what the zone boundary protects, and this part is about what it cannot, which is any cause that reaches more than one zone. Those causes exist in peacetime too. War just compressed the demonstration into two days.

What actually happened in me-central-1

The event opened on the AWS Health Dashboard at 4:51 AM PST on March 1 as an investigation into "issues with AWS services in the ME-CENTRAL-1 Region", quickly narrowed to a "localized power issue" in a single availability zone, mec1-az2. The dashboard log is unusually candid, and at 9:41 AM PST it explained why the power was out: "At around 4:30 AM PST, one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire. The fire department shut off power to the facility and generators as they worked to put out the fire." The next day AWS named the cause, in an update The Register covered the same day: drone strikes, part of the conflict in the Middle East. "In the UAE, two of our facilities were directly struck", and the strikes "caused structural damage, disrupted power delivery to our infrastructure, and in some cases required fire suppression activities that resulted in additional water damage".

Through the first day, the incident looked like a textbook single-zone event, and AWS said so in the same 9:41 AM update: "Customers who were running their applications redundantly across the AZs are not impacted by this event." Multi-AZ was, at that hour, the correct and sufficient answer, and an update less than an hour earlier had carried a caveat you have read before if you read Part 1: due to increased demand in the unaffected zones, "customers may experience longer than usual provisioning times". The same phrase AWS would publish during May's us-east-1 event, the incident Part 1 covered two weeks ago, even though on the calendar March came first. Mass failover into surviving zones is a thundering herd everywhere, in every region, in every incident.

Then, at 10:46 PM PST, roughly eighteen hours after the first strike: "a localized power issue has affected another Availability Zone in the ME-CENTRAL-1 Region (mec1-az3) ... At this point it is not possible to launch new instances in the region ... Other AWS Services, such as DynamoDB and S3 are also experiencing significant error rates and latencies." Two of three zones down, and the failure had climbed out of the zone layer entirely.

Eighteen hours is not simultaneous, and that is the point

The two zones did not fall together. They fell about eighteen hours apart, to the same cause. That distinction is the whole argument of this post, so it is worth being precise about. Zone independence is a probability claim: AWS engineers zones onto separate power, separate cooling, separate flood plains, and separate network paths so that the internal faults of one facility, a failed chiller, a failed transformer, a bad switch fabric, stay inside it and do not coincide with the internal faults of another. On that claim's own terms, March changed nothing. Neither zone took the other down. There was no shared transformer, no cascading dependency. The isolation engineering worked all the way through.

What failed was the assumption underneath the probability math: that zone failures are independent events. They are independent only when the cause originates inside a facility. A cause that originates outside the facilities, a storm, a grid collapse, a wildfire, an armed conflict, is sampled once and applied to every zone in its reach. The zones shared a metro area, a power grid, an airspace, and a war. Correlated failure does not mean two zones die in the same instant; it means one cause reaches both on its own schedule. The eighteen-hour gap is what correlation actually looks like on a dashboard: not a synchronized crash, but the same hand knocking twice.

This is why the wartime-special-case framing fails. Drones are exotic; shared metro infrastructure is not. Every multi-AZ region concentrates its zones within single-digit milliseconds of each other, which in practice means the same city-scale blast radius for weather, utilities, and politics. The March event is the extreme end of a family of causes that includes every event big enough to be named after the city it happened in. When InfoQ analyzed the incident later that month under the title "War in Iran Damages Multiple AWS Data Centers, Challenging Multi-AZ Assumptions", the community discussion it summarized had already moved past the drones to the general case: zones are a cluster of buildings in the same city, and the assumptions worth challenging are about everything that can reach a city.

When the region itself is the failure domain

The most instructive update in the whole log came at 2:53 AM PST on March 2, when AWS explained what two lost zones do to S3: "Amazon S3 is a regional service and designed to withstand the total loss of a single Availability Zone ... As the second AZ became impaired, S3 error rates increased. With two Availability Zones significantly impacted, customers are seeing high failure rates for data ingest and egress."

Read that sentence as an architecture document, because it is one. S3 and DynamoDB are the services you do not architect for zone failure, because AWS does it for you, replicating across zones so that one zone's loss is invisible. That design has a stated tolerance: one zone. The second zone exceeded it, and the services that every "resilient" architecture in the region leaned on, for state, for backups, for the coordination data of the failover itself, started refusing reads and writes region-wide. Your multi-AZ application did not just lose its instances; it lost the regional substrate it planned to recover onto. mec1-az1 was healthy, and it did not matter, because a healthy zone attached to an impaired regional control plane is a room with the lights on in a building with no plumbing. This is the part multi-AZ cannot answer. The regional services are built on the same three zones you are, and their tolerance for correlated zone loss is a number AWS picked, not a number you picked.

"Ideally in Europe"

By 6:22 AM PST on March 2, AWS's guidance had arrived at its bluntest sentence: "We recommend customers enact their disaster recovery plans and recover from remote backups into alternate AWS Regions, ideally in Europe."

That sentence deserves a slower read than it got in March. First, the provider whose region it is told customers to leave it. That is the ceiling of in-region redundancy, stated by the vendor: there exists a class of event for which the correct amount of me-central-1 in your recovery plan is none, and you find out which class you are in mid-incident, from a dashboard. Second, "ideally in Europe" is doing quiet, careful work. The nearest alternative, me-south-1 in Bahrain, is a short hop away and would be the reflexive DR choice on latency grounds, and a facility there had been affected by a nearby strike in the same campaign. AWS was telling customers, without spelling it out, that the correlated cause had a geography, and that a useful recovery region is one outside that geography, not merely outside the region ARN. Distance from your primary is not a compliance checkbox; it is a bet about how far your correlated causes travel. Pick your DR region by asking what could plausibly hit both, not what keeps replication latency pretty.

Third, the advice only works for customers who could follow it. "Recover from remote backups" presumes the remote backups exist, created before March 1, because the same update log shows S3 "data ingest and egress" failing region-wide during the event. Copying your data out of a region is a peacetime activity. By the time the provider is advising it, the window for starting has closed.

Several months

The event's last update, dated April 30, is one paragraph long and ends the story without ending it: the region "has suffered damage as a result of the conflict in the Middle East and is currently unable to reliably support customer applications", customers should "migrate all accessible resources to other Regions and restore inaccessible resources from remote backups as soon as possible", billing operations are suspended, and recovery "is expected to take several months".

Put that against a standard DR document. An RTO is written as a duration, and nearly every RTO I have reviewed silently assumes the recovery target is the place you already are: restore the snapshot, relaunch the fleet, same region, hours at worst. "Several months" does not stress that number, it deletes the axis it was written on. If the region is not coming back this quarter, your actual recovery time is however long a cold cross-region rebuild takes, including the parts nobody rehearsed: IAM and networking recreated from code you hopefully have, data restored from replicas you hopefully made, DNS cut over, capacity found in a destination region absorbing everyone else's exodus at once. For teams with cross-region replication already flowing, March was a bad week. For teams whose DR plan was in-region snapshots, it was a migration project announced by artillery.

The lab: which physical zones are you actually in?

Part 1 closed its lab section by noting that AWS incident reports name zones by ID, use1-az4, and that without the ID-to-name mapping you cannot tell whether that was your us-east-1a. March is the incident where that stops being trivia. The dashboard said mec1-az2, then mec1-az3. To know whether your workloads sat in the struck zones or the surviving one, you had to resolve those IDs against your account, mid-incident, because a zone name like me-central-1a is an account-scoped alias: AWS assigns letters to physical zones independently per account, so two accounts' me-central-1a are usually different physical zones. "We are spread across a, b and c" is a statement about your account's naming, not about physical placement.

The lab for this part, 02-az-id-truth, is deliberately the smallest one in the series: a single read-only script whose only API calls are DescribeAvailabilityZones and GetCallerIdentity, the latter just to label output. It prints the name-to-ID mapping for the current account and region. This is the real output for one of my accounts:

$ python3 az_ids.py --region us-east-1
Region: us-east-1

ZONE NAME  | ZONE ID  | STATE
---------------------------------
us-east-1a | use1-az1 | available
us-east-1b | use1-az2 | available
us-east-1c | use1-az4 | available
us-east-1d | use1-az6 | available
us-east-1e | use1-az3 | available
us-east-1f | use1-az5 | available
...

In this account, use1-az4, the zone in May's incident report, is us-east-1c. An engineer here who read "use1-az4" in May and glanced at a diagram labeled "us-east-1a" would have relaxed about the wrong zone and worried about a healthy one. The script also takes --compare <profile>, which reads a second account's mapping in the same region and prints the two side by side with a per-name verdict on whether they resolve to the same physical zone, then counts the mismatches; the same name resolving to different IDs across accounts is the normal case, not the exception. A shared runbook that says "evacuate us-east-1a" does different things in different accounts. Runbooks, placement decisions, and incident lookups belong in zone IDs. There is a --json flag for feeding the mapping into inventory tooling, and the exit code is 0 whenever the run completes, reserving 2 for runs that cannot happen at all: no boto3, no credentials, no region, an unknown profile, or the describe call itself failing.

Failure modes and what to watch

Be honest about the limits first: a customer cannot architect me-central-1 out of a war. Nothing in your Terraform prevents the correlated cause; what you control is whether it finds you with your data already elsewhere. The watchlist that follows from March is short. Enumerate the causes that reach all your zones at once: the shared metro, the shared grid, the shared weather, the shared jurisdiction, and treat the answer as your real region-loss probability, which is low but not the zero the zone math quietly assumed. Hold cross-region backups for anything you cannot regenerate, in a region chosen for being outside the correlated geography, and test the restore, because March's version of that test had a fire department in it. Write RTOs twice, once for in-region recovery and once assuming the region is gone for a quarter, and get the second number in front of whoever owns the risk. And keep an inventory of your physical zone IDs per account, because when the next report says a zone ID, the time to learn your mapping is not during the event. The zone boundary is real, and Part 1 showed it holding. March showed the thing above it: zones can be perfectly isolated from each other and still not be isolated from the world.

Read this next

For consulting on AWS, cloud architecture, resilience reviews, and platform work, start at ercanermis.com.

References