On the evening of May 7, cooling failed in one data hall of one availability zone in us-east-1, racks lost power, and AWS's blast radius did what the documentation promises: the damage stayed inside use1-az4. Every other zone kept serving. The companies that went down that night went down inside their own architecture, and we know that with unusual precision because Coinbase published a postmortem on Monday that says so in its own words. Trading was, in their phrasing, "unavailable or degraded for roughly eight hours, with full recovery of all systems taking another twelve", during an event in which the platform's zone boundary worked.

This is Part 1 of Nothing Fails Alone, a series about the gap between the availability you buy from a cloud provider and the availability you actually build on top of it. Each part takes one real incident, extracts the architectural claim it tests, and ships a small lab you can run against your own account. The labs live in a companion repo, nothing-fails-alone on GitHub, and every one of them is read-only.

What actually happened in use1-az4

The first incident report on the AWS Health Dashboard was timestamped 5:25 PM PDT on May 7, and since the resolved event has since dropped off the dashboard, the wording below is as quoted by The Register at the time: problems in the use1-az4 availability zone of us-east-1. A follow-up update stated the mechanism plainly: "EC2 instances and EBS volumes hosted on impacted hardware are affected by the loss of power during the thermal event." By 6:47 PM PDT, AWS said it was working to bring temperatures back to normal levels and warned that other services depending on the affected EC2 instances and EBS volumes in that zone could also be impaired.

Coinbase's postmortem fills in the physical cause: multiple chiller units failed at the same time in a single data hall, and the cooling loss triggered a thermal-safety shutdown of the affected racks. That is what "loss of power during the thermal event" means in practice. The racks were not destroyed; they were deliberately powered down because running compute without cooling destroys it. EC2 instances and EBS volumes on those racks went offline together, in one building, in one zone.

Recovery was gradual and honest about being gradual. At 10:11 PM PDT that night, AWS reported additional cooling capacity online and some racks recovered, with more to be recovered "in a controlled and safe manner". Cooling capacity was back to pre-incident levels at 1:50 PM PDT on May 8, roughly twenty hours after the first report, with instance recovery trailing the cooling restoration. If your mental model of an AZ event is a fifteen-minute blip, recalibrate: a thermal event is measured in hours because physics is, and racks come back rack by rack, not all at once.

What did not happen

The other availability zones in us-east-1 stayed healthy. That is the part of this incident that should get more attention than it does, because it is the part AWS actually promised. Availability zones are physically separate facilities with independent power and cooling, and on May 7 that independence was tested by a real facility failure and held. The failure domain was one data hall; the blast radius stayed inside one zone.

There was exactly one honest caveat, and it is worth quoting because it matters later in this series. AWS shifted traffic away from the stricken zone, advised customers to move workloads to the other us-east-1 zones, and then admitted: "Customers may experience longer than usual provisioning times." Of course they may. When a zone dies, every correctly built multi-AZ architecture in that region starts replacing capacity in the surviving zones at the same time. The zone boundary held, but the surviving zones are a shared pool, and mass failover is a thundering herd against that pool. Keep that caveat in mind; Part 3 of this series is about what your recovery plan shares with everyone else's recovery plan.

Coinbase's own words

Postmortems from public companies are usually sanded down to nothing. This one is not, which is why it is worth reading in full. Two sentences carry the whole architecture story. First: "Our matching engine was pinned to a single building." Second, the explanation of how: the Coinbase Exchange matching engine "runs as a Raft-based replicated cluster inside an AWS Cluster Placement Group". And then the sentence that turns a zone event into an eight-hour outage: "There was no automated cross-zone failover."

A cluster placement group is single-AZ by construction. That is not a limitation AWS hides; it is the product. The point of the cluster strategy is to pack instances onto hardware that is physically close together, on the same high-bandwidth network segment, so that node-to-node latency is as low as EC2 can make it. You cannot have that and zone separation at the same time, because zone separation is distance and distance is latency. For a matching engine, where microseconds of node-to-node latency are the product, a cluster placement group is the correct choice, and Coinbase says as much: "We make this choice deliberately."

The problem is what lived inside it. Raft exists to survive the loss of a minority of nodes. Put five Raft nodes in a cluster placement group and you have built a consensus protocol whose members share a building, which means they share a failure domain, which means the protocol can no longer do the one thing it exists to do. When AWS terminated EC2 instances inside Coinbase's placement group at 9:29 PM ET, three of five matching-engine nodes went down together and quorum was gone. A quorum that shares a data hall is not a quorum; it is one failure with five names.

With no automated cross-zone failover behind it, recovery became a people process. It required an emergency code change, shipped during the incident, to remove a startup assumption that all five cluster nodes were resolvable. The matching engine came back in cancel-only mode at 2:25 AM ET on May 8, full trading resumed at 3:49 AM ET, the retail site and mobile app were fully recovered by 9:53 AM ET, and the backlog of event-streaming topics cleared by 2:00 PM ET. Roughly eight hours down, another twelve to full recovery, during an event that AWS contained to one zone.

Multi-AZ is something you buy and something you build

The general shape of this incident is not a Coinbase story. It is the gap between purchasing multi-AZ infrastructure and building a multi-AZ system, and that gap has a small number of recurring shapes. Each of these is a configuration that looks redundant on the architecture diagram and is not:

  • The single-subnet Auto Scaling group. The ASG will happily replace failed instances forever, into the same zone, because the subnet list you gave it resolves to one AZ. When that zone degrades, desired capacity cannot be met until a human edits the group. Self-healing that heals into the failure is not self-healing.
  • RDS without Multi-AZ. No synchronous standby means zone loss is not a failover, it is a restore: from a snapshot, measured in hours, with data loss up to the last backup point. The DR plan next to it usually assumes sub-minute failover, because someone read the Multi-AZ documentation and nobody checked the console flag.
  • The quorum in a cluster placement group. The Coinbase pattern: three, five, or seven nodes of Raft, etcd, or ZooKeeper packed into a group that is single-AZ by definition. The consensus layer reports itself as healthy and fault-tolerant right up until the building disagrees.
  • The cross-AZ NAT dependency. A private subnet in zone B whose default route points at a NAT gateway in zone A. Zone A's failure takes out the outbound traffic of instances whose own zone is perfectly healthy. This one is invisible on every diagram because the arrow points at "NAT", not at a zone.

None of these are exotic. All of them pass a load test, a game day that only kills instances, and most architecture reviews, because each one fails only when a whole zone does, and whole zones fail rarely enough that the assumption goes untested for years.

The lab: a single-AZ concentration detector

Claims should be testable, so this part ships with a lab: 01-single-az-detector, a single Python file that scans the current account and region for the patterns above and three more. It is read-only in the strict sense: every API call it makes is a Describe*. It runs seven checks: single-AZ Auto Scaling groups, cluster placement groups, quorum-sized clusters inside placement groups, single-AZ RDS instances, RDS clusters whose members all sit in one zone, ElastiCache replication groups without Multi-AZ, and private subnets whose default route crosses zones to reach a NAT gateway.

$ python3 detect.py
SEVERITY | RESOURCE                      | AZ                    | WHY
--------------------------------------------------------------------------...
CRITICAL | placement-group/matching-prod | us-east-1a (use1-az4) | 3 instances in cluster placement group 'matching-prod': a quorum-sized cluster in a group that is single-AZ by construction. ...
HIGH     | asg/api-workers               | us-east-1c (use1-az2) | Auto Scaling group spans exactly one AZ. Every replacement instance lands in the same zone ...
HIGH     | rds-cluster/orders            | us-east-1a (use1-az4) | All members of DB cluster 'orders' sit in one AZ. Aurora failover needs a reader in a different zone ...
MEDIUM   | subnet-0a1b2c3d -> nat-0e4f5a6b | us-east-1b (use1-az6) -> us-east-1a (use1-az4) | Private subnet in us-east-1b (use1-az6) routes 0.0.0.0/0 through a NAT gateway in us-east-1a (use1-az4) ...

4 finding(s).

The check that earns the CRITICAL severity is QUORUM_IN_ONE_AZ: a cluster placement group holding exactly 3, 5, or 7 running instances. Those counts are not arbitrary; they are the shape of a Raft, etcd, or ZooKeeper quorum, and a quorum-shaped workload concentrated in one zone is the exact configuration that took Coinbase down. A placement group with some other instance count is still flagged, but as HIGH: it is a latency optimization that someone should confirm was chosen on purpose.

Two behaviors are deliberate. First, the exit code: the scan exits 0 whenever it completes, findings or not, because this is a report, not a gate. If IAM denies one of the APIs, that check is skipped with a one-line warning on stderr, the remaining checks still run, and the exit code stays 0; you get exit code 2 only when the scan cannot run at all, no credentials, no region, no boto3. Check stderr before trusting a clean table. Second, every finding reports the zone as both name and ID, for example us-east-1a (use1-az4). The name is an alias that AWS randomizes per account; the ID is the physical zone. The Health Dashboard said use1-az4, and without the ID you cannot tell whether that was your us-east-1a. That distinction looks like trivia today. Part 2 is about the incident where it stops being trivia.

Failure modes and what to watch

The trade-off underneath all of this is legitimate, so name it instead of pretending it away. Single-AZ placement buys latency, cross-AZ data transfer costs money, and synchronous replication across zones costs write latency. Coinbase did not stumble into a placement group; they engineered into it and accepted a risk. The failure was not the choice, it was that the choice's consequence, no automated path out of the zone, was discovered at 9:29 PM during the incident rather than decided in a design review before it.

What to watch for in your own account, beyond running the detector: quorum arithmetic, not just quorum location, because two of three etcd nodes in one zone is the same defect with a friendlier table row. ASGs that list several subnets which all resolve to the same zone, which is why the detector resolves VPCZoneIdentifier subnets instead of trusting the AZ list. Caches without replicas: a zone loss takes ElastiCache cold, and the resulting miss storm lands on your database at the worst possible moment. And provisioning capacity during a real zone event, which is the one thing the detector cannot see, because it is a property of everyone else's architecture, not yours. AWS told you so itself, in the middle of the incident: longer than usual provisioning times. The zone boundary is real, and it held. What you build on top of it is yours.

Read this next

For consulting on AWS, cloud architecture, resilience reviews, and platform work, start at ercanermis.com.

References