Every recovery plan carries a second assumption, quieter than the first. The first is that some of the infrastructure survives the event; Parts 1 and 2 of this series tested that one. The second is that when the event comes, you can still deploy the fix: the API answers, the console loads, your pipeline can assume a role, the Auto Scaling group can launch a replacement. On June 13th, 2023, a latent defect in the subsystem that manages Lambda's compute capacity degraded function invocations in us-east-1, and for part of the next four hours the AWS Management Console in that region served error pages while STS threw elevated error rates. Nothing in most customers' infrastructure broke that day. What broke was the layer they would have used to fix it. When the control plane is inside the blast radius, the remediation path is inside the failure domain.

This is Part 3 of Nothing Fails Alone. Part 1 took the May us-east-1 incident and showed the zone boundary holding while a customer architecture failed inside it. Part 2 took the March me-central-1 event and showed what the boundary is engineered against, facility faults, and what it is not, one cause reaching two zones. Both parts ended near the same caveat, published by AWS in the middle of both incidents: "longer than usual provisioning times". Part 1 promised that this part would be about what your recovery plan shares with everyone else's recovery plan. Here is the answer: it shares the control plane. Your failover calls the same regional APIs as every other failover in the region, operated by the provider, degraded by the same events that make you need it, and that dependency deserves the same scrutiny you give your replicas.

June 2023: the fix needed the thing that failed

The mechanism has been on the public record since AWS published its post-event summary for June 13th, 2023. It is worth reading precisely, because it is the cleanest documented case of a control plane failing while the infrastructure under it kept running. At 10:01 AM PDT, the Lambda Frontend fleet in us-east-1 began scaling for ordinary daily traffic. At 11:49 AM it crossed, in AWS's words, "a capacity threshold that had previously never been reached within a single cell", which "triggered a latent software defect". Execution environments were allocated but never used, the subsystem "responsible for managing the underlying compute capacity" could not provision replacements that worked, and Lambda invocations in the region started failing.

Then the dependency graph did its work. The summary lists Amazon STS, the AWS Management Console, Amazon EKS, Amazon Connect, and Amazon EventBridge as degraded "as a result of the degraded Lambda function invocations". STS returned elevated error rates from 11:49 AM to 2:10 PM PDT "with three distinct periods of impact". The console in us-east-1 served, from 11:48 AM to 2:02 PM, either an "AWS Management Console is currently unavailable" page or a "504 Time-out". Federated sign-in degraded too: "Existing IAM sessions were not impacted, but new sign-in federation via SAML was degraded." Full recovery came at 3:37 PM PDT, three hours and forty-eight minutes after impact began.

Map that onto an ordinary runbook. Step one, sign in to the console: degraded. Step two, let the pipeline assume a role for fresh credentials: STS, degraded, three separate times. Step three, invoke the remediation Lambda: that is the outage. None of your instances failed. Your database did not fail over. The event never touched your infrastructure at all, and it still took away the tools your plan assumed. The lesson is structural, not historical: the operating layer and the failing layer were the same layer, and they will be again.

October 2025: the meltdown, revisited from the control plane

When us-east-1 went down in October 2025, this site covered it the same day in When the Cloud Sneezes, the World Catches a Cold. That post was about the width of the blast radius, the absurdity of half the internet depending on one region, and it ended on what to change. This part is the follow-up it promised, written with the benefit of AWS's official post-event summary, and the summary rewards a colder reading than the one the day allowed: look at what happened to the tools, not the apps.

The trigger was DNS: "a latent race condition in the DynamoDB DNS management system that resulted in an incorrect empty DNS record" for dynamodb.us-east-1.amazonaws.com, a record the automation "failed to repair". DynamoDB's endpoint was restored within hours. The control-plane damage ran much longer. EC2, per the summary: "Between 11:48 PM PDT on October 19 and 1:50 PM PDT on October 20, customers experienced increased EC2 API error rates, latencies, and instance launch failures", fourteen hours during which the standard recovery move of every multi-AZ architecture in the region, launch replacement capacity, was itself impaired. STS returned "API errors and latency" from 11:51 PM to 9:59 AM. Console sign-in with an IAM user failed with "increased authentication failures" until 1:25 AM, Identity Center users in the region could not sign in at all, and root-credential and federated sign-in threw errors even in other regions, because the sign-in path itself lived in us-east-1.

Hold the two posts side by side and the division of labor is clear. The October post asked what it costs to depend on one region. This one asks the narrower, nastier question inside it: when the region had its bad day, why did well-built multi-AZ architectures stay down for hours? Because their recovery was a control-plane operation. Scaling out was the fix, and scaling out was what fourteen hours of impaired instance launches took away. Everyone's plan converged on the same degraded API at the same time, which is exactly what "your plan shares the control plane with everyone else's" means in practice.

February 2026: Azure, same shape, different logo

If this were an AWS quirk, it would be an AWS problem. It is not. On February 2nd, 2026, Microsoft logged incident FNJ8-VQZ, and its post-incident review opens: "Between 18:03 UTC on 02 February 2026 and approximately 00:30 UTC on 03 February 2026, a platform issue caused some customers to experience degraded performance and control plane failures, for multiple Azure services across multiple regions." A security remediation policy meant to disable anonymous access on Microsoft-managed storage accounts was, through "a data synchronization problem in the targeting logic", applied to accounts "intentionally configured to allow anonymous read access for platform functionality": the accounts serving VM extension packages. Virtual machines hit "failures when deploying or scaling", AKS hit "failures in node provisioning", and GitHub Actions jobs "queued and timed out while waiting to acquire a hosted runner".

Notice what did not happen: running VMs kept running. The data plane was fine. What died was the ability to create, scale, or change, across multiple regions at once, because the control plane is shared in a way the data plane is not. That is the property worth generalizing. A control plane is one logical system spanning the zones, sometimes the regions, that its data plane keeps carefully separate; it is the piece under maximum load precisely when everyone is recovering; and it fails in ways multi-region diagrams do not show, because the diagram draws your resources, not the machinery that mutates them. Two providers, four months apart, same shape.

Static stability, defined properly

The defense has a name: static stability. A statically stable system keeps meeting its requirements when a dependency fails, without having to change anything, because whatever it would have asked the dependency for is already in place. The definition earns the word "static": recovery that requires no mutation cannot be blocked by the layer that performs mutations. Concretely, that means a small family of precomputed decisions:

  • Pre-provisioned headroom. Each zone runs enough capacity that the surviving zones carry the full load with no launch. Zone loss becomes a load-balancer decision, a data-plane operation, instead of an Auto Scaling event, a control-plane operation.
  • Pre-created, not created-on-failover. The standby's DNS records, target group attachments, and standby instances exist before the incident. Failover flips health checks and weights; it does not run Terraform.
  • Pre-baked images. An AMI that boots to serving without reaching package mirrors, config services, or secrets endpoints keeps a launch useful even when everything around the launch is degraded.
  • Break-glass paths that avoid the failed layer. October's summary is blunt about sign-in: IAM users, Identity Center, and federation all degraded together. An emergency access path is only real if it does not traverse the region and identity plane you are trying to rescue, and only real if it is exercised; an untested break-glass credential is a wish.

The common thread is moving work from during-the-incident to before-the-incident, from the control plane's schedule to yours. You cannot make the provider's control plane reliable. You can arrange to not need it for the first hour.

The lab: measure the difference, do not assert it

The lab for this part, 03-static-stability, breaks a series rule knowingly: it is not read-only. Terraform builds a VPC, an internet-facing ALB, and six t3.micro instances, and it costs money until you destroy it, so it belongs in a sandbox account and in the trash the same day. What it buys for that price is a measurement of the exact gap this post describes, on your account, instead of my assertion of it. AWS's own fault-injection documentation states the premise plainly: for the AZ power-interruption scenario, "EC2 API calls made by the Auto Scaling control plane to recover lost capacity in the AZ will fail". The lab builds both answers to that sentence behind one load balancer. Path A is the standard design: an Auto Scaling group with one instance per zone and a scaling policy, so losing a zone means the ASG must call ec2:RunInstances to recover. Path B is the statically stable design: the same workload pre-provisioned at double capacity, two instances per zone, so either zone alone already carries the full load.

The measurement script deserves a note, because its first draft was wrong in an instructive way. Simply removing the failed zone's subnet from an ASG is not zone loss: it triggers rebalancing, and rebalancing launches replacements before terminating anything, so healthy capacity never dips and there is nothing to measure. Real zone loss kills the instances. So measure.py removes the subnet and then terminates that zone's in-service instances via TerminateInstanceInAutoScalingGroup with ShouldDecrementDesiredCapacity=False, leaving each group down real capacity it must recover elsewhere. For path A it then times the interval from fault to full healthy count at the load balancer, an interval that is pure control plane: launch call, boot, health checks. For path B it watches the healthy count through a settle window and records the minimum, which drops from four to two; two is exactly the post-failure requirement, so the pass condition is tested against a real dip rather than holding trivially. This is the shape of the table the script prints, not a result I obtained; I have not run this against a live account for this post, and the numbers a reader gets are theirs:

PATH                    | TIME TO RECOVER               | API CALLS REQUIRED                | SURVIVES CONTROL PLANE LOSS
path-a (control plane)  | measured on your run          | yes: ec2:RunInstances via the ASG | no
path-b (static)         | 0s, capacity already in place | none                              | yes

Whatever number path A prints for you, read it as a best case: it was measured against a healthy, cooperative control plane on a quiet day. October's control plane was neither. Lab 04 exists to close that gap, by injecting the insufficient-capacity errors that turn path A's number into what it would have been that night.

The honest cost, and where it stops being worth it

Path B runs double the steady-state capacity to serve the same traffic. That is not a footnote, it is the deal: static stability is buying insurance with idle compute, every hour, against an event measured in hours per year. With two zones the premium is 100 percent; with three zones each sized to absorb a lost one, it drops to 50; with more zones or an overprovisioning-aware sizing, less. Pay it for the serving path whose downtime is measured in revenue or safety, for small fleets where doubling is cheap in absolute terms, and for anything whose recovery would otherwise depend on the control plane working mid-incident.

Stop paying it where the math stops working. Batch and asynchronous work should queue through a zone loss, not pre-provision against it. Dev and staging do not need it. Very large fleets pay real money for headroom, and there the honest move is to split the tier: static stability for the critical slice, control-plane recovery accepted, eyes open, for the rest. And know what the technique does not cover. It would not have saved Coinbase in Part 1, because their failure was in their own data plane, a quorum sharing a building. It does not answer Part 2's region loss, since headroom inside a doomed region is doomed headroom. It protects the first hours, on the bet that the control plane comes back before your spare capacity stops being enough. That bet held in June 2023 and in October 2025. Whether your version of it holds is a thing you test, which is where this series goes next.

Read this next

For consulting on AWS, cloud architecture, resilience reviews, and platform work, start at ercanermis.com.

References