Both of the load-bearing assumptions in this series are testable today, with a managed service, in an afternoon. AWS Fault Injection Service ships a scenario named "AZ Availability: Power Interruption" that will stop every tagged instance in one availability zone, feed InsufficientInstanceCapacity errors to the Auto Scaling groups trying to replace them, pause EBS volume IO, and cut subnet traffic, then lift the faults and restart what it stopped. The control-plane assumption from Part 3 has no off-the-shelf fault, but an honest approximation of it fits in one JSON file and one IAM deny. Almost nobody runs either, and the reason is not tooling; the reason is that a test can fail and an assumption cannot.

This is Part 4 of Nothing Fails Alone, the last one. Part 1 took the May us-east-1 incident and showed the zone boundary holding while a quorum died inside it. Part 2 took the March me-central-1 event and showed one cause reaching two zones eighteen hours apart, past every isolation guarantee. Part 3 took three control-plane incidents across two providers and showed recovery plans dying with the layer they run on. The spine of the series, in two sentences: the availability you buy ends at the boundary of what the provider promised, and everything above that line is architecture you own. What you own and never exercise is not architecture, it is an assumption with your name on it. This part exercises it, then scores the series.

The AZ drill comes off the shelf

The FIS scenario library's AZ Availability: Power Interruption is a published experiment template that induces the documented symptoms of losing one zone, and its published JSON carries eight actions across seven fault types (stopping instances appears twice, once for standalone instances and once for the Auto Scaling groups' own). Tagged running instances in the target zone are stopped and restarted after the configured duration. Launch requests from tagged Auto Scaling groups get InsufficientInstanceCapacity for the duration, and so do the capacity calls of any IAM roles you name, because, per the scenario page, "EC2 API calls made by the Auto Scaling control plane to recover lost capacity in the AZ will fail" during a real power interruption. EBS volume IO is paused. Subnet traffic is denied by swapping in a cloned network ACL full of deny rules. RDS clusters with a writer in the zone are failed over, and ElastiCache replication groups get their zone's power interrupted. That is most of what May's incident did to use1-az4: the failure that took Coinbase down for a night is now a library item.

Be precise about what it is not. The instances are stopped through the front door, the StopInstances API: an orderly shutdown, not a power cut, so nothing about crash consistency gets tested. Fargate tasks are not covered. RDS Multi-AZ clusters with two readable standbys are not supported. One drift note: the scenario page describes a recovery action, aws:arc:start-zonal-autoshift, that its own JSON snapshot does not contain; the console's library carries the newest revision, and the lab follows the published JSON.

The lab is 04-game-day in the series repo, and it breaks the series' read-only rule for the second and last time: --inject starts real experiments against whatever account your credentials reach, so it belongs in a sandbox, pointed at Part 3's lab stack, and nowhere near production.

Names, IDs, and a stop condition shipped empty

Part 2 made a fuss about zone names versus zone IDs, and the game day is where the fuss pays off. Incident reports speak IDs: the May report said use1-az4, the March updates said mec1-az2 and then mec1-az3. Your account speaks names, aliases that AWS randomizes per account. And the tool, for the most part, demands the name. The instance and subnet targets filter on Placement.AvailabilityZone, which matches your account-local name, and the published scenario fills the EBS, RDS, and ElastiCache target parameters with the name too. Exactly two actions accept the physical ID: the insufficient-capacity pair, aws:ec2:api-insufficient-instance-capacity-error and aws:ec2:asg-insufficient-instance-capacity-error, whose availabilityZoneIdentifiers parameter is documented to take zone IDs as well as names. Even there, the published scenario fills the field with a name, us-east-1a.

A game day therefore starts with the translation problem Part 2 shipped a lab for. The runner resolves it once, at the top: the only zone flag run.py accepts is --az-id, the physical zone, the thing an incident report names. It resolves the account-local name via DescribeAvailabilityZones, prints both, and fills each placeholder with the form its consumer requires. You rehearse the failure under the name an incident report would give it, and the rehearsal configures itself in the dialect your account happens to speak.

The second thing the runner refuses to inherit is quieter. The published scenario's stop-condition block looks like this, verbatim:

"stopConditions": [
    {
        "source": "aws:cloudwatch:alarm",
        "value": ""
    }
]

The shape of a guardrail with nothing in it, and the page's limitations section says stop conditions are yours to add. A defensible default, since AWS cannot know your steady state; also a loaded one, because the JSON pastes cleanly and an empty string does not look like a missing safety system. The lab's position: run.py exits before creating anything if --alarm-arn is not supplied, because an experiment without a meaningful stop condition is not a test, it is a bet. Steady state is a number you commit to before the fault, or the game day cannot fail, and a game day that cannot fail cannot tell you anything.

What a pass looks like, measured

Run the experiment against Part 3's two-path stack, failing one of its two zones:

python3 run.py experiments/az-power-interruption.json \
  --az-id euw1-az1 \
  --fis-role-arn arn:aws:iam::111111111111:role/fis-game-day \
  --alarm-arn arn:aws:cloudwatch:eu-west-1:111111111111:alarm:game-day-abort \
  --target-group-arn ... \
  --inject

Without --inject the same command is a dry run: it prints the resolved template and starts nothing. With it, the runner creates and starts the experiment, then polls to a terminal state, logging each action's status and sampling the target group's healthy count as the measurement. That timeline is the deliverable; what follows is the shape of a pass, not a result I have obtained, because I have no production-shaped account to earn it in. Path B, the statically stable one, never drops below its post-failure requirement of two healthy targets, since the surviving zone already carries the load. Path A drops to half and stays there for the entire fault duration, because the same experiment that stopped its instances is feeding its replacement launches InsufficientInstanceCapacity. Part 3's measurement script timed path A's recovery against a cooperative control plane; the game day shows the same path against an uncooperative one, and the difference between those two numbers is the size of the assumption your DR plan was making.

Four numbers are worth recording. Time to recovered capacity, from the healthy-target samples. The count of runbook steps that needed a control-plane call which would have failed, counted by walking your runbook during the fault window. The gap between Part 3's cooperative number and the game day's uncooperative one. And the fate of each finding from Part 1's detector, since every row it prints is a prediction the game day can settle: the single-AZ Auto Scaling group really does strand its capacity behind a capacity error; the healthy-zone subnet whose default route crosses into the failed zone really does lose egress. A detector finding is a hypothesis. A game day is the experiment.

The control-plane drill FIS cannot give you

Part 3's assumption is harder, and here the honest sentence comes first: FIS has no control-plane-loss action. The full actions reference never contains the phrase, and nothing in it takes a control plane away. The closest are error injections scoped to callers you name: the insufficient-capacity pair above, and a trio of generic API error actions (internal, throttle, unavailable) injecting failures into requests made by named IAM roles, for the EC2 and Kinesis namespaces only. The plane stays up; chosen callers get told no. Nothing you can rent reproduces the console serving 504s while STS fails three separate times.

So the lab's second template is an approximation and is labelled as one, in a top-level approximation key inside the file itself, so the label cannot drift from the thing it labels. The FIS half injects InsufficientInstanceCapacity on the workload ASGs' launches and on the deploy role's capacity calls, RunInstances, CreateFleet, StartInstances, CreateCapacityReservation, in the failed zone for twenty minutes. The other half is not FIS at all: a manual, scoped, time-boxed IAM deny on ec2:RunInstances, autoscaling:*, and cloudformation:*, which is what turns the runbook's "just push the emergency stack" step into the failure it would have been in June 2023. run.py prints the apply and remove commands and never executes them, the time box is decided before the policy is attached, and the deny never goes on the role you will need to remove it. That last rule is not pedantry; it is the difference between a drill and an incident.

What the approximation does not reproduce, from the file's own list: real control-plane degradation is latency and brownout, rarely a clean error; services the file never names, ELB, Route 53, IAM itself, the console, keep working here and might not in a real event; reads keep succeeding, so your dashboards look healthier than they would; non-capacity mutations like TerminateInstances and load balancer configuration changes keep succeeding, so this denies capacity creation, not the control plane; and it is one zone's capacity, where a real event can be regional. Measure one extra thing on this drill: time to detect. How long did it take a human to notice that the fix was not working? If someone tells you they simulated control-plane loss, ask which of those five gaps they covered. Mine covers none of them either; it covers the narrower claim that your architecture does not need the denied calls in the first hour.

Scoring the series

Four incidents, four labs, one spine. What the series established, held to its own standard: a claim counts only if it rests on the operator's published words. First, zone isolation is real against the faults it is engineered for: May's thermal event stayed inside use1-az4, and the architecture that failed that night failed by its own postmortem's admission, not AWS's. Second, zone failures are not independent events: one cause reached mec1-az2 and mec1-az3 eighteen hours apart, and two lost zones took regional services past their stated one-zone tolerance. Third, the control plane fails while the data plane keeps running, on more than one provider, and recovery plans that are control-plane operations fail with it: 2023 and October 2025 on AWS's own post-event summaries, February 2026 on Azure's.

Now the other column, because a retrospective that only counts wins is marketing. What remains assumption: everything quantitative about you. I never ran these labs against a production fleet, so every number in the series is the shape of an output, honestly labelled but still a shape. The thundering-herd cost of mass failover, AWS's "longer than usual provisioning times", appears in two incidents and has never been given a number, by AWS or by me. Part 3's static-stability premium was arithmetic, not a bill from a real fleet. And whether your break-glass path works is unknowable from here.

What I would do differently: run the game day first, not last. The series ordered its labs pedagogically, detect, translate, measure, inject, and that order is backwards operationally: one injection run produces the finding list the detector can only predict. And I would have distrusted secondary sources sooner. The rule, quote only from the operator's own page, kept earning its keep to the end: it caught wrong durations, a timestamp off by half a day, and claims whose apparent sources, when fetched, did not contain them. A series about untested assumptions nearly shipped several, which is about as clean a demonstration of its own thesis as I could have arranged.

Game days fail too

The instrument deserves the same skepticism as the thing it measures, so end on the three ways a game day lies to you.

  • The test that only proves the test ran. FIS resolves targets by tag, and a tag mismatch does not fail the experiment: the AZ scenario skips actions whose targets resolve to nothing, and the lab keeps that behavior (emptyTargetResolutionMode: skip) so partial stacks can run. The price: an experiment can reach completed having stopped nothing. Green result, zero faults injected. Read the per-action table, not the status line; the control-plane template sets fail instead, since there an empty target makes the run pointless. The exit codes agree: completed exits 0, a stopped experiment exits 1 even though stopping means your alarm worked. The guardrail firing is the guardrail working, and still not a pass.
  • The stop condition that fires too early. Bind the alarm to an internal metric with a tight threshold and the experiment stops two minutes in, before the interesting failure. You learn only that the alarm fires, and the temptation afterward is to loosen or remove it, which is how you arrive back at the scenario's empty string by choice. The fix is not less guardrail; it is binding the alarm to the user-facing steady-state metric, at the threshold where you would want a real incident stopped.
  • The account that is not production. A sandbox pass proves the mechanism, not the outcome. No production traffic, no data gravity, no noisy neighbors, no operator woken at 3 AM. Add the injection's own softness, orderly shutdowns, clean errors instead of brownouts, and the honest chain of evidence reads: the detector predicts, the sandbox game day tests the prediction's mechanism, and production inherits a labelled extrapolation. That is much better than nothing and much less than proof, and pretending otherwise is how "we did chaos engineering" becomes one more untested assumption.

That is where the series ends. Nothing fails alone: every part found the failure entangled with something beside it, a quorum with a building, a zone with a war, a fix with a control plane. Nothing passes alone either. Every pass is entangled with the environment that produced it, and the label saying so is the most load-bearing line in the report.

Read this next

For consulting on AWS, cloud architecture, resilience reviews, and platform work, start at ercanermis.com.

References