Cloud or On-Prem: What an SRE Checks Under the Service

The infrastructure provider is the largest single dependency any service has, and an SRE treats it like every other dependency: reads what it actually commits to, maps where it fails together, and checks what happens when it does not answer. That is true whether the provider is AWS or a platform team running racks two floors down. The questions are identical on both; what changes is who answers them. On a cloud the provider answers and you verify. On-prem you answer, and there is nobody to verify you. This post is the checklist, in six areas: the commitment, failure domains, control planes, limits, capacity, and upkeep.

This is Part 4 of Budget for Failure, the second 201-level part and the last of the series. Part 1 defined the role, Part 2 gave it numbers, and Part 3 took it into the application code. This part goes under the code.

1. The commitment: your dependencies cap your SLO

Part 2 listed what four AWS SLAs commit to. Now do the arithmetic most architecture reviews skip. A request that must pass through a load balancer, a compute tier, a database, and object storage succeeds only when all four do. If the failures are independent, availabilities multiply:

ALB (multi-AZ)      0.9999
EC2 (multi-AZ)      0.9999
RDS (Multi-AZ)      0.9995
S3 Standard         0.9990
------------------------------
product             0.9983   =  99.83%
allowed downtime    about 73 minutes in 30 days

Before your own code has thrown one exception, the contractual floor under this stack is 99.83 percent. AWS normally runs well above its SLA floors, so the practical figure is better, but the conclusion holds: you cannot promise 99.99 percent on top of a hard dependency that commits to 99.9. You lower the promise, or you make the dependency soft with a cache, a queue, or a fallback, as in Part 3.

On-prem there is no SLA document to multiply, so you build the same table from your own history: measured availability of the storage array, the hypervisor cluster, the core network, the power. If nobody has those numbers, that is the first finding of the review.

2. Failure domains: what dies together

The multiplication assumes independence. A failure domain is a set of things that share a cause of failure, and the SRE's job is to know them and to place replicas across them.

Three checks apply on either side. Are the replicas of each stateful system really in different domains, or do three nodes of a quorum sit in one zone or one rack? Does anything single-homed sit in the path of something multi-homed, such as one NAT gateway serving three zones, or two "redundant" switches on the same power feed? And after the largest domain you plan to survive is gone, is the remaining capacity enough to carry the full load? Surviving a zone loss with three zones means running each at no more than two thirds.

The Nothing Fails Alone series is four worked incidents of exactly this, including a read-only script that finds single-zone concentrations in an AWS account.

3. Control plane and data plane

Every platform has two halves. The data plane does the work: instances run, load balancers forward, DNS answers. The control plane changes things: launch an instance, modify a route, scale a group. Control planes are more complex and fail more often, and they tend to fail during large incidents, which is exactly when a recovery plan wants to use them.

So the question for each recovery procedure is whether it needs a control plane call to succeed. A design that survives a zone loss because capacity is already running elsewhere is statically stable. One that survives because autoscaling will launch replacements depends on the EC2 control plane at the moment every other customer is calling it. The Amazon Builders' Library article on static stability is the reference.

On-prem the same split exists under other names: the Kubernetes API server, vCenter, the IPAM and DNS automation, the identity provider that every login and every service token depends on. Ask what still works when each is down, and whether the tooling used to fix an outage (the VPN, the bastion, the deploy pipeline, the chat system) runs on the infrastructure it is supposed to repair.

4. Limits: the outage that looks like a quota

Saturation, from Part 1's golden signals, has a provider-side form that host metrics never show. On AWS, the limits that end up in postmortems are rarely CPU:

• Service quotas. Running On-Demand vCPUs per instance family, Lambda concurrent executions per account, Elastic IPs, rules per security group. A scale-out that hits a quota fails at the worst moment.
• API rate limits. Control plane APIs throttle per account and Region. A deploy tool, an autoscaler, and a monitoring agent all calling Describe* can throttle each other.
• Address space. A subnet that has run out of free IPs stops new pods, new Lambda network interfaces, and new instances, with an error that mentions none of those.
• Per-resource ceilings. IOPS and throughput on an EBS volume, connections on an RDS instance class, ports on a NAT gateway.

Treat each as a saturation SLI with an alarm well before 100 percent. For many quotas CloudWatch publishes usage in the AWS/Usage namespace, and the SERVICE_QUOTA() metric math function returns the current limit, so one expression gives percent of quota used. For the rest, an inventory script on a schedule is enough.

On-prem the limits are physical and slower to raise: rack units, power and cooling per rack, switch ports, licensed cores, storage controller throughput, the size of the address plan made five years ago.

5. Capacity: elastic is not infinite

The cloud's promise is that capacity arrives in minutes. It usually does, and an SRE plans for the times it does not. A specific instance type in a specific zone can return InsufficientInstanceCapacity, most often during a zone event when everyone is failing over into the same surviving zones. The defenses are diversity (several instance types and all zones in the Auto Scaling group), pre-provisioning for the failover case, and On-Demand Capacity Reservations for the workloads that cannot wait.

On-prem the lead time is weeks to months, so capacity planning is a forecasting discipline: measured growth, a procurement calendar, and headroom sized as N+1 or N+2 against the failure domains above. The decision that most affects reliability is made at purchase time, long before the incident that tests it.

In both worlds, load test to find the first limit instead of estimating it. The number that matters is the traffic level at which the SLI from Part 2 starts to degrade, and which resource caused it.

6. Upkeep: the incidents with a date on them

A large share of incidents are scheduled in advance by someone and then forgotten. On a cloud: instance retirement notices, database maintenance windows, engine and Kubernetes versions reaching end of standard support, certificate expiry, deprecated runtimes. AWS publishes most of these per account through AWS Health, which can be routed through EventBridge into the same queue as your other operational work. Unread, they become surprise reboots.

On-prem the list is longer and entirely yours: firmware, hypervisor and operating system patching, disk and battery replacement, hardware refresh cycles, vendor support contracts, spare parts on site. Each needs an owner and a calendar. A platform that is three major versions behind is spending error budget in advance.

The last check in this area is how you find out the provider is having a bad day. A status page is written by people during an incident and lags it. Measure the provider yourself, from outside: a synthetic canary that exercises the real user path from another network tells you about a regional problem before the dashboard does, and it keeps the SLI denominator alive when real traffic cannot reach you.

Who owns what

Neither column is shorter because the work vanished. A cloud moves the lower rows to a provider with more practice at them than almost any single company, and in exchange you take on a dependency you cannot inspect. That trade is usually right. It is still a trade, and the SRE is the person who can state both sides of it.

Failure modes and what to watch

Treating the SLA as a forecast. It is a pricing term with a credit attached. Design to measured behavior and failure domains, not to the percentage on the page.

Multi-zone on the diagram only. Three zones in the subnet list and one in the running state, or a failover that has never been exercised. Test it on purpose, inside the error budget.

Recovery that needs what is broken. Runbooks whose first step is a console login or a pipeline run in the affected Region.

Limits discovered in production. Every quota and ceiling in section 4 is cheap to find in a load test and expensive to find during a launch.

Where this leaves you

The path of the series, in one pass. You own reliability as an engineering problem and cap the manual work so there is time to solve it (Part 1). You state how reliable the service should be as an SLO, and you spend the gap deliberately as an error budget with a signed policy (Part 2). You go into the repository and fix the timeouts, retries, and rollouts that decide the number, and you page on burn rate instead of CPU (Part 3). And you know what the platform under you promises, where it fails together, and what you will do when it does (Part 4). The tools for all four are ones most engineers already use. What makes the work SRE is that every one of them is now answering a single question: are users getting the reliability we agreed on, and what is the cheapest way to keep it so?

Read this next

• Part 3 of this series: SRE in the Codebase: Timeouts, Retries, and Alerts as Code.
• The Fix Shares Fate With the Outage: three incidents where the recovery path depended on the control plane that was down.
• Your LLM Bill Is an Observability Problem at ercan.ai: the same provider-dependency thinking applied to model APIs, where quotas and cost are the saturation signals.

For consulting on AWS, cloud architecture, SRE practice, CI/CD pipelines, Linux, and more, start at ercanermis.com.

References

• Amazon Builders' Library: Static stability using Availability Zones
• AWS whitepaper: AWS Fault Isolation Boundaries
• Amazon CloudWatch: Visualizing your service quotas and setting alarms
• Amazon EC2: On-Demand Capacity Reservations
• AWS Health: Monitoring events with Amazon EventBridge
• Amazon Compute Service Level Agreement
• Google SRE Book, Chapter 18: Software Engineering in SRE (capacity planning)