SLI, SLO, SLA, Error Budget: The Numbers SRE Runs On

An SLI is a measurement of what users experienced, an SLO is the target you set for that measurement, an SLA is a contract that says what happens when the target is missed, and the error budget is the difference between 100 percent and the SLO, counted in failed requests you are allowed to have. A 99.9 percent SLO over 30 days on a service handling 10 million requests is a budget of 10,000 failures. Those four numbers are how an SRE answers the question the rest of the organization cannot: how reliable is this service supposed to be, and is it safe to ship today?

This is Part 2 of Budget for Failure. Part 1 described the SRE role and ended on its dependency: none of it works without a number. This part is still 101 level. It defines the terms, reads the AWS SLAs for what they actually promise, computes a first SLI from load balancer metrics you already have, and turns the result into a budget with a policy attached.

SLI: the ratio of good events to valid events

A service level indicator is a number between 0 and 100 percent, computed the same way every time: good events divided by valid events. For a request-driven service the two most useful SLIs are:

• Availability: requests answered without a server error, divided by all requests.
• Latency: requests answered faster than a threshold (say 300 ms), divided by all requests.

Two design decisions hide in that definition. First, what counts as valid. A 404 for a URL that does not exist is the service working correctly; a 429 from your own rate limiter is arguable; a 500 is not. Decide once and write it down, because the denominator is where SLIs get quietly gamed. Second, where you measure. The closer to the user, the more honest the number. A metric taken at the load balancer sees failures the application never logs, such as a target that refused the connection. A metric taken inside the application misses them.

Averages do not qualify as an SLI. An average latency of 120 ms is compatible with one request in twenty taking four seconds. The ratio form ("97.2 percent of requests under 300 ms") tells you how many users had a bad time, which is the thing you are trying to bound.

SLO: the target, and the window it is measured over

A service level objective is an SLI plus a target plus a time window: "99.9 percent of requests succeed, measured over a rolling 30 days". All three parts are required. A target without a window is unfalsifiable, since any bad hour can be averaged away by choosing a longer period after the fact.

The target is where the nines come in, and the nines are easier to reason about as allowed downtime. For a 30-day window (43,200 minutes):

Read the table from the bottom. At 99.99 percent you have about four minutes a month. A human paged at 03:00 needs longer than that to find a laptop, which means a four-nines SLO is a commitment to automated detection and automated recovery, with people involved only afterwards. Each extra nine is roughly a tenfold cut in allowed failure and, in practice, a large step in cost. The right SLO is the lowest number your users would not notice the difference above, not the highest number that looks good on a slide.

One hundred percent is never the right target. It forbids every deploy, every migration, and every dependency you do not control, and your users reach you through home Wi-Fi and mobile networks that are far less reliable than anything you run.

SLA: the contract, and what AWS actually owes you

A service level agreement is an SLO with consequences attached, signed with a customer. The consequence is usually money. That is the whole difference, and it is why the SLA is always looser than the internal SLO: you want to notice and react well before a breach becomes a refund.

AWS is the SLA most of us read first, and it is worth reading closely because the wording teaches the concept. Four that sit under a typical web service:

Three things in those documents matter more than the percentages.

The remedy is a service credit. If S3 Standard drops to 99.5 percent for a month, you are eligible for 10 percent of that month's S3 charges for the affected Region, applied against a future bill, after you file a claim with evidence. Your lost revenue, your own SLA penalties, and your weekend are not part of the calculation. An SLA is a pricing term. It is not an insurance policy and it is not a prediction of how the service will behave.

The commitment is conditional on your architecture. The 99.99 percent for EC2 applies only when all running instances are deployed across two or more Availability Zones in the Region. Run in one zone and you are on the 99.5 percent instance-level commitment, which allows about 3.6 hours of downtime a month. The SLA encodes the design AWS expects of you.

"Unavailable" has a narrow legal definition. Each SLA defines its own measurement, per minute or per five-minute interval, and its own exclusions. Slow is not unavailable. Elevated error rates in one API you depend on may not count at all.

Your first SLI, from metrics you already have

If your service sits behind an Application Load Balancer, CloudWatch already holds an availability SLI. Three metrics in the AWS/ApplicationELB namespace are enough: RequestCount, HTTPCode_Target_5XX_Count (errors your code returned), and HTTPCode_ELB_5XX_Count (errors the load balancer generated itself, for example when no healthy target answered). Save this as sli.json:

[
  { "Id": "req", "ReturnData": false,
    "MetricStat": { "Stat": "Sum", "Period": 86400,
      "Metric": { "Namespace": "AWS/ApplicationELB", "MetricName": "RequestCount",
        "Dimensions": [{ "Name": "LoadBalancer", "Value": "app/checkout/0123456789abcdef" }] } } },
  { "Id": "t5xx", "ReturnData": false,
    "MetricStat": { "Stat": "Sum", "Period": 86400,
      "Metric": { "Namespace": "AWS/ApplicationELB", "MetricName": "HTTPCode_Target_5XX_Count",
        "Dimensions": [{ "Name": "LoadBalancer", "Value": "app/checkout/0123456789abcdef" }] } } },
  { "Id": "e5xx", "ReturnData": false,
    "MetricStat": { "Stat": "Sum", "Period": 86400,
      "Metric": { "Namespace": "AWS/ApplicationELB", "MetricName": "HTTPCode_ELB_5XX_Count",
        "Dimensions": [{ "Name": "LoadBalancer", "Value": "app/checkout/0123456789abcdef" }] } } },
  { "Id": "sli", "Label": "availability_percent",
    "Expression": "100 * (req - FILL(t5xx, 0) - FILL(e5xx, 0)) / req" }
]

aws cloudwatch get-metric-data \
  --start-time "$(date -u -v-30d +%Y-%m-%dT00:00:00Z)" \
  --end-time "$(date -u +%Y-%m-%dT00:00:00Z)" \
  --metric-data-queries file://sli.json \
  --query 'MetricDataResults[0].[Timestamps,Values]'

That returns one availability figure per day for the last 30 days (the date -v syntax is macOS; on Linux use date -u -d '30 days ago'). FILL(…, 0) matters: CloudWatch publishes no datapoint for an error metric in a period with zero errors, and without the fill a perfect day comes back as missing data instead of 100 percent.

For latency, the same namespace has TargetResponseTime, and CloudWatch's percentile rank statistic turns it into the ratio form directly: PR(:0.3) returns the percentage of requests that completed in 300 ms or less.

The error budget: what the SLO lets you spend

Once the business has agreed that 99.9 percent is good enough, the remaining 0.1 percent stops being a defect to eliminate and becomes an allowance to use. If the service sits at 99.97 percent with ten days left in the window, there is room to take risk: ship the migration, upgrade the cluster, run the failover test. If it sits at 99.88 percent, there is not, and nobody has to win an argument to establish that. A service that ends every window with the budget untouched is not a triumph either: the SLO is looser than users need, or the team is paying for reliability nobody asked for.

The budget can be counted in events or in time, and the two behave differently.

SLO                      99.9% over 30 days
requests in window       10,000,000
error budget             (1 - 0.999) x 10,000,000 = 10,000 failed requests

bad requests so far      6,400
budget consumed          6,400 / 10,000 = 64%
budget remaining         3,600 requests = 36%

Event-based budgets weigh failure by traffic. Ten minutes of total outage at the Tuesday peak costs far more than ten minutes at 04:00 on Sunday, which matches what users experienced. This is the form to prefer for anything request-driven.

Time-based budgets count bad minutes: 99.9 percent over 30 days is 43 minutes and 12 seconds. They are easier to explain to a non-engineer and they fit services with no steady request stream, such as a batch pipeline or a low-traffic internal tool. The cost is that a minute counts as fully bad or fully good, so you need a rule for what makes a minute bad (for example, "error rate above 1 percent"), and a partial outage is either invisible or counted in full.

The window matters as much as the target. A rolling 30-day window always reflects the recent past, and a bad day leaves the calculation exactly 30 days later; it is the better fit for tracking user experience. A calendar window resets on the first of the month, which lines up with reporting and with most SLAs, and has one bad property: an incident on the 2nd spends the budget for four weeks, while the same incident on the 30th is forgiven two days later.

What spends the budget

Everything that makes a user's request fail draws from the same account, whoever caused it. That is the point. In roughly descending order of how much a typical team spends:

• Changes. Deploys, config pushes, feature flags, schema migrations. In most postmortem collections, change is the largest single cause of incidents. If you track DORA's change failure rate, you are already measuring the main spender.
• Dependencies. A regional API slowdown, a third-party payment provider, a DNS problem. It is your budget even when it is not your fault, because the user does not care whose fault it is.
• Capacity and load. Throttling, cold caches, a scaling policy that reacts two minutes too late.
• Planned work. Maintenance windows, failover tests, game days. A deliberate zone evacuation drill that costs 300 failed requests is budget well spent, and having a budget is what makes the drill approvable.
• Hardware and platform. Instance retirements, zone events. Smaller than people expect, if the architecture is multi-AZ.

Keep a ledger. After each incident or risky change, record how much budget it consumed and its cause. Three months of that ledger tells you where reliability engineering time should go better than any roadmap discussion will.

In this example one deploy cost more than a third-party outage and a planned test together. The investment case writes itself: canary analysis and automatic rollback before anything else.

The policy is the product

A budget with no consequence is a dashboard. What makes it work is a short written policy, agreed in advance by engineering and product leadership, that says what changes when the budget runs low. The Google SRE Workbook publishes an example policy worth copying the structure of. A workable first version fits on half a page:

• Budget above 50 percent: normal release process. Risky changes and experiments are encouraged.
• Budget below 50 percent: changes need a canary stage and a tested rollback. No experiments in production.
• Budget exhausted: feature releases stop until the service is back within SLO. Only reliability fixes, security patches, and changes needed to restore the budget go out. The team's priority is the top items from recent postmortems.
• A single incident consumes more than 20 percent of the budget: a postmortem is mandatory, with at least one high-priority action item.
• Disagreement: a named executive decides. The escalation path is written down before it is needed.

Two properties make this hold. It is symmetrical: the same document that freezes releases when the budget is gone tells operations to stop blocking them when it is healthy. And it is pre-agreed: the product owner signed it in a calm week, so invoking it during a bad one is execution, not confrontation. If leadership will not sign a policy that can stop feature work, you do not have an SLO yet. You have a metric.

Let CloudWatch keep the ledger

The SLI above was computed by hand with get-metric-data. For the running budget, CloudWatch has a managed SLO feature inside Application Signals. You define the indicator, the attainment goal, and the interval, and it tracks attainment and remaining error budget continuously. Three choices in its model map directly onto this post:

• Period-based or request-based. A period-based SLO divides the interval into periods (one minute, for example), judges each as good or bad, and reports good periods over total periods. A request-based SLO counts good requests over total requests. These are the time-based and event-based budgets described above.
• Rolling or calendar interval. Both are supported, with the trade-off already described.
• What the SLI is built on. Services instrumented with Application Signals get standard latency and availability metrics, with 5xx responses counted as faults and 4xx responses treated as successful. If you are not instrumented, an SLO can be set on any CloudWatch metric or metric math expression that results in a single time series, which includes the load balancer expression above.

The console shows the outcome as two figures: attainment against the goal, and budget remaining. For a request-based SLO the remaining budget can go up as well as down during the interval, because it depends on the ratio of good to total requests so far, and a quiet week of clean traffic dilutes an earlier incident.

Failure modes and what to watch

Averaging percentages. A day with 100 requests at 90 percent and a day with a million requests at 99.99 percent do not average to 95. The 30-day SLI is total good requests over total valid requests. Sum the numerators and the denominators, then divide.

A denominator with holes. RequestCount only counts requests for which the load balancer was able to choose a target, so the expression above slightly overstates the damage in a total outage. That is the safe direction to be wrong in. The SLI also cannot see requests that never reached the load balancer: a DNS failure, a WAF block, a client on a broken network.

The SLO nobody agreed to. A target chosen by the platform team alone has no authority when it collides with a launch date. The number has to be owned by whoever owns the roadmap.

Too many SLOs. Start with one or two per user-facing service: availability and latency on the request path that earns the money. Twenty SLOs produce twenty budgets, one of which is always exhausted, and the policy becomes noise.

Setting the target before seeing the data. Run the query against the last 30 days first. An SLO picked before you have seen the SLI is a wish. One set slightly below measured history, then tightened on purpose, is a target you can defend.

Finding out at the end of the month. A budget read on a monthly report can only explain the past. Acting while there is budget left takes an alert on the rate of spending, and making the service spend less takes changes in its code. Both are Part 3.

Read this next

• Part 3 of this series: SRE in the Codebase: Timeouts, Retries, and Alerts as Code, where the budget meets the repository.
• Part 1 of this series: What an SRE Does: The Job Behind the Title.
• Evals Before Agents: You Can't Ship What You Can't Score at ercan.ai, the same discipline applied to LLM systems: define the measurement before you argue about quality.

For consulting on AWS, cloud architecture, SRE practice, CI/CD pipelines, Linux, and more, start at ercanermis.com.

References

• Google SRE Book, Chapter 4: Service Level Objectives
• Google SRE Book, Chapter 3: Embracing Risk
• Google SRE Workbook, Appendix B: Example Error Budget Policy
• Amazon Compute Service Level Agreement
• Amazon Elastic Load Balancing Service Level Agreement
• Amazon RDS Service Level Agreement
• Amazon S3 Service Level Agreement
• CloudWatch metrics for your Application Load Balancer
• Amazon CloudWatch: Service level objectives (SLOs)