What an SRE Does: The Job Behind the Title

Site reliability engineering is what you get when you treat operations as a software problem: a team of engineers who own the reliability of a production service, measure it from the user's side, and spend at least half their time writing code that removes operational work instead of doing that work by hand. An SRE is not a renamed sysadmin and not a DevOps engineer with a pager. The role has a defined scope, a defined set of things it watches, and a defined relationship with developers. This post covers all three.

This is Part 1 of Budget for Failure, a four-part series on what site reliability engineering is and how to practice it. Parts 1 and 2 are 101 level: the role, then the numbers it runs on (SLI, SLO, SLA, error budgets). Parts 3 and 4 are 201 level: what an SRE needs to command in the codebase, and what to check on the provider side, whether that is a cloud or your own data center. Examples are on AWS; the practice is not AWS-specific.

Where the role comes from

The term comes from Google, where Ben Treynor Sloss built the first team in 2003 and later described it in one line in the SRE book: SRE is what happens when you ask a software engineer to design an operations team. The engineer's answer to a growing service was not to hire operators in proportion to traffic. It was to automate the operator's work, and to put a hard cap on how much manual work the team was allowed to accept.

The relationship to DevOps is usually summarized, again by Google, as "class SRE implements interface DevOps". DevOps is a set of principles: shared ownership, small changes, automation, measurement. SRE is one concrete implementation of those principles with opinions attached, such as exactly how reliability is measured and exactly what happens when the measurement says a service is not reliable enough.

What an SRE is responsible for

The SRE book lists the scope in one sentence: an SRE team is responsible for the availability, latency, performance, efficiency, change management, monitoring, emergency response, and capacity planning of its services. Unpacked into what that means on a working week:

• Availability and latency. Defining what "working" means for users as a number, measuring it, and defending it. Part 2 is entirely about this.
• Change management. Most outages are caused by a change. The SRE owns the machinery that makes change safe: progressive rollouts, automated rollback, and the evidence that a release is healthy.
• Monitoring and alerting. Deciding which signals deserve a human at 03:00 and which belong on a dashboard. A page that needs no action is a defect in the alerting, and the SRE fixes it like any other defect.
• Emergency response. Being on call, running incidents with a clear command structure, and restoring service first and diagnosing second.
• Postmortems. After an incident, a written, blameless analysis of what happened and which systemic fix prevents the class of failure. Blameless is a technical requirement: people who expect punishment stop reporting what they saw.
• Capacity planning. Knowing how much headroom exists, how demand is growing, and how long new capacity takes to arrive, in minutes on a cloud, in months on-prem.
• Efficiency and performance. Reliability bought with unlimited spare capacity is not engineering. Cost per request is an SRE number too.
• Eliminating toil. Covered below, because it is the part that makes the rest possible.

What an SRE looks at

A DevOps engineer coming from infrastructure tends to watch resources: CPU, memory, disk, node counts. An SRE starts one layer up, at what the user experienced, and uses resource metrics only to explain it. The standard starting set is the four golden signals from the monitoring chapter of the SRE book:

Two details separate an SRE reading of these from a casual one. Latency is read as a distribution, never an average: the 99th percentile is where the unhappy users are. And saturation is read against the limit that will actually be hit first, which is rarely CPU. It is the database connection pool, the NAT gateway port allocation, the account-level Lambda concurrency, or the EBS volume's IOPS budget.

Beyond the four signals, an SRE watches the things that predict next month's incident: the rate and failure rate of changes, the age of the oldest unpatched dependency, how much of last week was spent on manual work, and how many pages the on-call engineer received.

The 50 percent rule and toil

The feature that makes SRE different from an operations team with a better title is a cap. In Google's model an SRE spends at least 50 percent of their time on engineering work, averaged over a few quarters. The rest may go to operational work: tickets, on-call, manual tasks.

The manual part has a name, toil, and a precise definition: work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as the service grows. Restarting a stuck worker, rotating a certificate by hand, approving a quota request, copying a database for a developer. Each is small. Together they are why a team of five cannot do any engineering.

The cap is what gives the SRE standing to say no. When toil goes above half the team's time, the response is not heroics. It is to automate the largest source, or to hand the operational load back to the team that generates it until it shrinks. On-call follows the same logic: the SRE book puts a sustainable load at about two incidents per on-call shift, because each incident needs follow-up and a postmortem, and a person who gets five pages a night does neither.

How an SRE helps developers

The worst version of this role is a gate: developers throw a release over a wall and a reliability team decides whether it may pass. The useful version treats developers as the customer and reliability as a product the SRE builds for them. Four concrete forms:

The paved road. A service template that arrives with structured logging, metrics, tracing, health checks, a deployment pipeline with canary and rollback, and sensible alarms already wired. A developer who takes the default gets a reliable service without reading a runbook. This is the highest-value thing an SRE team ships.

The production readiness review. Before a new service takes real traffic, the SRE walks through it with the developers: what does it depend on, what happens when each dependency is slow, what are the timeouts, how is it rolled back, who is paged. It is a review of design, done early enough to change the design. Part 3 goes through what to look for.

The error budget. A shared, pre-agreed number that tells both sides when it is safe to ship fast and when to slow down. It replaces the argument about whether a release is "too risky" with a reading. Part 2 builds it.

Shared on-call. Developers who carry the pager for their own service write different code. Many organizations start a service on the developers' rotation and move it to SRE only once it meets a reliability bar, and hand it back if it falls below. The SRE's job is to make that rotation humane, with good alerts and good runbooks, not to absorb the pain so nobody upstream feels it.

SRE and DevOps: where the work differs

If you run pipelines, Terraform, and Kubernetes for a living, most of the tooling carries over. What changes is the question each tool is answering.

The last row is the one most infrastructure engineers underestimate. An SRE who cannot read the service's code can only observe failures from the outside. Reliability is decided in how a function handles a slow dependency, and that is in the repository, not in the cluster.

Failure modes and what to watch

The rename. An operations team given the SRE title and the same ticket queue has changed nothing. Without the toil cap and the authority that comes with it, there is no time for engineering, and the title describes an aspiration.

The gatekeeper. A reliability team that approves releases becomes the bottleneck the organization routes around. Influence comes from the paved road and from data, not from a veto.

The hero. One engineer who knows every system and fixes every outage is a single point of failure with a salary. Incidents that end with "it took Sam twenty minutes" need a runbook, not applause.

Reliability with no number. "As reliable as possible" cannot be planned, staffed, or traded against features. Everything in this post depends on being able to say how reliable a service should be and whether it currently is. That is the vocabulary of SLIs, SLOs, and error budgets, and it is Part 2.

Read this next

• Part 2 of this series: SLI, SLO, SLA, Error Budget: The Numbers SRE Runs On, where reliability becomes arithmetic.
• Nothing Fails Alone, four incident reviews that show what the emergency response and postmortem parts of this job look like in practice.
• Agents on Call at ercan.ai, a build log on handing part of the on-call triage described here to an AI agent.

For consulting on AWS, cloud architecture, SRE practice, CI/CD pipelines, Linux, and more, start at ercanermis.com.

References

• Google SRE Book, Chapter 1: Introduction
• Google SRE Book, Chapter 6: Monitoring Distributed Systems
• Google SRE Book, Chapter 5: Eliminating Toil
• Google SRE Book, Chapter 11: Being On-Call
• Google SRE Book, Chapter 15: Postmortem Culture
• Google SRE Workbook, Chapter 1: How SRE Relates to DevOps