SRE in the Codebase: Timeouts, Retries, and Alerts as Code

Most of a service's reliability is decided in its repository, in how each function behaves when a dependency is slow, not in the cluster it runs on. An SRE therefore has to read application code well enough to review it and change it: find the network call with no timeout, the retry loop that will triple the load during an outage, the health check that will take every instance out of service at once. This post is the review list, with the fixes, and it ends with the one piece of code the SRE owns outright: alerts that page on error budget burn rate, written in Terraform.

This is Part 3 of Budget for Failure, and the first 201-level part. Part 1 defined the role and Part 2 gave it numbers: a 99.9 percent availability SLO over 30 days, measured at an Application Load Balancer. That example continues here. Code samples are Python; the patterns are the same in any language.

How much code an SRE needs

You do not need to be the strongest feature developer on the team. You need three abilities. Read the request path: starting from the handler, follow one user request through every network call it makes and say what happens if each call takes thirty seconds. Review a pull request for operational behavior, which is a different reading from reviewing it for correctness. And write production-quality changes yourself: the timeout, the metric, the rollback hook. An SRE who files a ticket asking developers to add a timeout has described the problem. One who sends the pull request has fixed it.

Every network call needs a deadline

The most common reliability defect in application code is a call that can wait forever. Python's requests library has no default timeout at all. The AWS SDK for Python defaults to 60 seconds for connect and 60 seconds for read, with automatic retries on top. A handler that calls DynamoDB with those defaults, behind a load balancer whose idle timeout is 60 seconds, can hold a worker for minutes after the user has already received a 504.

When a dependency slows down, calls without deadlines turn a slow dependency into a dead service: every worker is parked waiting, the pool is exhausted, and requests that never needed the slow dependency fail too. Set the limits on purpose:

import boto3
from botocore.config import Config

sdk = Config(
    connect_timeout=1,
    read_timeout=2,
    retries={"mode": "standard", "max_attempts": 3},
)
ddb = boto3.client("dynamodb", config=sdk)

Choose the read timeout from the dependency's measured latency, a little above its 99.9th percentile, not from a round number. Then check that the limits nest. If the user-facing budget is 3 seconds, the handler cannot make three sequential calls that may each take 2 seconds with 3 attempts. The outer deadline has to be passed down, so that inner calls give up when there is no longer anyone waiting for the answer.

Retries: the fix that causes outages

A retry turns a transient failure into a success, and a sustained failure into a larger one. Three rules keep it on the right side.

Retry at one layer. If the client retries three times, the API retries its database call three times, and the SDK retries three times underneath, one user action becomes up to 27 requests against a dependency that is already struggling. Pick the layer closest to the failure and make the others fail fast.

Back off with jitter. Clients that retry on a fixed schedule arrive together and knock the recovering service down again. The AWS Architecture Blog analysis of this is the standard reference; its "full jitter" variant sleeps a random time between zero and an exponentially growing cap:

import random, time

def call_with_retry(fn, attempts=3, base=0.1, cap=2.0, deadline=None):
    for attempt in range(attempts):
        try:
            return fn()
        except RetryableError:
            if attempt == attempts - 1:
                raise
            pause = random.uniform(0, min(cap, base * 2 ** attempt))
            if deadline and time.monotonic() + pause > deadline:
                raise
            time.sleep(pause)

Retry only what is safe to repeat. A timed-out read is safe. A timed-out "charge the card" is not, unless the operation is idempotent: the client sends a unique key with the request and the server stores the result under it, so a repeat returns the first outcome instead of charging twice. In review, any retry around a write is a prompt to ask where the idempotency key is.

Fail smaller: breakers, shedding, degradation

Timeouts and retries handle a dependency that is briefly unwell. For one that is down, the code needs a way to stop trying and a decision about what the user sees in the meantime.

Circuit breaker. After a run of failures, stop calling the dependency for a short interval and fail immediately. It protects your worker pool and gives the dependency room to recover.
Load shedding. When the service is saturated, reject the excess early with a 429 or 503 instead of accepting everything and serving all of it slowly. A cheap fast rejection of 20 percent of requests beats a timeout for 100 percent.
Graceful degradation. Decide per dependency whether it is hard or soft. If the recommendations service is down, the product page should render without recommendations, not return a 500. Each dependency converted from hard to soft comes out of the availability multiplication, which Part 4 does in full.

Health checks that tell the truth

A health check is code, and it is frequently wrong in one of two directions. A shallow check that returns 200 whenever the process is alive keeps a broken instance in rotation. A deep check that tests the database from every instance has a worse failure: when the database blips, every instance fails its check at the same moment, and the load balancer has no healthy target left. An Application Load Balancer in that state fails open and routes to all targets anyway, which saves you, but an orchestrator configured to replace unhealthy instances will start killing a healthy fleet during a database incident.

The usual answer is two endpoints. Liveness says the process is running and should not be restarted. Readiness says this instance can do useful work, and checks only what is local to it: configuration loaded, connection pool created, caches warm. Shared dependencies are monitored through the SLI, not through per-instance health checks.

Emit the SLI from the code

The load balancer SLI from Part 2 sees status codes. It cannot tell a successful checkout from a 200 that carries an error message in the body, and it cannot split by customer tier or by operation. The application can. For each request, write one structured log line with the operation, the outcome as the business defines it, the duration, and the trace ID. On AWS, the CloudWatch embedded metric format turns that same log line into metrics with no extra API call:

{
  "_aws": {
    "Timestamp": 1787130000000,
    "CloudWatchMetrics": [{
      "Namespace": "Checkout",
      "Dimensions": [["Operation"]],
      "Metrics": [
        { "Name": "Requests", "Unit": "Count" },
        { "Name": "Failures", "Unit": "Count" },
        { "Name": "Duration", "Unit": "Milliseconds" }
      ]
    }]
  },
  "Operation": "PlaceOrder",
  "Requests": 1,
  "Failures": 0,
  "Duration": 184,
  "trace_id": "1-66c2f1d0-3f1b2c4d5e6f7a8b9c0d1e2f",
  "customer_tier": "enterprise"
}

Keep high-cardinality fields such as the trace ID and customer as log properties, not metric dimensions; every unique dimension value is a separate billed metric. This is the instrumentation that belongs in the service template from Part 1, so that developers get it without asking.

Make change the safe path

Part 2's ledger showed a single deploy consuming 31 percent of a month's budget. The code-side defenses are the highest-return work on this list.

Progressive rollout with automatic rollback. Send the new version 10 percent of traffic, compare its error ratio and latency with the old version, and roll back without a human if it is worse. CodeDeploy, ECS deployments, and Lambda aliases can all stop and roll back on a CloudWatch alarm. A bad release then costs minutes of 10 percent of traffic.
Separate deploy from release. Ship code dark behind a feature flag and turn it on gradually. Turning a flag off is faster than any rollback.
Backward-compatible schema changes. Expand, migrate, contract: add the new column, write to both, move readers, then remove the old one in a later release. A migration that requires old and new code never to run side by side makes every rollout an outage and every rollback impossible.

Alerts as code: page on burn rate, not CPU

The last piece of code an SRE owns is the alerting itself. The alarm set on a typical AWS account pages for causes: CPU above 80 percent, memory above 90, any 5xx count above 10. It is noisy where nothing is wrong (an autoscaling group at 95 percent CPU that is meeting its latency target) and silent where something is (a deploy failing 2 percent of checkouts with every host metric flat). The SLO from Part 2 gives a better condition to page on: are we spending the error budget fast enough that, left alone, we miss the objective? That speed is the burn rate.

burn rate = observed error ratio / (1 - SLO)

SLO 99.9%  ->  budget ratio 0.001
error ratio 0.1%   ->  burn rate 1      ->  budget lasts 30 days
error ratio 0.6%   ->  burn rate 6      ->  budget lasts 5 days
error ratio 1.44%  ->  burn rate 14.4   ->  budget lasts 50 hours
error ratio 100%   ->  burn rate 1000   ->  budget lasts 43 minutes

An alert needs a threshold and a window, and the two together decide how much budget is gone by the time it fires:

budget consumed at alert = burn rate x alert window / SLO window

14.4 x 1 hour  / 720 hours =  2%
6    x 6 hours / 720 hours =  5%
1    x 3 days  / 30 days   = 10%

Those three rows are the recommendation from the alerting chapter of the Google SRE Workbook, and they are a sensible default for a 30-day SLO: page when 2 percent of the budget went in an hour or 5 percent went in six hours, and open a ticket when 10 percent went in three days. Fast, severe problems get a fast page. Slow leaks, the kind a threshold alarm never catches, get a ticket for the next working day.

Why two windows

A single long window has a reset problem. If errors ran at 15 percent for ten minutes and then stopped, the one-hour average stays above the 14.4x threshold for most of the following hour. The alarm keeps firing at an engineer who is looking at a healthy system.

The fix is to require a second, short window to be over the same threshold at the same time. The workbook's guideline is a short window one twelfth of the long one. The long window says the spend is significant; the short window says it is still happening. Together:

Action	Long window	Short window	Burn rate	Error ratio at 99.9%	Budget consumed
Page	1 hour	5 minutes	14.4	1.44%	2%
Page	6 hours	30 minutes	6	0.6%	5%
Ticket	3 days	6 hours	1	0.1%	10%

Detection is fast where it needs to be. In a total outage the one-hour average crosses 1.44 percent after about 52 seconds, and the five-minute window is already over. When the outage ends, the five-minute window drops below the threshold within minutes and the page clears, even though the hour is still polluted.

Building it in CloudWatch with Terraform

CloudWatch has the two pieces this needs: metric math alarms, to alarm on an error ratio instead of a raw count, and composite alarms, to AND the long and short windows. One metric alarm per window, with a single evaluation period as long as the window, and no actions attached:

locals {
  slo    = 0.999
  budget = 1 - local.slo
  lb     = "app/checkout/0123456789abcdef"

  windows = {
    fast_long  = { period = 3600,  burn = 14.4 }
    fast_short = { period = 300,   burn = 14.4 }
    slow_long  = { period = 21600, burn = 6 }
    slow_short = { period = 1800,  burn = 6 }
  }

  elb_metrics = {
    req  = "RequestCount"
    t5xx = "HTTPCode_Target_5XX_Count"
    e5xx = "HTTPCode_ELB_5XX_Count"
  }
}

resource "aws_cloudwatch_metric_alarm" "burn" {
  for_each = local.windows

  alarm_name          = "checkout-burn-${each.key}"
  alarm_description   = "Error ratio above ${each.value.burn}x burn over ${each.value.period}s"
  comparison_operator = "GreaterThanThreshold"
  threshold           = each.value.burn * local.budget
  evaluation_periods  = 1
  treat_missing_data  = "notBreaching"

  metric_query {
    id          = "ratio"
    expression  = "(FILL(t5xx, 0) + FILL(e5xx, 0)) / req"
    label       = "error ratio"
    return_data = true
  }

  dynamic "metric_query" {
    for_each = local.elb_metrics
    content {
      id = metric_query.key
      metric {
        namespace   = "AWS/ApplicationELB"
        metric_name = metric_query.value
        stat        = "Sum"
        period      = each.value.period
        dimensions  = { LoadBalancer = local.lb }
      }
    }
  }
}

The threshold is the burn rate multiplied by the budget ratio: 14.4 x 0.001 = 0.0144 for the fast pair, 0.006 for the slow pair. Change slo to 0.9995 and every threshold follows. Then the composite alarm, which is the only thing that pages:

resource "aws_cloudwatch_composite_alarm" "slo_page" {
  alarm_name        = "checkout-availability-slo-page"
  alarm_description = "Burning 30-day error budget at 14.4x (1h) or 6x (6h). Runbook: https://wiki.example.com/runbooks/checkout-slo"

  alarm_rule = join(" OR ", [
    "(ALARM(${aws_cloudwatch_metric_alarm.burn["fast_long"].alarm_name}) AND ALARM(${aws_cloudwatch_metric_alarm.burn["fast_short"].alarm_name}))",
    "(ALARM(${aws_cloudwatch_metric_alarm.burn["slow_long"].alarm_name}) AND ALARM(${aws_cloudwatch_metric_alarm.burn["slow_short"].alarm_name}))",
  ])

  alarm_actions = [aws_sns_topic.pager.arn]
  ok_actions    = [aws_sns_topic.pager.arn]
}

The OR inside one composite alarm also solves suppression. A hard outage satisfies both the fast and the slow pair; because they feed one alarm, the engineer gets one page, not two.

The three-day ticket tier is easier on the managed path. If the SLO lives in CloudWatch Application Signals, you attach burn rate look-back windows to it (up to 10,080 minutes, which is seven days) and alarm on the burn rate metric it publishes. The AWS threshold formula is X% x SLO interval length / look-back window, and it carries a trap: 14.4 is only correct for a 30-day interval. Two percent of a 28-day budget in one hour is 13.44. Derive thresholds from the percentages; never copy the multipliers.

The production readiness review

Put together, this is the review an SRE runs with developers before a service takes real traffic, and repeats when the architecture changes:

Does every outbound call have a timeout, and do the timeouts fit inside the caller's deadline?
Is there exactly one retry layer, with backoff and jitter, and are retried writes idempotent?
For each dependency: hard or soft, and what does the user see when it is down?
Do health checks avoid shared dependencies?
Does the service emit request, failure, and duration metrics per operation, and propagate a trace ID?
Is there an SLO, an owner for it, and a burn-rate alert that pages a named rotation with a runbook link?
Can a release be rolled back in under five minutes without a schema change?
What limit is reached first under ten times the traffic, and what happens then?

Failure modes and what to watch

Low traffic breaks burn-rate math. At 100 requests an hour, a 14.4x burn on a 99.9 percent SLO is 1.44 failed requests, so two unlucky timeouts page someone. The options in the SRE Workbook are synthetic traffic, aggregating small services into one SLO, a lower target, or a longer window. In metric math you can gate on volume with IF(req > 200, ratio, 0), at the cost of blindness below the gate.

Missing data is not good news by default. treat_missing_data = "notBreaching" is right for a ratio that vanishes with no traffic and wrong if traffic vanished because DNS is broken. Pair the SLO alarm with a plain alarm on RequestCount falling to zero in hours when it never should.

Buckets are not sliding windows. A CloudWatch datapoint with a one-hour period is a bucket, so the long-window alarm can react later than an ideal sliding calculation. The short window keeps detection of severe incidents fast; treat slow-burn timing as approximate.

Resilience code that is never exercised. A circuit breaker that has not opened in production is a guess. Fault injection, in a test environment first and then in production inside the error budget, is how the fallback paths earn trust.

Everything above assumes the platform holds. Timeouts and retries are written against a provider that is mostly working. What that provider actually promises, where its failure domains are, and what changes when the provider is your own data center is Part 4.