Observability and SLOs: Error Budgets That Get Met
Updated: 2026-07-12
SLOs and error budgets only work when the budget drives real decisions. A feature freeze that triggers on exhaustion, deploy velocity that adjusts to consumption. With two or three well-chosen SLIs, a clear freeze policy, and simple tools like Prometheus with Sloth, a team can sustainably balance velocity and reliability in production.
SLOs (Service Level Objectives) and error budgets are classic SRE concepts popularised by Google. Most mid-size teams know them, many "have" them, few genuinely manage them. The difference is whether the error budget informs decisions: whether a feature freeze triggers when the budget is exhausted, whether deploy velocity adjusts when it is consumed fast. This article is about making it really work, not just documenting.
Key takeaways
-
SLI measures something user-relevant; SLO defines the objective; the error budget is the gap between 100% and the SLO.
-
The policy applied when budget is exhausted is where real value plays — if there are no consequences, the SLO does not exist.
-
Choosing good SLIs is the hardest part: purely infra SLIs (CPU, memory) do not measure user experience.
-
The most common anti-patterns are aspirational SLOs, not respecting the freeze, and SLOs without owners.
-
Multi-window multi-burn-rate alerts are the standard to avoid alert fatigue without losing real signal.
The basic concept
Three definitions to keep clear:
-
SLI (Service Level Indicator): metric measuring something user-relevant. E.g. "requests completing in <500ms", "availability" (uptime).
-
SLO (Service Level Objective): target for the SLI. E.g. "99.9% of requests in <500ms over 30 days".
-
Error budget: the gap between 100% and the SLO. If SLO is 99.9%, budget is 0.1% = 43 minutes per month.
The powerful part: if the budget is exhausted, you stop deploying new features and focus on stability.
Starting without ceremony
You do not need an SLO committee. For any service:
-
Pick 2-3 relevant SLIs: p99 latency on critical endpoints, availability (successful requests / total), freshness (data updated within X time) when applicable.
-
Define the SLO by discussing it with product: "what latency would degrade the experience?" Target: 99-99.9% for critical services; 95-99% for less critical ones.
-
Period: rolling 30 days as the default.
That is enough to start. Sophistication comes later if needed.
Prometheus implementation
# SLI: fraction of successful requests in <500ms
- record: service:sli:success_fast:ratio_rate5m
expr: |
sum(rate(http_requests_total{service="api", status!~"5..", duration_bucket="0.5"}[5m]))
/
sum(rate(http_requests_total{service="api"}[5m]))
# Error budget burn rate
- record: service:error_budget:burn_rate_1h
expr: |
(1 - service:sli:success_fast:ratio_rate5m) / 0.001
- alert: ErrorBudgetBurnFast
expr: service:error_budget:burn_rate_1h > 14.4
for: 2m
annotations:
summary: "Consuming 2% of monthly error budget per hour"
Multi-window multi-burn-rate alerts (Google SRE Workbook) are the standard: high short-window burn = urgent alert; sustained burn = moderate alert.
Error budget policy: the political part
Defining the SLO is easy. The policy applied when it is exhausted is where real value plays.
Typical tiered policy:
-
>50% consumed: caution, more deploy review.
-
>75% consumed: reduce non-essential changes.
-
>100% consumed: feature freeze, fixes only. Invest in stability.
-
>150% consumed: escalate to management, audit causes.
The key is that the policy is respected. If product overrides the freeze when triggered, the SLO does not exist in practice.
SLI design: the real work
Choosing good SLIs is the hardest part. Red flags: pure infra SLIs (CPU, memory), SLIs not correlating with an annoyed user, service-level aggregates mixing critical and non-critical endpoints, SLIs without clear time window.
Good SLIs measure user-visible metrics: public endpoint latency, errors the user sees, correct data. If the product changed, the SLI may have stopped correlating with real experience. A quarterly review is necessary.
Error budget as a conversation tool
The biggest value of SLOs and error budgets is aligning cross-discipline conversations:
-
Product understands that moving faster has a quantifiable cost in budget consumed.
-
Engineering has a clear threshold for requesting stabilisation time, without having to "sell" the argument.
-
Management sees metrics that correlate with customer satisfaction.
Without this, decisions about features versus stability are political. With budgets, they are data-driven.
The context of blameless post-mortems and SLOs are complementary tools: SLOs define when to act; post-mortems explain why things got that far and how to prevent recurrence.
Alert fatigue and SLOs
A common mistake is escalating every burn-rate alert to SEV-1. The result: teams woken up every few days by noise that requires no immediate action.
The right pattern:
-
Multi-window multi-burn-rate: alerts matter only when the burn is persistent across multiple windows.
-
Differentiated SEV levels: extreme burn rate = SEV-1; moderate burn = ticket for the next day.
-
Ticketing instead of paging for non-critical cases.
The goal is to alert when there is real action to take, not on every transient spike.
Tools
Typical stacks: Prometheus + Grafana + recording rules (DIY, flexible); Sloth[1] (sweet spot for small teams); Pyrra[2] (SLO as code + native UI); Datadog SLOs (integrated, easy, vendor lock-in).
Anti-patterns
Things that break SLOs in practice:
-
Aspirational SLOs without realism: 99.99% when the service is really at 99%. Budget always consumed, policy always ignored.
-
Not respecting the freeze when exhausted: immediately destroys mechanism credibility.
-
SLOs without clear owner: nobody maintains them, they go stale.
-
Too many SLOs: 20 SLOs per team = none gets real attention.
-
Manipulable SLIs: gaming destroys the meaning of the system.
Quarterly review
SLOs are not static. Each quarter, revisit them against the data from the period:
-
SLO too lax? (budget always available): tighten it so it demands something.
-
SLO too strict? (always exhausted): relax it or invest in architecture.
-
Does the SLI still represent real user experience? If the product changed, the correlation may have broken.
Conclusion
SLOs and error budgets work when applied rigorously, not as ornamental documentation. The test is simple: do decisions change based on the budget? If yes, the system works. If not, it is theatre. Start with 2-3 well-chosen SLIs, a clear freeze policy, and simple tools (Prometheus + Sloth). Sophistication comes later; first, respect the basics.