SLO, Error Budgets & Burn Rate Alerting

TL;DR

• SLI measures performance; SLO sets targets; SLA contracts with consequences. Hierarchy: SLA > SLO > SLI.
• Error budget is the allowed unreliability (1 − SLO%). A 99.9% SLO allows 43.2 minutes/month of downtime.
• Multi-window multi-burn-rate alerting uses two windows (long for sustained issues, short for confirmation) to avoid false positives and catch real problems fast.
• Burn rate measures how fast error budget is consumed. 14.4x burn rate exhausts the 30-day budget in 2 days.
• Error budget without policy enforcement is useless—define deployment freezes and reliability work thresholds.
• Measure SLIs at the load balancer or client layer, not in application code, to capture infrastructure failures.

Core Concepts

SLI (Service Level Indicator)

Quantitative, measurable performance metrics. Examples: request success rate (200-399 / total), P99 latency < 300ms, throughput ≥ 10k req/s.

SLO (Service Level Objective)

Target for an SLI over a window (typically 30 days rolling). Examples: 99.9% availability, P95 latency < 200ms. Internal targets with safety buffer above SLA.

SLA (Service Level Agreement)

Legal contract with external customers. Specifies consequences (credits, penalties) for missing SLO. Always less aggressive than internal SLO to maintain buffer zone.

Error Budget

Allowed unreliability: Error Budget = 1 − SLO%. At 99.9% SLO, 0.1% budget = 43.2 min/month. Spent on outages, deployments, maintenance, and degradations.

Burn Rate

Ratio of actual error rate to SLO error rate. A 1x burn rate consumes budget at the expected monthly pace. 14.4x exhausts 30-day budget in 2 days.

SLO/SLI/SLA Hierarchy

┌────────────────────────────────────────────┐
│ SLA (External Contract)                    │
│ "99.9% uptime or customer gets credits"    │
│                                            │
│  ┌──────────────────────────────────────┐  │
│  │ SLO (Internal Target)                │  │
│  │ "99.95% availability"                │  │
│  │ (stricter, provides buffer)          │  │
│  │                                      │  │
│  │  ┌──────────────────────────────┐    │  │
│  │  │ SLI (Measurement)            │    │  │
│  │  │ successful_requests /        │    │  │
│  │  │ total_requests               │    │  │
│  │  │ measured at load balancer    │    │  │
│  │  └──────────────────────────────┘    │  │
│  └──────────────────────────────────────┘  │
└────────────────────────────────────────────┘

Comparison: SLO Targets & Error Budgets

Multi-Window Multi-Burn-Rate Alerting

Burn rate = (actual error rate) / (SLO error rate). Two time windows prevent false positives and confirm ongoing issues.

Why two windows?

• Long window detects sustained problems (avoids noise from brief spikes).
• Short window confirms issue still ongoing (avoids stale alerts).
• Both must breach simultaneously → high-confidence signal.

Error Budget Policy

Configuration Reference: Prometheus Alerting Rules

Multi-window multi-burn-rate SLO alert configuration:

groups:
- name: slo-alerts
  interval: 30s
  rules:
  # PAGE: 14.4x burn rate (exhausts budget in 2 days)
  - alert: HighErrorBudgetBurn_14_4x
    expr: |
      (sum(rate(http_requests_total{code=~"5.."}[1h])) / sum(rate(http_requests_total[1h]))) > 0.00144
      AND
      (sum(rate(http_requests_total{code=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))) > 0.00144
    for: 0m
    labels:
      severity: page
      budget_impact: high
    annotations:
      summary: "Critical error budget burn (14.4x) — 2-day depletion"
      description: "Error budget will exhaust in 2 days at current rate"

  # PAGE: 6x burn rate (exhausts budget in 5 days)
  - alert: HighErrorBudgetBurn_6x
    expr: |
      (sum(rate(http_requests_total{code=~"5.."}[6h])) / sum(rate(http_requests_total[6h]))) > 0.0006
      AND
      (sum(rate(http_requests_total{code=~"5.."}[30m])) / sum(rate(http_requests_total[30m]))) > 0.0006
    for: 0m
    labels:
      severity: page
    annotations:
      summary: "High error budget burn (6x) — 5-day depletion"

  # TICKET: 3x burn rate (exhausts budget in 10 days)
  - alert: MediumErrorBudgetBurn_3x
    expr: |
      (sum(rate(http_requests_total{code=~"5.."}[24h])) / sum(rate(http_requests_total[24h]))) > 0.0003
      AND
      (sum(rate(http_requests_total{code=~"5.."}[2h])) / sum(rate(http_requests_total[2h]))) > 0.0003
    for: 0m
    labels:
      severity: ticket
    annotations:
      summary: "Medium error budget burn (3x) — 10-day depletion"

Common Pitfalls

Interview Questions

Q — Your API has a 99.9% availability SLO over 30 days. Last month: 15-min outage, 20-min degradation (50% errors), and 5-min outage. Did you meet SLO? Design alerting to catch faster.

— Budget = 43.2 min/month. Consumed: 15 + (20 × 0.5) + 5 = 30 min = 69.4%. SLO met (30 < 43.2) but barely (13.2 min remaining). Multi-window alert detects 100% errors within 5 min at 14.4x burn rate. 50% sustained errors caught at 6x burn rate over 6h + 30m. Measure SLI at load balancer to capture infrastructure failures. Dashboard shows real-time budget, burn rate trend, incident breakdown. Document error budget policy for deployment freeze at 20%.

Q — How do you prevent alert fatigue from brief traffic spikes that self-resolve?

— Use multi-window multi-burn-rate: both long and short windows must breach simultaneously. A 5-minute spike won’t breach the 1-hour long window. Only sustained issues trigger alerts. Example: 14.4x burn rate requires breach over 1h AND 5m. Catches real problems, filters transient noise.

Q — Team wants a 99.99% SLO to sound impressive to customers. What do you say?

— 99.99% = 4.3 minutes downtime/month. For most services, this is unachievable without extreme cost (multi-region, redundant components, constant capacity). Better approach: Set realistic 99.9% SLO internally, commit 99.5% in SLA to customers (buffer zone). Focus on reliability outcomes, not optics. If 99.99% is genuinely needed (critical payments system), budget for it explicitly—it’s expensive.