Alerting on Service Level Objectives is a solved problem in the sense that the solution exists, has been published, and works reliably when implemented correctly. It is an unsolved problem in the sense that most organisations implementing SLOs do not implement it correctly — they alert on threshold breaches of individual metrics rather than on error budget consumption rates, and they use single measurement windows rather than the dual-window AND-gate structure that distinguishes a genuine sustained burn from a transient spike.
The cost of getting this wrong is operational. An alerting system that pages too frequently trains engineers to treat pages as background noise — the alert equivalent of a car alarm that everyone has learned to ignore. An alerting system that pages too infrequently allows budget burns to proceed undetected until the SLO is already breached. Both failure modes are self-reinforcing: the first erodes the operational culture that makes on-call sustainable; the second erodes the error budget that makes deployment velocity permissible.
The Google SRE Workbook's multi-window burn rate alerting model solves both failure modes simultaneously. This post derives it formally — not as a recipe to follow, but as a logical structure to understand. Understanding why the model works is what enables practitioners to adapt it correctly to their specific SLO windows, service characteristics, and operational environments.
The Single-Window Alerting Trade-off
Before deriving the multi-window model, the problem it solves must be precise. The fundamental constraint of single-window alerting is this: for any fixed measurement window W and any fixed alerting threshold T, the alert exhibits an irreducible trade-off between sensitivity (detecting genuine budget consumption early) and specificity (not firing on transient spikes that self-correct).
────────────────────────────────────────────────────────────────────────────
SINGLE-WINDOW ALERTING: THE UNAVOIDABLE TRADE-OFF
Given:
SLO window: 28 days
SLO target: 99.9% (error budget = 0.1%)
Single measurement window W: T minutes
SHORT WINDOW (W = 5 minutes):
Sensitivity: HIGH — detects burn rate changes quickly
Specificity: LOW — every brief error spike triggers the alert
Example: A 30-second traffic anomaly causing 5% errors for 30s
Error rate in 5-minute window: (0.3/5) × 5% = ~0.3%
This is 3× the error budget rate → alert fires
But: the anomaly self-corrected; budget consumed was negligible
Result: PAGE AT 2 AM for a self-correcting 30-second spike
LONG WINDOW (W = 1 hour):
Sensitivity: LOW — requires sustained errors to trigger
Specificity: HIGH — short spikes diluted across the window
Example: A 10-minute complete outage (100% error rate)
Error rate in 60-minute window: 10/60 = 16.7% → 167× budget rate
This exceeds any reasonable threshold → alert fires
But: by the time the 60-minute window accumulates enough signal,
4.2% of the 28-day budget has already been consumed
At severe burns, detection is too late to prevent material impact
FUNDAMENTAL CONSTRAINT:
For any single window W:
Shorter W → more false positives (specificity decreases)
Longer W → slower detection (sensitivity decreases)
No single W value optimises both simultaneously
────────────────────────────────────────────────────────────────────────────
The dual-window AND-gate escapes this constraint by using two windows simultaneously: a long window to confirm that a burn is sustained (not a transient spike), and a short window to confirm that the burn is still ongoing (not a historical artifact). The AND of the two conditions provides the noise suppression of the long window with the recency validation of the short window.
Formal Definition: Burn Rate
Before the AND-gate logic can be analysed, the burn rate must be defined precisely.
────────────────────────────────────────────────────────────────────────────
BURN RATE: FORMAL DEFINITION
Let:
E = error budget as a proportion (e.g., 0.001 for 99.9% SLO)
W_SLO = SLO measurement window in hours (e.g., 672h for 28 days)
r(t) = error rate at time t (proportion of requests failing)
Budget consumption rate at 1× burn:
r_baseline = E / W_SLO
(the error rate at which the budget would be exactly exhausted at window end)
For 99.9% SLO over 28 days:
r_baseline = 0.001 / 672 = 0.00000149 per hour
= 0.00149 per 1000 hours (negligible hourly)
Burn rate B at time t:
B(t) = r(t) / r_baseline
Interpretation:
B = 1 → error rate on pace to exactly exhaust budget in 28 days
B = 14 → error rate 14× r_baseline; budget exhausted in 28/14 = 2 days
B = 6 → budget exhausted in 28/6 ≈ 4.7 days
B = 3 → budget exhausted in 28/3 ≈ 9.3 days
B = 0 → no errors; budget not being consumed
Time to budget exhaustion at constant burn rate B:
t_exhaustion = W_SLO / B
B = 14: t_exhaustion = 672h / 14 = 48h (2 days)
B = 6: t_exhaustion = 672h / 6 = 112h (~4.7 days)
B = 3: t_exhaustion = 672h / 3 = 224h (~9.3 days)
B = 1: t_exhaustion = 672h / 1 = 672h (28 days, exactly at window end)
IMPORTANT DISTINCTION:
t_exhaustion is the time until budget runs out IF burn continues.
The detection window (1h, 6h) is the window used to MEASURE B(t).
These are not the same. Detection at B=14 over a 1-hour window means:
→ The burn has been at 14× for the last 1 hour
→ If it continues, budget exhausts in ~47 more hours
→ The 1 hour of burning has consumed 14/672 ≈ 2.1% of total budget
────────────────────────────────────────────────────────────────────────────
The AND-Gate: Formal Analysis
For each alert tier, two windows are defined: a long window W_L that estimates the sustained burn rate, and a short window W_S that validates the burn is current. The alert fires if and only if the burn rate exceeds threshold B_threshold in BOTH windows simultaneously.
────────────────────────────────────────────────────────────────────────────
AND-GATE FORMAL SPECIFICATION
Alert condition:
FIRE if: B(W_L) > B_threshold AND B(W_S) > B_threshold
Where:
B(W) = mean burn rate over window W
W_L = long window (1h, 6h, 1d, 3d depending on tier)
W_S = short window (5m, 30m, 2h, 6h depending on tier)
NOISE SUPPRESSION PROPERTY:
A transient spike of duration d < W_S that creates burn rate B_spike:
B(W_S) = (d/W_S) × B_spike + ((W_S-d)/W_S) × 0
= (d/W_S) × B_spike
For the AND condition to fire:
(d/W_S) × B_spike > B_threshold
Required spike: B_spike > B_threshold × (W_S/d)
Example: W_S = 5 minutes, B_threshold = 14, spike duration d = 30 seconds
Required spike rate: 14 × (5/0.5) = 14 × 10 = 140× burn rate
A 30-second spike must create a 140× burn rate to trigger the short window
At 99.9% SLO, 140× burn rate = 14% error rate for 30 seconds
Most transient anomalies fall below this threshold → noise suppressed
RECENCY VALIDATION PROPERTY:
A historical burn event that ended T hours ago:
For T > W_S: B(W_S) ≈ 0 → AND condition fails → no stale alert
For T < W_S: B(W_S) > 0 proportional to recency → may still page
For T > W_L: B(W_L) ≈ 0 → AND condition fails definitively
This property ensures that a burn event that resolved T hours ago
stops generating pages once T > W_S — typically within 5 minutes
of the error rate returning to normal.
MINIMUM DETECTION LATENCY:
Minimum time before alert fires after burn begins (at constant B > B_threshold):
t_min_detection = W_S (must accumulate W_S of signal in short window)
For the 14× tier with W_S = 5 minutes:
t_min_detection = 5 minutes
Budget consumed before detection: 14 × (5/672/60) × 100% ≈ 0.17%
Fraction of 28-day budget consumed before page: 17% (of total 0.1% budget)
────────────────────────────────────────────────────────────────────────────
Deriving the Four Tiers: Why These Multipliers?
The four burn rate thresholds — 14×, 6×, 3×, 1× — are not arbitrary. Each is calibrated to a specific combination of urgency (time to budget exhaustion) and detection window (how much budget is consumed before the alert fires).
────────────────────────────────────────────────────────────────────────────
TIER DERIVATION: CALIBRATION CRITERIA
DESIGN CRITERIA FOR EACH TIER:
1. Time to budget exhaustion at threshold burn rate (urgency)
2. Budget consumed before detection (detection cost)
3. Appropriate organisational response (page vs ticket)
4. Detection window feasibility (window short enough to fire early)
TIER 1 — PAGE IMMEDIATELY (14× burn, 1h/5m windows)
Urgency:
t_exhaustion = 28 days / 14 = 2 days
"If nothing changes, SLO is breached in 2 days"
→ Requires immediate human response; cannot wait for business hours
Detection cost:
Budget consumed before short-window detection (5 min): ~0.17% of budget
Budget consumed before long-window confirmation (1 hr): ~2.1% of budget
This is acceptable: detection while most of the budget is still intact
Why 14×, not 20× or 10×:
20× → t_exhaustion = 1.4 days; detection before 1 hour often impossible
10× → t_exhaustion = 2.8 days; urgency threshold ambiguous for paging
14× provides a clean "2-day" exhaustion horizon with 1-hour detection window
TIER 2 — PAGE (WITHIN 30 MIN) (6× burn, 6h/30m windows)
Urgency:
t_exhaustion = 28 days / 6 = 4.67 days
"Budget exhausted in under 5 days; needs response today"
→ Requires on-call page; response within 30 minutes
Detection cost:
Budget consumed before short-window detection (30 min): ~0.74% of budget
Budget consumed before long-window confirmation (6 hr): ~8.9% of budget
At 6× burn, 6-hour confirmation costs ~8.9% of total budget;
acceptable given the lower urgency vs Tier 1
Why 6h long window:
6h window smooths out periods of elevated but not catastrophic error rate
A service spiking to high error rates for 1-2 hours dilutes to below
threshold in a 6-hour window → prevents Tier 2 pages for recoverable spikes
TIER 3 — TICKET (BUSINESS HOURS) (3× burn, 1d/2h windows)
Urgency:
t_exhaustion = 28 days / 3 = 9.3 days
"Budget exhausted in ~9 days; needs attention this week"
→ Does not warrant waking someone up; create a ticket
Detection cost:
Budget consumed before 2h short-window detection: ~0.89% of budget
At 3× sustained burn, this is manageable
The 1-day long window provides very high noise suppression
Why 1-day long window:
A 3× burn rate is only ~3× the normal error rate
Day-to-day traffic variation can produce sustained periods at 2-3×
without representing a genuine reliability problem
1-day window ensures the ticket tier only fires on genuinely
sustained moderate burns, not daily traffic variation
TIER 4 — TREND REVIEW (1× burn, 3d/6h windows)
Urgency:
t_exhaustion = 28 days / 1 = 28 days (exactly exhausts at window end)
"On pace to breach SLO; needs review but not urgent"
→ Weekly SRE sync item; not a ticket or a page
Why 3-day long window:
1× burn rate is definitionally "on pace to exhaust at window end"
This is the background noise rate — the service is barely over its
error budget allocation
A 3-day window provides extremely high noise suppression;
only genuinely sustained 1× burns generate the trend alert
────────────────────────────────────────────────────────────────────────────
TIER SUMMARY TABLE (28-day window, 99.9% SLO)
Tier Multiplier Long Short t_exhaust Budget at Response
Window Window Long Det.
──── ────────── ────── ───── ───────── ───────── ──────────────
1 14× 1h 5m 2 days ~2.1% Page immediately
2 6× 6h 30m 4.7 days ~8.9% Page within 30min
3 3× 1d 2h 9.3 days ~10.7% Ticket this week
4 1× 3d 6h 28 days ~10.7% Weekly review
────────────────────────────────────────────────────────────────────────────
Complete Prometheus Implementation
# Multi-Window Burn Rate Alerting — Complete Implementation
# SLO: 99.9% availability over 28-day rolling window
# All four tiers with AND-gate dual windows
groups:
- name: slo.burnrate.recording
interval: 30s
rules:
# Base SLI: request success ratio
- record: sli:http_request_success:ratio_rate5m
expr: |
sum(rate(http_requests_total{
job="api-server",
status!~"5.."
}[5m]))
/
sum(rate(http_requests_total{job="api-server"}[5m]))
# Burn rate at multiple windows — the key recording rules
# These power all four alert tiers
- record: slo:error_budget_burn_rate:ratio_rate5m
expr: (1 - sli:http_request_success:ratio_rate5m) / (1 - 0.999)
- record: slo:error_budget_burn_rate:ratio_rate30m
expr: |
(1 - sum(rate(http_requests_total{status!~"5.."}[30m]))
/ sum(rate(http_requests_total[30m])))
/ (1 - 0.999)
- record: slo:error_budget_burn_rate:ratio_rate1h
expr: |
(1 - sum(rate(http_requests_total{status!~"5.."}[1h]))
/ sum(rate(http_requests_total[1h])))
/ (1 - 0.999)
- record: slo:error_budget_burn_rate:ratio_rate2h
expr: |
(1 - sum(rate(http_requests_total{status!~"5.."}[2h]))
/ sum(rate(http_requests_total[2h])))
/ (1 - 0.999)
- record: slo:error_budget_burn_rate:ratio_rate6h
expr: |
(1 - sum(rate(http_requests_total{status!~"5.."}[6h]))
/ sum(rate(http_requests_total[6h])))
/ (1 - 0.999)
- record: slo:error_budget_burn_rate:ratio_rate1d
expr: |
(1 - sum(rate(http_requests_total{status!~"5.."}[1d]))
/ sum(rate(http_requests_total[1d])))
/ (1 - 0.999)
- record: slo:error_budget_burn_rate:ratio_rate3d
expr: |
(1 - sum(rate(http_requests_total{status!~"5.."}[3d]))
/ sum(rate(http_requests_total[3d])))
/ (1 - 0.999)
# Budget remaining — the dashboard and gate metric
- record: slo:error_budget_remaining:ratio
expr: |
1 - (
(1 - sli:http_request_success:ratio_rate5m)
/ (1 - 0.999)
)
- name: slo.burnrate.alerts
rules:
# ── TIER 1: PAGE IMMEDIATELY ─────────────────────────────────────────
# 14× burn, 1h long / 5m short AND-gate
# Budget exhausts in ~2 days if burn continues
- alert: SLO_BurnRate_P1_14x
expr: |
slo:error_budget_burn_rate:ratio_rate1h > 14
AND
slo:error_budget_burn_rate:ratio_rate5m > 14
for: 2m
labels:
severity: critical
tier: "1"
burn_multiplier: "14"
annotations:
summary: >
SLO burn rate at 14× — budget exhausts in ~2 days if sustained.
Budget remaining: {{ with query "slo:error_budget_remaining:ratio" }}
{{ . | first | value | humanizePercentage }}{{ end }}
runbook: "https://wiki.internal/sre/runbooks/slo-burn-p1"
dashboard: "https://grafana.internal/d/slo-burn/burn-rate"
# ── TIER 2: PAGE WITHIN 30 MINUTES ───────────────────────────────────
# 6× burn, 6h long / 30m short AND-gate
# Budget exhausts in ~4.7 days if burn continues
- alert: SLO_BurnRate_P2_6x
expr: |
slo:error_budget_burn_rate:ratio_rate6h > 6
AND
slo:error_budget_burn_rate:ratio_rate30m > 6
for: 5m
labels:
severity: page
tier: "2"
burn_multiplier: "6"
annotations:
summary: >
SLO burn rate at 6× — budget exhausts in ~4.7 days if sustained.
runbook: "https://wiki.internal/sre/runbooks/slo-burn-p2"
# ── TIER 3: TICKET (BUSINESS HOURS) ──────────────────────────────────
# 3× burn, 1d long / 2h short AND-gate
# Budget exhausts in ~9.3 days if burn continues
- alert: SLO_BurnRate_Ticket_3x
expr: |
slo:error_budget_burn_rate:ratio_rate1d > 3
AND
slo:error_budget_burn_rate:ratio_rate2h > 3
for: 15m
labels:
severity: warning
tier: "3"
burn_multiplier: "3"
annotations:
summary: >
SLO burn rate at 3× — budget exhausts in ~9.3 days if sustained.
Review in next engineering standup.
# ── TIER 4: WEEKLY TREND REVIEW ──────────────────────────────────────
# 1× burn, 3d long / 6h short AND-gate
# Budget on pace to exhaust at window end
- alert: SLO_BurnRate_Trend_1x
expr: |
slo:error_budget_burn_rate:ratio_rate3d > 1
AND
slo:error_budget_burn_rate:ratio_rate6h > 1
for: 1h
labels:
severity: info
tier: "4"
burn_multiplier: "1"
annotations:
summary: >
SLO burn on pace to exhaust budget at 28-day window end.
Review budget trend in weekly SRE sync.
Istio STRICT mTLS: The SLI Source Matters
In environments running Istio in STRICT mTLS mode, the choice of SLI source — application metrics versus Envoy proxy metrics — determines whether the burn rate calculation captures all failure modes or only the ones the application can see.
# Istio-aware burn rate: captures mTLS-layer rejections
# that application-level metrics cannot see
- record: sli:http_request_success:ratio_rate5m
expr: |
sum(rate(istio_requests_total{
reporter="destination",
response_code!~"5.."
}[5m]))
/
sum(rate(istio_requests_total{
reporter="destination"
}[5m]))
# Why reporter="destination" (not "source"):
# source = measured at the caller's sidecar
# destination = measured at the called service's sidecar
#
# mTLS handshake failures at the destination sidecar:
# → Appear as errors in destination metrics
# → Do NOT appear in source metrics (connection never established)
# → Do NOT appear in application metrics (request never reached app)
#
# Using reporter="destination" captures:
# ✓ Application-level 5xx errors
# ✓ mTLS policy rejection errors
# ✓ Sidecar proxy errors
#
# Using reporter="source" OR application metrics misses:
# ✗ mTLS-layer rejections (certificates expired, policy violation)
# ✗ These create phantom budget consumption invisible to burn rate
Common Antipatterns
The Single-Window Implementation antipattern → Implementing burn rate alerting with only the long window, skipping the short-window AND-gate. This produces alerts that fire on historical burn events long after the error rate has recovered — operators respond to an ongoing emergency that resolved thirty minutes ago. The short window is not optional; it is the recency validation that prevents this failure mode.
The Threshold Without Derivation antipattern → Using the 14×/6×/3×/1× thresholds without verifying they are calibrated to your actual SLO window. The thresholds in this post are derived for a 28-day window. For a 7-day window, a 14× burn rate exhausts the budget in 12 hours — which changes the appropriate detection window and the response urgency. Derive your thresholds from your window.
The Missing Tier 4 antipattern → Implementing only the page tiers (Tier 1 and 2) and skipping the ticket and trend tiers. Tier 4 is the early warning system for budget degradation that has not yet reached urgent levels. A service that runs at sustained 1× burn for two weeks will exhaust its budget at the 28-day window end — but will never have triggered a page. Without Tier 4, this pattern is invisible until the budget is gone.
The App-Metric SLI in Istio antipattern → Computing burn rates from application-level HTTP response codes in an Istio service mesh environment. As described above, mTLS-layer failures are invisible to application metrics. In STRICT mTLS mode, a certificate rotation gone wrong or a PeerAuthentication policy misconfiguration will consume error budget without generating any application-level error signals. Use Envoy proxy metrics.
The
forDuration Misconfiguration antipattern → Setting theforduration on Tier 1 alerts to more than 2 minutes. Theforfield adds a minimum firing duration — it requires the condition to be true for the specified time before the alert fires. For Tier 1 at 14× burn rate, every minute offorduration adds burn at 14× before the page is sent. Afor: 15mon a Tier 1 alert delays notification by 15 minutes at a rate that consumes significant budget.
Maturity Progression
────────────────────────────────────────────────────────────────────────────
STAGE ALERTING MATURITY STATE NORTH STAR SIGNAL
────────────────────────────────────────────────────────────────────────────
Reactive Threshold alerts on individual Alert volume is high.
metrics. No SLO concept. Engineers filter noise
No burn rate concept. by experience. P95
latency alerts fire
on every deploy.
Defined SLOs defined. Single-window First burn rate alert
burn rate alerts implemented implemented. Some noise
for Tier 1 only. App-level reduction vs. threshold
SLI source. alerting.
Measured Dual-window AND-gate for all Tier 2 and 3 alerts
four tiers. Istio proxy SLI. implemented. Alert
Budget remaining recorded. volume tracking.
False positive rate tracked. False positive rate
measured and declining.
Optimised All tiers calibrated to actual Zero false positives
SLO window. Short window on Tier 1 in last
tuned for service traffic 30 days. Tier 4 trend
characteristics. Multi-service alerts surfacing budget
burn rate dashboard. degradation proactively.
Generative Burn rate alerting extended Burn rate model used
to infrastructure layer: to govern deployment
database, cache, queue. gates. Budget policy
Alert quality metrics in tiers automated from
SRE quarterly report. burn rate state.
────────────────────────────────────────────────────────────────────────────
Five Action Items for This Week
Verify your Tier 1 alert threshold is calibrated to your actual SLO window. Take your SLO window in hours, divide by 14. That is the time-to-exhaustion at your Tier 1 threshold. If your SLO window is 7 days (168 hours), your 14× burn rate exhausts the budget in 12 hours — which changes whether a 1-hour long window is the right detection window.
Check that your current alerting implementation uses the AND-gate, not an OR-gate or single window. Pull your alert rules and verify that both the long-window and short-window conditions are required simultaneously. A single-window rule or an OR-gate rule does not have the noise suppression properties derived in this post.
Switch your SLI source from application metrics to Istio Envoy proxy metrics if you are running a service mesh. Query
istio_requests_total{reporter="destination"}versus your application-level HTTP response metrics for the same service over the same window. If the numbers differ, the delta is budget consumption that your current alerting cannot see.Add the budget remaining recording rule and verify it is accessible from your deployment gate. The budget remaining metric is the input to the error budget policy tier classification. If your Argo CD PreSync hook cannot query it, the deployment gate has no connection to the SLO state it is supposed to enforce.
Measure your Tier 1 false positive rate over the last 30 days. Count the number of Tier 1 alerts that fired and then resolved without human intervention within 30 minutes. Each of those is a false positive. If the count is above zero, examine whether the AND-gate is correctly configured or whether the short-window threshold is too sensitive for your service's normal traffic variance.
"The burn rate model is not a set of magic numbers. It is a logical structure derived from the properties you want your alerting system to have: page at high urgency, not on transient spikes; ticket at moderate urgency, reliably; surface trends before they become emergencies. Understanding the derivation — rather than just copying the thresholds — is what enables you to adapt the model to services that are unusual: high-cardinality SLIs, multi-modal traffic patterns, services with known daily error rate variance. The AND-gate is not Google's magic. It is the logical solution to a well-specified alerting design problem."
