💡 Originally published on devtocash.com — where this guide stays updated. I write hands-on DevOps/SRE deep-dives there weekly.
What This Agent Does
An error budget agent answers the three questions every burn-rate alert forces an SRE to answer at 2 a.m.: how fast are we burning, what exactly is burning it, and does this justify slowing releases down? The burn-rate math lives in PromQL recording rules — deterministic, testable, no model involved. The LLM enters only for the two parts that are genuinely judgment work: attributing the burn to a specific route, status code, or release, and drafting the policy call (page, ticket, or a deploy-freeze recommendation) with the evidence attached. And the freeze itself ships as a pull request a human merges, never as an action the agent takes directly.
If your team already runs SLOs, you have the raw ingredients. What most teams don't have is the connective tissue: the burn-rate alert fires, someone opens four dashboards, eyeballs sum by (route) breakdowns, cross-references the deploy log, and then argues in Slack about whether "we're at 62% budget consumed on day 11" means anything. That twenty-minute ritual is the agent's job. The SLO fundamentals themselves are covered in the error budgets guide and the SLI/SLO implementation walkthrough — this post assumes those exist and builds the triage layer on top.
The Math Stays in Code: Multi-Window Burn Rates
Never let a language model compute a burn rate. It's arithmetic over counters, and arithmetic is what recording rules are for. The standard multi-window setup, for a 99.9% availability SLO over 30 days:
# slo-rules.yaml — recording rules the agent reads, never writes
groups:
- name: slo-checkout-api
rules:
- record: slo:error_ratio:rate5m
expr: |
sum(rate(http_requests_total{app="checkout-api",status=~"5.."}[5m]))
/ sum(rate(http_requests_total{app="checkout-api"}[5m]))
- record: slo:error_ratio:rate1h
expr: |
sum(rate(http_requests_total{app="checkout-api",status=~"5.."}[1h]))
/ sum(rate(http_requests_total{app="checkout-api"}[1h]))
- record: slo:error_ratio:rate6h
expr: |
sum(rate(http_requests_total{app="checkout-api",status=~"5.."}[6h]))
/ sum(rate(http_requests_total{app="checkout-api"}[6h]))
- record: slo:error_ratio:rate30d
expr: |
sum(rate(http_requests_total{app="checkout-api",status=~"5.."}[30d]))
/ sum(rate(http_requests_total{app="checkout-api"}[30d]))
Burn rate is just the error ratio divided by the budget ratio (0.001 for a 99.9% target). A burn rate of 1 means you'll spend exactly your budget over the window; 14.4 means the whole month's budget goes in two days. The classic paging thresholds from the Google SRE workbook — burn rate above 14.4 on both the 1h and 5m windows, or above 6 on both 6h and 30m — stay in Alertmanager, where they belong. The agent doesn't decide whether the alert fires; it picks up after it fires, the same division of labor as the Alertmanager MCP server: deterministic systems detect, the agent explains and recommends.
The Tools: Read-Only, Parameterized, Boring
The agent gets three tools, all read-only PromQL behind fixed templates — the same discipline as the Prometheus MCP server, narrowed further because this agent has exactly one job:
# budget_tools.py — the agent's entire read surface
import os, httpx
PROM = os.environ["PROM_URL"]
BUDGET_RATIO = 0.001 # 99.9% SLO
WINDOW_DAYS = 30
def _prom(query: str) -> float | None:
r = httpx.get(f"{PROM}/api/v1/query", params={"query": query}, timeout=15)
r.raise_for_status()
res = r.json()["data"]["result"]
return float(res[0]["value"][1]) if res else None
@mcp.tool()
def get_burn_state(app: str) -> dict:
"""Current burn rates across windows, budget consumed this period,
and projected days to exhaustion at the current 6h burn rate."""
windows = {w: _prom(f'slo:error_ratio:{w}{{app="{app}"}}') or 0.0
for w in ("rate5m", "rate1h", "rate6h", "rate30d")}
burn = {w: round(v / BUDGET_RATIO, 2) for w, v in windows.items()}
consumed = round(windows["rate30d"] / BUDGET_RATIO, 4) # fraction of budget
burn_6h = burn["rate6h"]
days_left = (round((1 - consumed) * WINDOW_DAYS / burn_6h, 1)
if burn_6h > 0 and consumed < 1 else None)
return {"burn_rates": burn, "budget_consumed": consumed,
"days_to_exhaustion_at_6h_rate": days_left}
@mcp.tool()
def burn_breakdown(app: str) -> dict:
"""Top error contributors over the last hour, by route and status."""
q = ('topk(8, sum by (route, status) '
f'(rate(http_requests_total{{app="{app}",status=~"5.."}}[1h])))')
r = httpx.get(f"{PROM}/api/v1/query", params={"query": q}, timeout=15)
r.raise_for_status()
return {"contributors": [
{"route": s["metric"].get("route", "?"),
"status": s["metric"].get("status", "?"),
"errors_per_s": round(float(s["value"][1]), 3)}
for s in r.json()["data"]["result"]]}
@mcp.tool()
def recent_deploys(app: str, hours: int = 24) -> list[dict]:
"""Deploys for this app in the window, from the CD system's API.
Returns [{version, deployed_at, author}] — read-only."""
...
Two design points earn their keep. First, get_burn_state returns days to exhaustion, not just percentages — "budget gone in 2.1 days at the current rate" is the number that moves a release decision, and computing it in code means the model can't fumble the projection. Second, recent_deploys is in the toolset because burn attribution without deploy correlation is astrology: the single most common answer to "what's burning the budget" is "the 14:20 deploy."
The Verdict: Forced Schema, Three Severities
One call, evidence in, structured verdict out:
VERDICT_TOOL = {
"name": "report_budget_verdict",
"description": "Triage an error-budget burn event.",
"input_schema": {
"type": "object",
"properties": {
"severity": {"enum": ["page", "ticket", "note"]},
"attribution": {"type": "string",
"description": "What is burning the budget: route, status, "
"and correlated deploy if any. Cite numbers."},
"freeze_recommended": {"type": "boolean"},
"reasoning": {"type": "string",
"description": "3-5 sentences. Reference budget consumed, "
"days to exhaustion, and burn-rate windows."},
},
"required": ["severity", "attribution",
"freeze_recommended", "reasoning"],
},
}
SYSTEM = (
"You triage SLO error-budget burn for an SRE team.\n"
"Severity: 'page' only when fast-burn windows (5m AND 1h) exceed 14.4, "
"or exhaustion is projected under 3 days. 'ticket' for slow burns that "
"will consume the budget before the window resets. 'note' otherwise.\n"
"Recommend a freeze ONLY when budget consumed exceeds 90%, or a "
"specific recent deploy is the dominant contributor and exhaustion is "
"projected inside the window. A freeze recommendation must name what "
"should be frozen (one service, not the org).\n"
"You have read-only tools. You cannot page anyone or freeze anything; "
"you produce a recommendation with evidence."
)
The severity ladder mirrors what the multi-window alerts already encode, and that redundancy is deliberate: the model is asked to re-derive the severity from the numbers, and the wrapper cross-checks it. If Alertmanager fired a fast-burn page and the agent says note, the wrapper escalates anyway and flags the disagreement for review. The agent can downgrade the noise level of your response to a burn; it is never allowed to downgrade the safety net.
Freezes Are Pull Requests, Not API Calls
The sharpest guardrail in this design: freeze_recommended: true does not stop anyone's deploys. It opens a pull request against the repo your CI already reads:
# .deploy-policy/freeze.yaml — proposed by the agent, merged by a human
frozen:
- app: checkout-api
reason: "Error budget 94% consumed on day 12 of 30. Fast burn (9.8x/6h)
attributed to /api/v1/payment route 500s beginning with deploy
v2.41.0 at 14:20 UTC. Projected exhaustion: 1.8 days."
until: "2026-08-20"
proposed_by: "error-budget-agent"
A ten-line GitHub Actions step in each service's deploy workflow greps this file and fails the run if its app is listed. That's the whole enforcement mechanism, and it has three properties an API-driven freeze doesn't: the recommendation arrives with reviewable evidence in the diff, the human decision is a merge with an audit trail, and unfreezing is git revert. This is the same argument as GitOps for AI agents — agents propose in a medium built for review — combined with the approval-gate principle that anything touching prod velocity deserves a human yes. A deploy freeze is an org-level write. It should cost a human one click, and it should never cost less.
The until date matters more than it looks. Open-ended freezes rot into permanent process; a freeze that expires forces the conversation the error budget exists to force — either reliability work happened and the budget recovered, or the team consciously extends it.
Shadow the Human First
Run the agent observe-only for two or three weeks before its verdicts reach anyone's pager or repo — the standard shadow mode sequence. Every burn-rate alert triggers a full triage run; verdicts land in a log channel. Then score them against what the humans actually did: every burn a human escalated should be a page or ticket, every freeze the team imposed by hand should have freeze_recommended: true, and — the noisier failure — every burn the team correctly ignored as a blip should be a note. The blips are where this agent earns or loses trust. An error-budget agent that cries freeze at every transient 6x burn gets muted within a month, and a muted agent is worse than the spreadsheet it replaced, because everyone believes something is watching.
Keep the recorded evidence bundles from shadow mode as regression fixtures and replay them on every prompt or model change. Attribution accuracy is easy to score offline: the fixture's dominant route and the correlated deploy are known; assert the verdict names them.
Honest Limits
The agent inherits every weakness of your SLIs. If your availability SLI counts HTTP 5xx and your worst incident mode is serving stale-but-200 responses, the budget never burns and the agent stays silent — SLI design is upstream of everything here. Attribution by route and deploy correlation is circumstantial: a deploy that lands mid-incident will be blamed for it, so the verdict schema forces the model to cite timing evidence rather than assert causation. Multi-service request paths blur ownership — a burn in checkout-api caused by a slow downstream still shows up as checkout's budget, and this agent won't untangle that; distributed tracing does. And the freeze-by-PR pattern assumes deploys flow through CI that reads the policy file — a kubectl apply from a laptop sails past it, which is an argument for closing that path generally, not for giving this agent more power. Start in shadow mode, wire the PR path second, and let the verdict quality argue for the pager integration. The first time the agent's PR shows up with the burning route, the offending deploy, and 1.8 days on the clock — before anyone opened a dashboard — the spreadsheet is done.
📌 Read the latest version of this guide — plus the full library of DevOps, SRE, Kubernetes, observability & cloud-cost guides — on devtocash.com.
