Failure Alerts: 2-Signal Polling Sets Metrics API Thresholds for Rollbacks

Short answer: run a small scheduled worker that polls both recent metrics and errors, pages through a separate notification provider when either count crosses its threshold, and uses a heartbeat monitor to catch the more dangerous case where the worker never runs.

For a fintech import, detection is only half the design. The alert has to carry enough context to decide whether to pause the release or roll back without guessing: deployment ID, import name, expected completion time, observed result count, error count, and the last successful run. Keep the first version boring. A Python sidecar can watch a Node.js importer just as well as code embedded in the service, and separating it makes rollback behavior easier to test.

How should a cron job poll a metrics API and alert on failures?

Treat the scheduled check as a two-signal watchdog. The metrics query answers, "Did the import produce enough results?" The error search answers, "Did it fail noisily?" Neither answer proves the job actually started, so a dead-man's-switch heartbeat covers silent absence. For a five-minute import, I would schedule the check after the normal completion window rather than at the same instant as the import.

The decision rule can stay simple: page when the error count reaches its configured threshold, or when the result count remains below its configured minimum after the grace period. Don't let the alerting process perform the rollback itself. It should produce an evidence bundle for the existing deployment controller, because an automatic rollback triggered by a delayed metric can turn one late batch into two overlapping batches.

That's the boundary.

There is one awkward implementation detail. The query routes are real, but their discovery metadata does not declare filter parameters. I’m not sure which response paths your account and metric shape will expose until you inspect an authenticated response. The example therefore sends no invented query string; instead, it makes the two verified requests and requires explicit dot paths for the numeric values. That calibration is a deployment setting, not hidden parsing magic.

The data flow and runnable worker

This runnable worker uses only the Python standard library. Set INFRAI_API_KEY, METRICS_API_BASE_URL, METRIC_COUNT_PATH, and ERROR_COUNT_PATH in the job environment; the base URL is the provider's versioned API root, while the latter two values are dot paths into the JSON responses you observe during setup. Set ALERT_WEBHOOK_URL to a separate notification provider. A production secret store should supply both credentials.

The code deliberately makes every HTTP method explicit, surfaces non-success bodies, honors Retry-After on HTTP 429, and caps exponential retries. It calls exactly two verified observability routes. No SDK is involved.

import json
import os
import time
from datetime import datetime, timezone
from urllib.error import HTTPError
from urllib.request import Request, urlopen


API_BASE = os.environ["METRICS_API_BASE_URL"].rstrip("/")
API_KEY = os.environ["INFRAI_API_KEY"]
METRIC_COUNT_PATH = os.environ["METRIC_COUNT_PATH"]
ERROR_COUNT_PATH = os.environ["ERROR_COUNT_PATH"]
ALERT_WEBHOOK_URL = os.environ["ALERT_WEBHOOK_URL"]
MIN_RESULTS = int(os.getenv("MIN_RESULTS", "1"))
MAX_ERRORS = int(os.getenv("MAX_ERRORS", "1"))
MAX_ATTEMPTS = 4


def retry_delay(headers, attempt):
    retry_after = headers.get("Retry-After")
    if retry_after and retry_after.isdigit():
        return int(retry_after)
    return min(2 ** attempt, 30)


def request_json(url, method, headers, body=None):
    payload = None if body is None else json.dumps(body).encode("utf-8")
    for attempt in range(MAX_ATTEMPTS):
        request = Request(url, data=payload, headers=headers, method=method)
        try:
            with urlopen(request, timeout=20) as response:
                return json.load(response)
        except HTTPError as error:
            response_body = error.read().decode("utf-8", errors="replace")
            if error.code == 429 and attempt + 1 < MAX_ATTEMPTS:
                time.sleep(retry_delay(error.headers, attempt))
                continue
            raise RuntimeError(
                f"{method} {url} returned HTTP {error.code}: {response_body}"
            ) from error
    raise RuntimeError(f"{method} {url} exhausted retries")


def number_at(document, path):
    value = document
    for part in path.split("."):
        value = value[int(part)] if isinstance(value, list) else value[part]
    if isinstance(value, bool) or not isinstance(value, (int, float)):
        raise TypeError(f"{path} must resolve to a number")
    return value


def main():
    api_headers = {
        "Authorization": f"Bearer {API_KEY}",
        "Accept": "application/json",
    }
    metrics = request_json(
        f"{API_BASE}/metrics/query", method="GET", headers=api_headers
    )
    errors = request_json(
        f"{API_BASE}/errors/search", method="GET", headers=api_headers
    )

    result_count = number_at(metrics, METRIC_COUNT_PATH)
    error_count = number_at(errors, ERROR_COUNT_PATH)
    failed = result_count < MIN_RESULTS or error_count >= MAX_ERRORS

    if not failed:
        print(json.dumps({"status": "healthy", "results": result_count,
                          "errors": error_count}))
        return

    alert = {
        "event": "scheduled_import_threshold_breached",
        "checked_at": datetime.now(timezone.utc).isoformat(),
        "result_count": result_count,
        "minimum_results": MIN_RESULTS,
        "error_count": error_count,
        "maximum_errors": MAX_ERRORS,
    }
    request_json(
        ALERT_WEBHOOK_URL,
        method="POST",
        headers={"Content-Type": "application/json"},
        body=alert,
    )


if __name__ == "__main__":
    main()

Run it once by hand with a captured low-result case and a known error case, then schedule that same command with cron. The code emits no success webhook, so point a separate heartbeat service at the scheduler or wrap the invocation with that service's documented ping mechanism. Also make the downstream notification provider deduplicate scheduled_import_threshold_breached events for a run identifier; repeated alerts are possible whenever the same unhealthy window is polled more than once.

The common mistake is setting MAX_ERRORS=1 and considering the work finished. In a payment-import pipeline, one malformed row might be tolerable while zero produced records is an immediate stop signal; another import may have the reverse policy. Put thresholds in deployment configuration, replay representative eval fixtures against them, and record why each threshold exists. Prompt and model changes deserve the same treatment when an AI extraction step feeds the import: compare the output count and validation failures before promotion, because a syntactically successful model call can still produce unusable business data.

Choose detection by rollback risk, not feature count

The right tool depends on who must act after the page. This is where a cheap-looking polling script can become expensive operationally — not because of API billing, but because it owns scheduling, state, deduplication, and notification behavior.

Infrai is credible here when the team values a plain REST API that any sidecar can call and wants observability queries under the same key as a broader backend surface. The catch is substantial: it is detection storage, not a full alerting platform. Stick with Prometheus plus Alertmanager or Datadog when routed escalation, richer incident workflows, and centrally managed alert rules matter more than keeping the integration small. Use Sentry when errors are the dominant signal. Add Healthchecks when "the task never ran" is the failure you fear most.

None of these choices supplies rollback correctness by itself. The deploy system still needs an immutable release identifier, a known-good target, and a rule preventing two imports from writing concurrently. Those controls sit outside the polling loop.

Test the threshold like an eval

A monitor that has never fired in staging is only a theory. Feed the parser stored response fixtures for four cases: healthy results, too few results, excessive errors, and a changed or missing JSON path. The final case should fail loudly rather than quietly converting missing data to zero. Then test HTTP 429 with both numeric Retry-After and no header, plus a non-success response whose body must appear in the worker log.

Keep a deliberate grace period. Metrics can arrive after the import process exits, and paging before that window closes creates false rollback pressure. Your mileage may vary, so derive the window from observed ingestion behavior rather than copying a universal number. I favor a small table of eval cases checked into the application repository: input fixture, expected alert decision, expected evidence fields. It makes threshold edits reviewable in the same way prompt evals make model changes reviewable.

One more limit matters for debugging: there is no distributed trace query or span tree here. Logs can carry trace_id and span_id for correlation, but that is not the same investigation experience. Source-map decoding, crash symbolication, Electron minidump parsing, and Session Replay are also outside this setup. If the on-call workflow depends on those features, select a platform built around them rather than extending the polling worker until it becomes a home-grown incident system.

The final gate is a rollback-ready operating rule. Before enabling the cron entry, confirm that a low result count and an elevated error count each produce one useful notification, while HTTP 429 produces bounded backoff instead of a tight retry loop. Verify the alert names the import and deployment, carries the observed and configured values, and points to the known-good release. Confirm separately that a missed schedule trips the heartbeat monitor. Finally, run the rollback drill without letting the watcher execute the rollback; a human or established deployment controller should own that state change until the team has enough evaluated history to justify automation.

Keep the watcher narrow.

Review the JSON paths whenever the response schema changes, keep query filtering out of the request until it is explicitly declared, and reassess the architecture when incident routing becomes a core requirement. That gives a notebook-to-production path with an honest boundary: two query signals, one heartbeat, one tested decision rule, and no pretend certainty about an undocumented filter.

References

• https://prometheus.io/docs/alerting/latest/alertmanager/
• https://docs.datadoghq.com/monitors/
• https://docs.sentry.io/product/alerts/
• https://healthchecks.io/docs/
• https://docs.python.org/3/library/urllib.request.html