Delivery Exception Detection — Node.js Metrics API Queries Feeding Lambda Webhooks

Short answer: for a small Node.js notification service, poll one delivery-failure metric from a separate scheduled function, evaluate a versioned rule, and send a deduplicated webhook; move to a full incident-management system only when rotations, escalations, and acknowledgement state become requirements.

That answer is deliberately narrower than “install an observability platform.” A logistics team needs to know when delivery notifications stop reaching recipients, but rollback safety changes the shape of the solution: the alert evaluator must not share a deployment fate with the service it watches, and rolling back either component must not replay an old incident or silently reinterpret stored state. The least complex useful design is therefore two small programs with one explicit contract between them.

This isn't a claim that polling wins everywhere. It wins only while the alert surface is small, a few minutes of detection latency is acceptable, and the team already has a trustworthy metrics query endpoint plus a webhook destination.

The delivery-failure ledger comes before the monitor

Start with the symptom the operator can act on. “Process is running” is a weak signal for a notification service; “delivery attempts are failing” is closer to the logistics outcome. The Google SRE monitoring guidance separates latency, traffic, errors, and saturation. Here, errors are the primary signal, traffic supplies the denominator, and latency can be a second rule if delayed delivery is operationally different from failed delivery.

Define the query contract before the poller. For each closed time window it should provide an event time, an attempted-delivery count, and a failed-delivery count. Keep retries straight: if one shipment notification is attempted three times, decide whether the metric represents three transport attempts or one final delivery outcome. Either is defensible. Mixing them is not. A rising attempt-level error rate can reveal provider trouble early, while final-outcome failures map more directly to shipments that need intervention; the dashboard label and runbook must say which one the alert uses.

Missing data needs its own state. Zero failures, zero attempts, and no query result are three different observations. Treating all three as zero produces a pleasant graph and a dangerous monitor. A closed five-minute window with at least 20 attempts and at least three final failures is a reasonable example policy for illustrating the state machine, not a universal threshold and not a benchmark. I'm not sure what threshold fits a particular delivery network without its normal traffic distribution, retry policy, and acceptable detection delay; a week of representative metric history would resolve that.

Silence lies.

Keep cardinality bounded. Region and notification channel may be useful alert dimensions because they identify an owner or a containment action. Shipment ID and recipient ID belong in traces or logs, not metric labels. The poller should receive aggregate facts, then attach a link or query recipe that lets an operator inspect individual failures under the access controls already used for customer data.

How does a Node.js poller turn a metrics API query endpoint into alerts?

It shouldn't. The Node.js app should publish the metric, while a scheduled observer polls it from another failure domain. A Lambda-style function is one way to host that observer, but the important property is independent scheduling and deployment, not the product category. If the application event loop stalls, its self-check cannot be the only mechanism expected to report the stall.

The observer can stay boring: fetch a normalized internal query result, reject incomplete windows, evaluate the versioned rule, derive a stable incident key, and post a webhook only on a state transition. The Python below assumes an adapter has normalized the metrics provider's response into window_end, attempted, and failed; that tiny adapter is where provider-specific query syntax belongs. Both URLs come from configuration, so the monitor does not pretend that every metrics service shares a route layout.

import hashlib
import json
import os
from datetime import datetime, timezone
from urllib.request import Request, urlopen

RULE_VERSION = "delivery-final-failure-v1"
MIN_ATTEMPTS = 20
MIN_FAILURES = 3


def read_json(url: str) -> dict:
    request = Request(url, headers={"Accept": "application/json"})
    with urlopen(request, timeout=10) as response:
        return json.load(response)


def evaluate(sample: dict) -> dict:
    attempted = int(sample["attempted"])
    failed = int(sample["failed"])
    window_end = datetime.fromisoformat(sample["window_end"])
    now = datetime.now(timezone.utc)

    if window_end.tzinfo is None or window_end > now:
        raise ValueError("window_end must be an aware, closed-window timestamp")

    firing = attempted >= MIN_ATTEMPTS and failed >= MIN_FAILURES
    incident_source = f"delivery-final-failure:{window_end.isoformat()}"
    incident_key = hashlib.sha256(incident_source.encode()).hexdigest()[:20]

    return {
        "incident_key": incident_key,
        "rule_version": RULE_VERSION,
        "state": "firing" if firing else "ok",
        "window_end": window_end.isoformat(),
        "attempted": attempted,
        "failed": failed,
    }


def post_webhook(url: str, payload: dict) -> None:
    body = json.dumps(payload, separators=(",", ":")).encode()
    request = Request(
        url,
        data=body,
        method="POST",
        headers={"Content-Type": "application/json"},
    )
    with urlopen(request, timeout=10) as response:
        response.read()


def handler(event, context):
    sample = read_json(os.environ["METRICS_QUERY_URL"])
    decision = evaluate(sample)
    post_webhook(os.environ["ALERT_WEBHOOK_URL"], decision)
    return decision

This sample intentionally stops short of claiming production-grade deduplication. A hash makes the identity deterministic, but exactly-once delivery doesn't appear because a function computed a key. Consider one five-minute window ending at 10:35: the observer posts its firing decision, the webhook receiver commits it, and the connection closes before the observer receives the acknowledgement. The scheduled runtime invokes the observer again. Without a unique constraint or conditional state write, the same logistics failure becomes two operator messages even though every component behaved within an ordinary retry contract. The receiver must therefore enforce uniqueness on incident_key, or the observer must perform a conditional write to durable state before sending; the second choice introduces another transition to model, because a crash after that write but before the webhook would otherwise suppress the alert. A small outbox record with pending, sent, and an immutable payload makes that transition inspectable. If neither side can enforce idempotency, accept at-least-once notifications explicitly and design the destination around duplicates. The ambiguous outcome is unavoidable — replay must be harmless.

There is another sharp edge: the example posts every evaluated result to keep the contract visible. In a deployed monitor, persist the last confirmed state and send only ok → firing and firing → ok transitions. Record query failures separately from delivery failures, because “the observer cannot read metrics” is not evidence that customer notifications recovered.

No magic here.

Stored state decides whether rollback is safe

A rollback changes code, but the damage usually enters through state. Suppose evaluator version 2 renames failed to terminal_failures, writes the new shape, and is then rolled back. Version 1 may read the record incorrectly or treat it as absent, producing a duplicate firing transition. The safe design uses additive state evolution: retain old fields during the compatibility window, give every record a schema version, and make the previous release read the new record before version 2 is allowed to send alerts.

Rule versions and incident identities serve different purposes. Include rule_version in the payload so an operator can reconstruct why the decision fired, but keep it out of the incident key when two evaluator releases represent the same operational condition. Otherwise a rollback creates a new identity for the same five-minute failure window. If a rule change truly represents a different condition, such as moving from final delivery failures to provider-attempt failures, give it a distinct rule name and run it in shadow mode first.

The deployment gate should exercise four fixtures: a quiet closed window, a firing window, missing data, and a replay of an already-recorded incident. Then deploy the observer without notification authority, compare its decisions with the active version, enable sends, and retain the prior artifact plus its readable state schema. A kill switch should disable outbound webhooks without disabling metric evaluation; that preserves evidence while containing alert noise.

Rollback safety also means the notification application's release cannot redefine the metric without coordination. During a field rename, publish old and new series long enough for both observer versions to query them. This costs temporary duplication, yet it is easier to reason about than a synchronized “flag day” across an application, a metrics backend, a scheduled function, durable incident state, and a chat or ticket webhook.

Failure domains matter more than feature lists

The choice is less about feature count than ownership. A small team with one actionable condition can own a poller. A team promising round-the-clock response across several services needs acknowledgement, escalation, scheduling, audit history, and tested delivery paths; rebuilding those capabilities around a webhook would turn a small monitor into an incident-management project.

A direct logging hook is still useful for enrichment. The Logback manual's custom-appender model is an example of routing events from inside an application process, though a Node.js service would use its own logging pipeline. The architectural boundary is the point: an in-process hook sees detailed context, while an external poller can observe that the process stopped producing expected signals. They solve different failure modes.

Stick with a full incident-management service when missed or delayed acknowledgement carries material business risk, when multiple teams share escalation policies, or when compliance requires durable audit trails. Prefer the scheduled poller when alerts are few, detection can wait for the query interval, and somebody explicitly owns its state store and webhook contract. For sub-minute detection or high-cardinality event routing, polling aggregates is the wrong shape; use a streaming or event-driven path and keep metrics for aggregate health.

Migration in four reversible stages

Yes, in small reversible steps. First, define the delivery outcome and backfill only enough historical windows to test the proposed rule. Second, run the observer in shadow mode and store decisions without contacting the webhook. Third, replay the same window twice and verify one incident identity. Fourth, enable a non-paging destination, exercise firing and recovery, then grant the production destination only after the previous evaluator release has read the current state format successfully.

The compact rule is: roll back computation freely, migrate durable state additively, and never let an alert transport decide what the metric means.

References

• Google SRE Book, “Monitoring Distributed Systems”: https://sre.google/sre-book/monitoring-distributed-systems/
• Logback Manual, “Appenders”: https://logback.qos.ch/manual/appenders.html