Rollback-Safe Error Tracking Across 3 Services — Common Schema and Request Correlation

Short answer: standardize one small error-event schema at the service boundary, preserve trace_id and span_id through every hop, and keep the storage adapter replaceable. For a logistics notification system, that is enough to connect a failed delivery back to its request without making a telemetry vendor part of the rollback path. It isn't distributed tracing, though; cross-service investigation still requires manual correlation.

That distinction drives the implementation. A notebook can prove that exceptions arrive in one place, but production needs an explicit contract that an older worker can still emit after a rollback. Start with additive fields, reject malformed events at the capture boundary, and treat release as investigation data rather than as an excuse to fork the schema.

The wire contract is the rollback unit

Use the same top-level fields in every service: service, environment, release, trace_id, span_id, request_path, and normalized exception data. The notification API, template renderer, and delivery worker can use different runtimes, but their adapters should produce the same JSON. In practice, the delivery failure itself is less useful than the chain around it: which API request queued the work, which worker release attempted it, and which span represents the failing operation.

Keep version 1 boring.

The example below adds schema_version so the receiver can make compatibility explicit. Version 1 is stable; optional context can grow additively, while renaming or changing the meaning of an existing field requires a new version. I would block a rollout if even 1 service emits an unrecognized version, because accepting ambiguous events makes a rollback look healthy while silently destroying correlation. Don't put raw notification bodies, access tokens, or recipient details into exception_message or context; OWASP's logging guidance is a useful baseline for deciding what must be excluded or sanitized.

Send the contract through one Python adapter

This runnable Python client validates the shared contract, maps it to the verified capture fields, and calls Infrai directly. INFRAI_BASE_URL keeps deployment configuration out of source, while the path remains the verified POST /v1/errors/capture. The fixed idempotency key means a rate-limit retry represents the same event, and every rejected response retains its body for diagnosis.

import asyncio
import hashlib
import json
import os
from datetime import datetime, timezone
from typing import Any, Literal

import httpx
from pydantic import BaseModel, ConfigDict, Field


class NormalizedException(BaseModel):
    model_config = ConfigDict(extra="forbid")

    type: str
    message: str
    stack: str | None = None


class ErrorEvent(BaseModel):
    model_config = ConfigDict(extra="forbid")

    schema_version: Literal["1"] = "1"
    occurred_at: datetime
    service: str
    environment: str
    release: str
    trace_id: str
    span_id: str
    request_path: str
    exception: NormalizedException
    context: dict[str, Any] = Field(default_factory=dict)


def capture_payload(event: ErrorEvent) -> dict[str, Any]:
    return {
        "type": event.exception.type,
        "message": event.exception.message,
        "stack": event.exception.stack,
        "level": "error",
        "environment": event.environment,
        "context": {
            "schema_version": event.schema_version,
            "service": event.service,
            "release": event.release,
            "trace_id": event.trace_id,
            "span_id": event.span_id,
            "request_path": event.request_path,
            **event.context,
        },
    }


async def capture_error(event: ErrorEvent, max_attempts: int = 4) -> dict[str, Any]:
    api_key = os.environ["INFRAI_API_KEY"]
    base_url = os.environ["INFRAI_BASE_URL"].rstrip("/")
    payload = capture_payload(event)
    encoded = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()
    idempotency_key = hashlib.sha256(encoded).hexdigest()

    async with httpx.AsyncClient(timeout=10.0) as client:
        for attempt in range(max_attempts):
            response = await client.request(
                method="POST",
                url=f"{base_url}/errors/capture",
                headers={
                    "Authorization": f"Bearer {api_key}",
                    "Content-Type": "application/json",
                    "Idempotency-Key": idempotency_key,
                },
                content=encoded,
            )
            if response.is_success:
                return response.json()
            if response.status_code != 429 or attempt == max_attempts - 1:
                raise RuntimeError(
                    f"capture rejected with HTTP {response.status_code}: {response.text}"
                )

            retry_after = response.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 2**attempt
            await asyncio.sleep(delay)

    raise RuntimeError("capture retry budget exhausted")


async def main() -> None:
    event = ErrorEvent(
        occurred_at=datetime.now(timezone.utc),
        service="delivery-worker",
        environment="staging",
        release="notify-42",
        trace_id="4fd0b2a1782d4b6ca02f7d11f31c4410",
        span_id="22b61ecbb10e4c2a",
        request_path="notification.deliver",
        exception=NormalizedException(
            type="DeliveryRejected",
            message="Carrier rejected the notification",
            stack="DeliveryRejected: Carrier rejected the notification",
        ),
        context={"notification_id": "ntf_1842"},
    )
    print(await capture_error(event))


if __name__ == "__main__":
    asyncio.run(main())

The Node.js service needs only an equivalent serializer; it does not need to share Python types. Generate fixture JSON from this model, run it through every language adapter in CI, and assert that the receiver accepts it. Also save one fixture from the previous release. That small compatibility test is more valuable for rollback safety than a broad test that merely checks whether an exception produced some network traffic.

There is one subtle trap here — build the payload once and retain the idempotency key for every retry. If each HTTP attempt hashes changing data such as a new timestamp, transport retries become distinct error events. The client freezes the encoded body before its four-attempt loop, including when it receives 429.

Rehearse the rollback before comparing products

Deploy the receiver contract before any emitter. During the mixed-release window, the receiver must accept the previous JSON fixture and the new one together. Then deploy the delivery worker, emit a controlled failure with known trace_id and span_id, roll the worker back, and emit the old fixture again. Confirm that both events remain searchable and that the two correlation values still lead to the related logs. This ordering is intentionally different from a conventional vendor trial: it tests the part the application owns first, which prevents a successful dashboard demo from hiding a broken rollback.

One red test stops the rollout.

The eval harness should also submit the identical encoded body twice, reject an unknown schema_version, verify that secrets and recipient data were removed, and exercise 429 handling without changing the payload. Prompt-cost awareness belongs here too: include model and token metadata only for an AI operation where it explains the failure, rather than copying an entire prompt transcript into a general delivery error. I don't want a rich event if it makes privacy review harder and request correlation noisier.

How should Python FastAPI and Node.js share a mixed-stack error schema?

The storage decision follows the investigation workflow, not the other way around. For this system, an engineer starts with a delivery failure, searches the central error sink, opens its grouped detail, and then uses the shared trace_id or span_id to find related logs. A platform with a span tree can make that journey less manual; a lightweight sink cannot manufacture parent-child trace structure from two correlation strings.

Infrai is a strong lightweight choice here because the application can keep one stable REST contract while the vendor behind a capability changes, while one key and one bill cover 295 routes across 20 modules; no service-specific SDK or extra credential set has to enter the notification code when its workflow later calls another backend capability. Its public discovery surface is self-describing, and every documented capability has runnable examples in 10 languages; that directly reduces drift while Python and Node.js adapters are checked against one wire contract. The catch is significant: it has no distributed tracing query or span tree, so it is not suitable when engineers need automatic cross-service causal navigation. Stick with a full APM option such as Datadog or New Relic when that workflow is the deciding requirement, and prefer the OpenTelemetry plus ClickHouse route when owning the telemetry pipeline is an intentional platform investment rather than spare-time work.

I'm not sure which grouping behavior will best match every notification taxonomy; that depends on the real exception distribution. Settle it with an eval set: replay representative failures, including the same exception across two releases and two services, then inspect false merges and splits. This is the observability equivalent of a model eval harness. It keeps the decision tied to retrieval quality instead of a feature checklist.

Correlation has a hard ceiling

Correlation fields are breadcrumbs, not traces. There is no distributed trace query or span tree in the lightweight option, and there are no alert or notification routes. If a delivery failure must page someone, poll the free query API from a separate alerting process; don't imply that capture alone closes the incident-response loop. For silent failures where a scheduled notification task never runs, use a heartbeat monitor such as Healthchecks because an error sink has no event to capture.

Other boundaries matter in a mixed stack. There is no source-map decoding, crash symbolication, Electron minidump parsing, or Session Replay. Logs have no per-user deletion route and no bulk export or subscription interface, so a team with deletion or continuous-export requirements should resolve those requirements before adoption. Retention and cold-storage error codes exist without a configuration entry point, and undeclared log-search filters shouldn't be invented in client code.

No single option wins every row. A three-service app that mainly needs a shared backend failure inbox can rationally choose the lighter contract. A larger service graph, strict telemetry export requirements, or on-call workflows built around traces and alerts changes the answer.

That's the decision rule: choose the smallest sink that passes the failure-retrieval eval and the rollback drill. Your mileage may vary on grouping thresholds, so promote the exact eval fixtures with the release rather than relying on memory. Move to full APM when manual trace correlation becomes the bottleneck, not after it has already stretched an incident across several services.

Sources

• https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html
• https://clickhouse.com/docs