Tenant Cost Attribution Explained — 3 Fallback Models Behind One Chatbot API

Short answer: for a marketplace SaaS chatbot that reviews code changes, choose the runtime pattern that records tenant, route, model, token usage, and fallback reason in one usage event; one credential and automatic fallback are useful, but neither matters if a team can't explain each tenant's bill.

Start with the accounting boundary, not the provider list. A direct integration preserves maximum control, a self-hosted gateway centralizes routing while keeping operations in-house, and a managed broker transfers more of that operational burden. The decision turns on who must own normalization, failover policy, and evidence when a customer questions a charge.

How should a SaaS app choose one chatbot API for fallback models?

Treat “one key” as a deployment property, not a selection criterion. It reduces secret distribution across services, but it doesn't prove that fallback is safe or that tenant charges are reconstructable. For a marketplace, the useful question is narrower: can the runtime accept a code diff, return structured findings, and emit a complete usage record even when a request moves to a second model?

A serious evaluation uses the same fixture against all three patterns. Give the fixture a tenant ID, repository ID, pull-request ID, fixed prompt version, and JSON output schema. Then force three outcomes: a normal response, a rate-limit response such as 429, and a timeout at the application's deadline. The test passes only if the final finding remains tied to the original tenant and every attempt is visible. Don't merge attempts into one opaque total. A fallback can produce a good answer while leaving finance with a bad ledger.

The desired flow is easy to say aloud: request enters; policy reads tenant limits; primary model runs; a retryable classification may select an allowed fallback; schema validation accepts or rejects the finding; one append-only usage event is written per attempt; the API returns the validated result and a correlation ID. That's the diagram-in-words. Each arrow needs an observable field.

Direct clients fit when the review workflow relies on provider-specific capabilities or when only one provider is likely to remain active. The application owns the adapters, so it can preserve native response details. The catch is adapter work: auth, timeouts, usage extraction, error classes, and schema validation can drift independently.

A self-hosted gateway fits when several teams need the same policy and the organization already operates shared services. LiteLLM is one open-source example of an LLM gateway and documents a proxy-oriented approach across model providers. This option can make routing rules inspectable, but someone still owns its availability, configuration review, and telemetry pipeline.

A managed broker fits when minimizing control-plane operations is more important than owning every routing detail. Read the contract carefully — especially the usage fields, model identifiers, export path, regional handling, and behavior when a provider throttles. I'm not sure any generic feature checklist can settle that trade-off; a replayable tenant fixture and an exported usage sample provide better evidence.

Make the usage event the control plane

Per-tenant cost visibility fails when metering is reconstructed from application logs after the fact. Logs are optimized for diagnosis. A ledger needs stable identities, explicit units, and append-only attempts. Keep both, but don't pretend they are interchangeable.

For each attempt, capture tenantId, requestId, attempt, provider, model, input and output token counts, timestamps, outcome, and fallback reason. Also capture the prompt and schema versions. Those version fields explain why two similar diffs may consume different resources after a rollout. Store monetary values only after applying the rate card that was effective at the attempt timestamp; token counts are evidence, while a mutable “current price” lookup is not.

No mystery totals.

Use metrics for aggregate control: attempts by model and outcome, fallback rate by tenant tier, validation failures, latency, and token units. Use traces to connect the inbound review request to each model attempt and the ledger write. Use logs for the detailed diagnostic context that doesn't belong in metric labels. Alert on ratios over a meaningful window rather than on a single fallback; one successful fallback is expected behavior, while a sustained shift can indicate throttling, a policy change, or a workload change.

Tenant identity must be assigned by trusted server-side authentication. Never accept a billing tenant ID from an unverified chatbot payload. Repository and pull-request identifiers are useful dimensions for internal analysis, but avoid putting high-cardinality or sensitive values into metric labels. They belong in access-controlled events and traces.

Implement a metered fallback loop

The following TypeScript keeps vendor adapters outside the policy loop. It also separates the billable usage event from the final response, which makes every attempt auditable. review() represents an adapter call; its concrete URL and authentication belong inside the adapter, where they can be tested against that provider's published contract.

type Finding = {
  file: string;
  line: number;
  severity: "low" | "medium" | "high";
  message: string;
};

type Usage = { inputTokens: number; outputTokens: number };

type ReviewResult = {
  findings: Finding[];
  usage: Usage;
};

type RuntimeError = Error & {
  kind: "rate_limit" | "timeout" | "invalid_output" | "fatal";
};

interface ModelAdapter {
  provider: string;
  model: string;
  review(diff: string, signal: AbortSignal): Promise<ReviewResult>;
}

type UsageEvent = {
  tenantId: string;
  requestId: string;
  attempt: number;
  provider: string;
  model: string;
  inputTokens: number;
  outputTokens: number;
  outcome: "accepted" | "retryable_error" | "rejected";
  fallbackReason?: "rate_limit" | "timeout" | "invalid_output";
  promptVersion: string;
  schemaVersion: string;
  occurredAt: string;
};

interface UsageLedger {
  append(event: UsageEvent): Promise<void>;
}

The loop below falls back only for named conditions. That's deliberate. Authentication and permission errors should stop immediately because another model doesn't repair a broken security boundary. The adapter is responsible for returning usage for accepted responses; retryable errors use zero token counts only when the provider contract confirms that no usage was reported. In production, preserve any usage returned with an unsuccessful attempt rather than assuming zero.

async function reviewWithFallback(input: {
  tenantId: string;
  requestId: string;
  diff: string;
  adapters: ModelAdapter[];
  ledger: UsageLedger;
}): Promise<{ findings: Finding[]; requestId: string }> {
  const promptVersion = "code-review-v3";
  const schemaVersion = "finding-v1";

  for (const [index, adapter] of input.adapters.entries()) {
    const attempt = index + 1;
    const controller = new AbortController();
    const timer = setTimeout(() => controller.abort(), 8_000);

    try {
      const result = await adapter.review(input.diff, controller.signal);
      await input.ledger.append({
        tenantId: input.tenantId,
        requestId: input.requestId,
        attempt,
        provider: adapter.provider,
        model: adapter.model,
        inputTokens: result.usage.inputTokens,
        outputTokens: result.usage.outputTokens,
        outcome: "accepted",
        promptVersion,
        schemaVersion,
        occurredAt: new Date().toISOString(),
      });

      return { findings: result.findings, requestId: input.requestId };
    } catch (error) {
      const failure = error as RuntimeError;
      const canFallback =
        failure.kind === "rate_limit" ||
        failure.kind === "timeout" ||
        failure.kind === "invalid_output";

      await input.ledger.append({
        tenantId: input.tenantId,
        requestId: input.requestId,
        attempt,
        provider: adapter.provider,
        model: adapter.model,
        inputTokens: 0,
        outputTokens: 0,
        outcome: canFallback ? "retryable_error" : "rejected",
        fallbackReason: canFallback ? failure.kind : undefined,
        promptVersion,
        schemaVersion,
        occurredAt: new Date().toISOString(),
      });

      if (!canFallback || attempt === input.adapters.length) throw error;
    } finally {
      clearTimeout(timer);
    }
  }

  throw new Error("No permitted model route completed the review");
}

There is one hard production detail hidden by the interface: response delivery and ledger durability cross two systems. An append that fails must not silently disappear. Use an outbox or another durable handoff so a completed review and its usage event can be reconciled with the same request ID. Idempotency belongs here too. A retried API request should reuse its request ID, and the ledger should reject duplicate attempt keys such as (requestId, attempt).

Structured output needs its own gate. Parse the result, validate the schema, cap finding counts and message lengths, and reject file paths that aren't present in the submitted diff. A syntactically valid JSON response can still point at the wrong file. Fallback should run after validation only when policy allows it; otherwise teams can pay for several semantically bad answers and call the runtime “reliable.”

Test policy before model quality

Model evaluation matters, but routing tests should fail fast on accounting and control errors. Build a deterministic adapter for CI that can return accepted findings, throw each error kind, delay beyond the deadline, and report known token units. Then run one concrete marketplace fixture all the way through: tenant market-17 submits request rev-8842 with a two-file diff; the primary adapter reports a rate limit; policy selects the second permitted adapter; that adapter returns one schema-valid finding; and the ledger receives two ordered rows. Assert more than the final status. The first row must retain the primary model ID, attempt 1, and rate_limit; the second must retain attempt 2, its own usage, and accepted. Both rows must carry the same tenant, request, prompt version, and schema version. Now replay the inbound request with the same idempotency key. The API may return the stored result, but the ledger row count must stay at two. Repeat with an invalid file path and confirm that validation rejects it before delivery. Finally, deny fallback for the tenant and confirm that the first retryable result ends the route without invoking another adapter. This small matrix catches the ugly accounting failures: double charging on retries, attributing the second attempt to a default tenant, losing the reason for a route change, and accepting output that names a file outside the submitted diff. It tests the runtime contract without depending on a live service or a changing model.

At deployment time, canary the policy by tenant cohort. Compare fallback rate, validation rejection rate, p95 latency, and usage units per accepted review against the previous policy. The “before” is a single aggregate bill and scattered request logs. The “after” is a request-level chain: tenant market-17, request rev-8842, primary attempt, explicit fallback reason, accepted attempt, and two independently reconcilable usage records. Those identifiers are examples, not benchmark results.

Set tenant budgets in usage units first, then translate them through versioned rate cards. This prevents a price update from rewriting historical consumption. A hard budget stop, a warning threshold, and an internal anomaly alert serve different purposes; encode them separately. Your mileage may vary on the right windows because review size and tenant activity differ, so choose thresholds from observed distributions rather than copying a universal percentage.

Limits and the decision rule

Choose direct clients when native controls and a small provider set outweigh adapter duplication. Choose a self-hosted gateway when a platform team can own the control plane and multiple applications benefit from shared policy. Choose a managed broker when one credential and reduced gateway operations matter, provided its usage export preserves attempt-level tenant attribution.

None of these patterns is suitable when the application cannot establish a trusted tenant identity or retain durable usage evidence. Stick with a single model when fallback would violate data-location rules, exceed a strict latency deadline, or make output behavior harder to validate than the availability gain is worth. For voice input, speech recognition is a separate architectural stage; an open-source system such as Whisper may fit there, but it does not replace the chatbot routing and metering layer.

The final decision rule is crisp: ship the option whose forced-failure test produces valid findings and a reconcilable tenant ledger. If two options pass, select by control-plane ownership and operational capacity — not by the length of the model catalog.

References

• https://github.com/BerriAI/litellm
• https://github.com/openai/whisper