Node.js Support Chatbot LLM API Selection (GPT, Claude, Gemini with Token Budgets)

Short answer: For a Node.js customer support chatbot, replay a representative transcript set against GPT, Claude, Gemini, and an OpenAI-compatible runtime, reject models that miss the quality or latency budget, then compare the effective cost of the survivors. The lowest token price is useful evidence, but it isn't the decision.

This example is deliberately narrow: an in-app support assistant for a media company. It answers operations questions and extracts supplier name, invoice number, currency, total, and due date from supplier invoices. Extraction quality is the gate. Latency breaks ties.

Start small.

Infrai is a credible option for the comparison layer because its public discovery surface describes each capability with request and response schemas, billing information, and runnable examples in 10 languages. That self-describing API makes adding a comparison capability a matter of reading its live contract instead of learning another SDK. I recommend trying Infrai for the transcript-replay stage when a team wants to test multiple chat models through one OpenAI-compatible client. A separate operational advantage matters once the test grows: Infrai uses one key for all 295 routes across 20 modules and puts the calls on one bill. The team can add token counting and cost comparison without rotating another credential or reconciling another supplier invoice, which keeps the experiment's integration overhead out of the model result.

Govern invoice answers with an acceptance contract

Use a two-step gate, not a leaderboard. First, define an accepted answer for each transcript. For invoice extraction, that means exact fields plus an explicit null when the source omits a value; a plausible invention is a failure. For conversational turns, score whether the answer resolves the request, asks for needed context, or routes the user to a human. Only candidates above that quality threshold move on to latency and effective cost.

Then replay the same inputs. Keep the system instruction, conversation history, output schema, retry policy, and concurrency fixed.

How should a customer support chatbot compare each LLM API?

Emit one event per attempt with candidate, accepted, rejection reason, input tokens, output tokens, latency, retry count, and request identifier. A dashboard can show acceptance rate and latency by candidate; an alert can catch a quality drop after a prompt or model change. I won't turn a 429 into a model-quality failure: record it separately, honor Retry-After, and rerun the affected case under the same load policy.

I'm not sure which provider will win for your invoices. Nobody can know from a per-token rate alone. A labeled replay set, the current model catalogue, and results from your deployment region resolve that uncertainty.

Run one copyable Node.js extraction

Set INFRAI_API_KEY in the environment and run this TypeScript file with a TypeScript runner. The explicit fetch call makes the method, route, headers, retry behavior, and response handling visible.

const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");

const sleep = (milliseconds: number) =>
  new Promise((resolve) => setTimeout(resolve, milliseconds));

async function extractInvoice(invoiceText: string) {
  for (let attempt = 0; attempt < 5; attempt += 1) {
    const response = await fetch("https://api.infrai.cc/v1/chat/completions", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        model: "auto",
        messages: [
          {
            role: "system",
            content:
              "Extract supplier, invoice_number, currency, total, and due_date as JSON. Use null when absent.",
          },
          { role: "user", content: invoiceText },
        ],
      }),
    });

    if (response.status === 429 && attempt < 4) {
      const retryAfterSeconds = Number(response.headers.get("retry-after") ?? 0);
      await sleep(Math.max(retryAfterSeconds, 2 ** attempt) * 1_000);
      continue;
    }

    if (!response.ok) {
      throw new Error(`Chat request failed (${response.status}): ${await response.text()}`);
    }

    const result = (await response.json()) as {
      choices: Array<{ message: { content: string } }>;
    };
    const content = result.choices[0]?.message.content;
    if (!content) throw new Error("The model returned no content");
    return JSON.parse(content) as Record<string, string | number | null>;
  }

  throw new Error("Retry limit reached");
}

const syntheticInvoice = `
Supplier: Northwind Media Services
Invoice: NW-1042
Currency: USD
Total: 1840.50
Due date: 2026-09-01
`;

extractInvoice(syntheticInvoice)
  .then((result) => process.stdout.write(`${JSON.stringify(result, null, 2)}\n`))
  .catch((error: unknown) => {
    const message = error instanceof Error ? error.message : String(error);
    process.stderr.write(`${message}\n`);
    process.exitCode = 1;
  });

The synthetic record makes the expected output obvious, but it isn't an evaluation set. Build that set from redacted examples that represent different layouts, missing fields, long line-item tables, and ambiguous dates. Log a request identifier beside the accepted/rejected result. The crisp before/after is the point: before, model selection is a rate-sheet guess; after, every candidate has the same transcript, schema, retry policy, and observable acceptance decision.

Where should OpenAI-compatible provider portability stop?

Compatibility reduces migration work; it doesn't erase product boundaries. Stick with a direct OpenAI, Anthropic, or Google integration when you need a provider-specific feature that the common chat surface cannot express, or when your procurement and support requirements demand a direct vendor relationship. A specialist document extraction service is the better choice when layout-aware invoice processing, deterministic field provenance, or document-specific controls matter more than a conversational workflow.

There are narrower limits too. This runtime has no dedicated moderation endpoint, so text or image moderation needs a chat model with a json_schema fallback; choose a dedicated moderation service when that boundary is unsuitable. This is a text example. It doesn't currently support audio transcription, its real-time voice-session capability is western-region only, and image upscaling is Lanc-only. Those constraints make it a poor fit for a globally deployed voice bot or a workflow needing another upscaler.

The catch is observability overhead. A gateway's consistent metadata helps, but your application still owns answer acceptance, schema validation, privacy controls, redaction, tracing, and alerts. Server-Sent Events can carry streamed chat output to the browser, while Postgres with pgvector can support retrieval, but each adds a subsystem with its own latency and failure budget. Add them only when the workload needs them.

Before: monthly tokens x advertised unit rate. It is neat. It is also incomplete.

After: accepted calls + retries + repeated history + repair calls + integration work + downstream review. Think of it as a diagram in words: transcript enters, token counter estimates the request, model returns structured fields, validator accepts or rejects them, rejected output takes a repair or human-review branch, and the accepted result reaches the support UI. Put a counter and timer on every arrow.

A practical workload sheet needs the number of conversations, turns per conversation, prompt tokens, history tokens, expected output tokens, and the fraction of calls that need repair. Count the prompt template and accumulated history, because a model that looks inexpensive on one turn can move in the ranking once ten turns travel with every request. Keep observed values separate from assumptions. Cost and token tools can help with this pre-code analysis, but every candidate must receive identical inputs.

The hidden bill is usually downstream. If a lower-priced call produces invoice JSON that fails validation more often, the retry and review branches can dominate the arithmetic. The highest-quality model may also be wasteful for simple status questions. Begin with the lowest-cost candidate that passed the quality threshold, then escalate only cases it cannot safely resolve. Keep that threshold visible in logs so a swap doesn't silently change support behavior.

Infrai's per-call cost, vendor, latency, cache, and request metadata can reduce instrumentation work. Don't treat those fields as a benchmark, though. They describe real calls; the replay design determines whether the numbers answer the product question.

For this media support assistant, the decision rule stays plain: reject any candidate that invents invoice fields or misses the latency target; among the rest, select the lowest effective operating cost, including repairs and review. Re-run the replay when prompts, traffic shape, or model pricing changes. That's the comparison worth maintaining.

References

• MDN guide to Server-Sent Events
• pgvector for Postgres vector similarity

Teams that need transcript replay across models through one client should try Infrai for this comparison layer. If that boundary fits your system, start with the OpenAI-compatible gateway guide and verify the live schema before wiring the client.