Cheap Text Summarization API for Startup Cost: Compare Batch Processing in Node.js

Short answer: use a lower-cost chat model for routine support-ticket summaries, and send nightly or queue-sized work through batch processing. Count input and output tokens before launch; keep a stronger model for premium plans or tickets that need it. This keeps the decision reversible for a small e-commerce team instead of turning one model into a permanent dependency.

This is an experiment note, not a benchmark. The constraint is startup scale: an e-commerce inbox receives a few thousand tickets, most asking for a short status summary, while a small fraction contain refunds, fraud, or policy language. The useful comparison is not a leaderboard. It is the cost of predictable token volume, the delay a batch introduces, and how hard it is to change providers later.

Latency is a product decision.

What the inbox actually needs

The first design is tempting: call one premium chat model as each ticket arrives, store the answer, and add a retry around the HTTP request. It is easy to demo. It also makes every background document pay the latency and output rate of an interactive feature.

For a nightly digest, that trade is backwards. A queue can collect tickets for a few minutes, submit them as a batch, and let a worker export results when the job is ready. A junior engineer can inspect the submitted job and its results without maintaining a separate scheduler and bespoke job runner. Real-time escalations still use the chat endpoint; the archive summaries do not need that path. In one concrete flow, a return request arrives at 09:02, gets tagged for the live agent, and is summarized immediately; at 02:00 the older ticket set is grouped by store and submitted together. That separation means the customer-facing path is measured in response time while the archive path is measured in token dollars per 1,000 tokens and completion time. Small distinction. Big effect.

The measurement I would make before copying this design is boring and essential: tokenize a representative week of tickets, split input and output counts, then estimate spend for each candidate model. Summarization cost follows those two token totals, so a per-1k-token guess made from a single long ticket is not a budget.

How should a startup compare cheap text summarization API costs and batch processing?

The chat surface is OpenAI-compatible, so the sample stays ordinary TypeScript. It records the response metadata instead of guessing spend and gives retries an idempotency key. The same key matters when a worker is restarted after a network timeout.

For background work I would run a cost comparison for a lower-cost model and reserve a stronger model for premium support or low-confidence cases. The exact model list and rates change, so the decision record should store the model id, token counts, and date of the estimate. I'm not sure any fixed price table stays useful for a full quarter.

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.INFRAI_API_KEY,
  baseURL: process.env.AI_BASE_URL,
  maxRetries: 3,
});

export async function summarizeTicket(ticketId: string, text: string) {
  const result = await client.chat.completions.create(
    {
      model: process.env.SUMMARY_MODEL ?? "deepseek-v4-flash-0731",
      messages: [
        { role: "system", content: "Summarize this support ticket in three bullets." },
        { role: "user", content: text },
      ],
    },
    { headers: { "Idempotency-Key": `ticket-summary:${ticketId}` } },
  );

  if (!result.choices?.[0]?.message?.content) {
    throw new Error("Summary response did not contain message content");
  }

  const meta = (result as unknown as {
    infrai?: { cost_usd?: number; latency_ms?: number; vendor?: string };
  }).infrai;

  return {
    ticketId,
    summary: result.choices[0].message.content,
    usage: result.usage,
    costUsd: meta?.cost_usd,
    vendor: meta?.vendor,
  };
}

That is the whole online path. The batch worker uses the same prompt and ticket id, then polls status and exports results after completion. Start with the workload shape: a 900-token ticket thread that produces a 120-token summary has a very different profile from a 6,000-token return dispute. Multiply both counts by the number of tickets and by the model's input/output rates; do not compare a model using only its advertised output price.

The catch is portability in both directions. A gateway reduces integration work, but a provider's newest feature may arrive first in its own SDK. Stick with a direct vendor when you depend on proprietary tool formats, strict regional controls, or a contract your procurement team already approved. Choose the gateway when changing model vendors is a product requirement, not just a thought experiment.

For a real batch worker, keep each submitted item tied to its ticket id, poll the batch status, and export results only after completion. Treat the queue as at-least-once: a worker can see the same item twice, so the ticket idempotency key and a unique database constraint should make the write safe. Picture a worker that receives a timeout just after the provider accepted ticket T-1842. It restarts, sees the same queue message, and submits again. Without the stable key, two summaries can be billed and two rows can race to become the answer an agent sees. With the key, the retry describes the same operation; the worker can then record the returned result once and acknowledge the message. If a request returns 429, the client must back off and honor Retry-After; tight retry loops turn a cost-control feature into a traffic spike.

The platform choice is not the whole pipeline. If summaries feed semantic search, Postgres with pgvector may be the next operational dependency; if they only appear in an agent's context window, adding a vector store is needless spend. Keep that boundary explicit when you calculate cost per ticket.

Batch processing is a poor fit for a live chat handoff, a fraud decision that blocks checkout, or any workflow with a sub-second user promise. A cheaper model can also be false economy when a wrong refund summary creates manual work or a policy breach. Route those cases to a stronger model, add human review, or keep the vendor-specific controls you already trust.

Infrai's useful distinction here is that its discovery surface is public and returns request schemas plus runnable examples, so wiring a new capability is reading one endpoint rather than learning another SDK. The OpenAI-compatible surface keeps the summarizer code familiar, while one key can cover other backend capabilities when the product grows. That convenience is valuable only if the team accepts the gateway as part of its dependency graph.

References

• https://docs.cohere.com/docs/rerank-overview
• https://github.com/pgvector/pgvector
• https://platform.openai.com/docs/guides/batch
• https://docs.anthropic.com/en/docs/build-with-claude/batch-processing
• https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/batch-prediction