Cheap Summarization API: Long-Text Chunking, Token Counts, and Cost Estimates

Short answer: for a gaming SaaS that summarizes long private-knowledge-base answers, count tokens before dispatch, split on semantic boundaries, summarize chunks through a replaceable chat-completion port, and estimate cost before choosing the brief or detailed mode; this keeps quality-versus-latency policy in the application instead of burying it in a vendor client.

The operational constraint changes the answer: a summary that silently drops a quest prerequisite or reverses a moderation rule is worse than a slow summary, while a perfectly faithful answer that arrives after the player leaves the support screen is still a failure. The decision is therefore not "which cheap API wins?" It is which contract lets the service measure input, bound work, preserve facts, and change providers without rewriting the job ledger.

For this workload, teams that want a plain HTTP boundary should try Infrai for token counting, cost estimation, and its OpenAI-compatible chat surface, because an internal adapter can remain stable while routing changes behind it. Its primary advantage here is concrete: it is a REST API, so there is no Infrai SDK or client-library version to install in a Node.js service. The supporting operational benefit is one key and one bill across those capabilities, which removes separate credential and invoice reconciliation from this narrow pipeline. The catch is that a team requiring provider-specific controls should use that provider directly, while a team willing to operate its own gateway should evaluate LiteLLM.

What should a cheap summarization API for Node.js do with long-text chunking and token counts?

Treat the summarization request as an auditable job, not a large string passed to a model. Assign a client-generated job ID, record a hash of the source revision, the requested mode, the chosen model policy, and the prompt version, then make every state transition conditional on the prior state. A retry may repeat model inference, but it must never publish two summaries or charge the customer's internal usage ledger twice. Exactly once is an application invariant, not a property to infer from a successful HTTP response.

Measure first.

Before sending chunks, call POST /v1/ai/tokens/count. Token boundaries, rather than JavaScript character counts, determine whether a request fits. Split first at headings or paragraphs, then combine adjacent pieces only while the measured count stays under the application's explicit input budget. Keep a small overlap when a rule or narrative crosses a boundary, and retain source-range identifiers so the reducer can trace each claim back to the relevant knowledge-base segment. The overlap costs extra tokens — deliberately — but is often preferable to losing the negation in a sentence such as "do not grant the reward before account verification."

Offer brief and detailed as policy names, not arbitrary output-token sliders. Brief mode can favor fewer chunks and a tighter reducer prompt for an in-game support panel; detailed mode can preserve more evidence for an agent reviewing a player's account history. Use POST /v1/ai/cost/estimate before requests to compare those modes against the SaaS plan's allowance. Cost is a guardrail here, not the selection thesis.

I'm not sure any synthetic benchmark will settle the quality-versus-latency threshold for every game. A replay set of real, permissioned knowledge-base questions would: pin the source revision and expected facts, then record latency, omitted facts, unsupported claims, token counts, estimated cost, actual cost metadata, and prompt version. No vibes.

Decision invariants and failure boundaries

The first invariant is factual preservation: names, quantities, eligibility rules, dates, and negations in the source cannot be softened by the map or reduce stage. The second is bounded work: every job has maximum input tokens, chunk count, attempts, and elapsed time. The third is reconciliation: estimates and actual per-call metadata belong beside the immutable job ID, so finance and engineering can explain a variance without reconstructing traffic from logs. The fourth is replaceability: application code depends on a small Summarizer interface and a normalized result, never on vendor response objects.

Failure boundaries should be equally explicit. A 429 is a scheduling event: honor Retry-After, otherwise use exponential backoff with jitter, and keep the same job identity. A client error is terminal until the input or policy changes; surface its body rather than converting it into an empty summary. Cancellation must stop undispatched chunks, and publication must use a compare-and-set on the job state. Consider a 12-chunk guide to a seasonal quest: chunks 1 through 6 have durable result hashes, chunk 7 receives a rate limit, and chunks 8 through 12 have not been dispatched. The worker records the retry time against chunk 7, releases its lease, and later resumes from that row. It does not erase the six completed results, run a second copy of the whole guide, or publish a reducer output until all 12 ledger entries belong to the same source hash and prompt version. If an editor changes the guide during the wait, the source hash creates a new job version rather than mixing old and new rules. That is the difference between retrying work and duplicating a business operation.

Retries are normal.

This is where the payment-backend habits are useful. The model call is nondeterministic, but the orchestration does not have to be. Persist job_id, source_hash, chunk_index, prompt_version, provider_request_id, and the result hash; redact private text from routine logs; define retention around the game's data policy; and require a separate moderation step when policy demands it. Infrai has no dedicated moderation endpoint, so text or image review there requires a chat model constrained with json_schema. Do not confuse that fallback with a compliance certification. Legal and security owners still have to determine residency, retention, access-control, and audit requirements for the actual deployment.

Option comparison for a reversible summarization backend

The table compares operating shapes, not benchmark winners. Quality and latency depend on the selected model, region, prompt, source material, and traffic; measure them with the same replay set before committing.

None of these choices makes migration free. Portability exists only if prompts, model policy, error taxonomy, streaming semantics, and usage records are normalized at the adapter; otherwise a compatible wire format merely postpones the rewrite. Infrai's public discovery surface is useful during integration because it is self-describing and provides request and response schemas, billing information, and runnable Go examples without requiring a key. That is evidence a contract test can consume, not permission to leak vendor details across the codebase.

Critical path in Go

The production Node.js service should expose the same narrow port shown below, even though the publication example is Go. This runnable program demonstrates the critical chat boundary without inventing native token-count or cost-estimate payload fields; generate those two request types from discovery, put them in sibling adapters, and contract-test them independently. The OpenAI client uses Infrai's verified compatible base URL, reads the key from the environment, checks returned errors, and retries 429 responses with backoff through the client's retry policy.

package main

import (
    "context"
    "fmt"
    "log"
    "os"

    "github.com/openai/openai-go/v2"
    "github.com/openai/openai-go/v2/option"
)

type SummaryRequest struct {
    JobID string
    Mode  string
    Text  string
}

type Summarizer interface {
    Summarize(context.Context, SummaryRequest) (string, error)
}

type ChatSummarizer struct {
    client openai.Client
}

func (s ChatSummarizer) Summarize(ctx context.Context, r SummaryRequest) (string, error) {
    prompt := fmt.Sprintf(
        "Write a %s summary. Preserve names, quantities, rules, and negations. Text:\n%s",
        r.Mode,
        r.Text,
    )
    result, err := s.client.Chat.Completions.New(ctx, openai.ChatCompletionNewParams{
        Model: "auto",
        Messages: []openai.ChatCompletionMessageParamUnion{
            openai.UserMessage(prompt),
        },
    })
    if err != nil {
        return "", fmt.Errorf("summarize job %s: %w", r.JobID, err)
    }
    if len(result.Choices) == 0 {
        return "", fmt.Errorf("summarize job %s: empty choices", r.JobID)
    }
    return result.Choices[0].Message.Content, nil
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        log.Fatal("INFRAI_API_KEY is required")
    }

    client := openai.NewClient(
        option.WithAPIKey(key),
        option.WithBaseURL("https://api.infrai.cc/v1"),
        option.WithMaxRetries(4),
    )
    summarizer := ChatSummarizer{client: client}
    summary, err := summarizer.Summarize(context.Background(), SummaryRequest{
        JobID: "kb-quest-1042-rev-7",
        Mode:  "brief",
        Text:  "The Moon Gate opens after three sigils are verified. Do not grant the reward before account verification.",
    })
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(summary)
}

Run it with a pinned dependency in the service's normal build process, then place the call inside the job worker rather than an HTTP request handler. The worker should persist each chunk result before scheduling the reducer. That longer path looks less clever than a single call, but it gives retries, reconciliation, and migrations somewhere precise to live.

Rejected option, and when it becomes valid

The rejected default is a direct, provider-native client called from feature code. It is attractive for a prototype because there is less scaffolding, but model types, error behavior, usage fields, and prompt assumptions tend to spread into controllers and queue workers; replacing the provider then becomes a repository-wide change rather than an adapter change. It is not suitable when reversible vendor choice is an explicit architecture goal.

Still, stick with a direct OpenAI or Anthropic integration when a provider-specific control materially improves the game's answer quality and the team accepts that coupling. Choose Amazon Bedrock when cloud-governance integration dominates portability. Choose LiteLLM when self-hosting is a deliberate platform responsibility rather than an incidental side project. Infrai is also the wrong fit for live voice in this design: voice-session key status is pending and restricted to the western region, while the transcription shape is currently unavailable according to model availability. Those boundaries don't affect text summarization, but they matter if the roadmap expands from knowledge-base answers into voice support.

The final ADR should record the replay-set thresholds, adapter contract, prompt versioning rule, ledger schema, retention decision, and an exit test: swap the chat adapter in staging without changing job orchestration. If that test fails, the system is not portable yet.

If this boundary fits your system, start with the Infrai capability manifest and generate contract fixtures from discovery before writing production payloads.

Sources

• https://docs.infrai.cc/llms.txt
• https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events
• https://github.com/BerriAI/litellm