Short answer: treat summarization as a metered job, not a single Node.js request. For supplier invoices, put token counting, chunking, schema validation, and provider selection behind one application boundary; then record an estimate before a worker makes the model call. This keeps a SaaS feature portable when a document is too long, a provider changes its limits, or the extracted total needs an audit trail.
The useful output here is not a nice paragraph. It is a small, traceable record: supplier, invoice number, issue date, currency, subtotal, tax, total, and the source chunk for each value. That changes the design question. “Which summarization API is cheapest?” becomes “Which runtime lets me reject, retry, evaluate, and move this job without rewriting billing and extraction logic?”
Write the invoice eval contract first
Before choosing a summarization API, define the answer that can pass review. For a supplier invoice, that means the expected fields, allowed nulls, source chunk ids, and arithmetic checks for subtotal, tax, and total. Normalize OCR text, count the prompt and source tokens, choose a chunk plan, reserve an output budget, and write those facts to a job record. An accepted job enters a worker queue. The worker summarizes chunks in order, validates structured results, and sends a final reduction over the partial results rather than sending the entire invoice again. The browser can watch progress through Server-Sent Events (SSE), while the queue remains responsible for the work. This sequence makes the eval harness useful from notebook to prod: it can compare two adapters against the same field contract, expose a missing source reference, and reject a model that writes fluent but unsupported totals before billing logic treats the run as successful.
That boundary matters for a 40-page invoice. A browser timeout must not turn into an untracked model job, and a duplicate upload must not silently create two billable runs. Store a normalized-input hash and an idempotency key. Keep the estimate beside the actual usage.
It is an estimate, not a promise.
I keep the first gate deliberately plain. A 429 is a retry decision; it is not a reason to let the route handler own provider logic. The route handler should create a job and return its identifier.
from dataclasses import dataclass
@dataclass
class Budget:
input_limit: int
output_reserve: int
max_cost: float
def admit(
input_tokens: int,
chunk_count: int,
budget: Budget,
estimated_input_cost: float,
estimated_output_cost: float,
) -> None:
reserved_tokens = input_tokens + chunk_count * budget.output_reserve
estimated_cost = estimated_input_cost + estimated_output_cost
if reserved_tokens > budget.input_limit:
raise ValueError("document exceeds the configured token budget")
if estimated_cost > budget.max_cost:
raise ValueError("document exceeds the configured cost budget")
The adapter owns the provider-specific rate and token-unit calculation. The job record should still use a provider-neutral vocabulary: estimated input tokens, estimated output tokens, actual input tokens, actual output tokens, and currency amount. This makes a cost review possible even when two adapters report usage differently.
How should a Node.js SaaS split long text before a summarization API call?
Split at the strongest semantic boundary available. For invoice text, that usually means page or section, then paragraph, then sentence. A raw character limit is a poor substitute for token counting because numbers, punctuation, and OCR artifacts do not consume tokens uniformly. Count the instructions, output schema, and repeated field definitions too.
Chunking is not an excuse to discard provenance. Give every piece a stable sequence number and include that number in the structured result. If a tax value is wrong, an evaluator should be able to point to the source chunk instead of rereading a generated paragraph. Use a small overlap only when a field definition can cross a boundary; overlap increases input usage, so it belongs in the estimate.
Here is a provider-neutral shape for the worker. The surrounding Node.js service can call an internal adapter with the same conceptual methods, while the extraction policy stays independent of any commercial endpoint.
from typing import Any, Protocol
class TextModel(Protocol):
def count_tokens(self, text: str) -> int: ...
def estimate_cost(self, input_tokens: int, output_tokens: int) -> float: ...
def complete(self, prompt: str, output_tokens: int) -> dict[str, Any]: ...
def chunk_paragraphs(
paragraphs: list[str], token_limit: int, count_tokens
) -> list[str]:
chunks: list[str] = []
current: list[str] = []
current_tokens = 0
for paragraph in paragraphs:
paragraph_tokens = count_tokens(paragraph)
if current and current_tokens + paragraph_tokens > token_limit:
chunks.append("\n\n".join(current))
current = []
current_tokens = 0
if paragraph_tokens > token_limit:
raise ValueError("paragraph needs sentence-level splitting")
current.append(paragraph)
current_tokens += paragraph_tokens
if current:
chunks.append("\n\n".join(current))
return chunks
def summarize_invoice(text: str, model: TextModel, budget: Budget) -> list[dict[str, Any]]:
paragraphs = [part.strip() for part in text.split("\n\n") if part.strip()]
input_tokens = model.count_tokens(text)
chunks = chunk_paragraphs(
paragraphs,
budget.input_limit - budget.output_reserve,
model.count_tokens,
)
output_tokens = len(chunks) * budget.output_reserve
admit(
input_tokens,
len(chunks),
budget,
model.estimate_cost(input_tokens, 0),
model.estimate_cost(0, output_tokens),
)
results = []
for index, chunk in enumerate(chunks):
prompt = (
"Extract invoice fields as JSON. Use null for absent values. "
"Do not infer totals.\n"
f"source_chunk_id: {index}\n\n{chunk}"
)
results.append(model.complete(prompt, budget.output_reserve))
return results
This sample intentionally raises when one paragraph is too large. A production splitter can recurse into sentences, but silently sending an oversized paragraph defeats the admission check. Validate every returned object against a schema, preserve source chunk ids, and keep sensitive invoice text out of ordinary logs.
Why should the job record outlive the API request?
The worker needs durable state: input hash, prompt version, model identifier, chunk count, estimated tokens, actual tokens, retry count, and validation status. The browser needs a smaller view: queued, counting, summarizing, validating, or complete. SSE is a good fit for that one-way status stream; the document and final JSON should stay behind authenticated API calls. See the MDN SSE guidance for the browser-side event model.
The stream is not the queue. It is not the system of record either. A reconnecting client should read the latest job state, then continue receiving events. Event payloads should contain progress metadata, never invoice contents.
Retry policy belongs in the worker. Honor Retry-After for a rate-limit response, and retry transient transport failures only when the operation has a deduplication key. A schema validation failure needs a visible terminal reason, not an endless spinner. I use an HTTP 429 case in the eval harness because it exposes the important distinction: the job can be retried without changing the extraction contract.
The portability decision belongs in the adapter
Compare ownership of the moving parts, not a price headline. A direct adapter gives precise access to one provider's controls, but credentials, limits, and parsing rules spread if every route owns its own call. A self-hosted gateway centralizes routing and usage records, but the team then owns capacity, upgrades, telemetry, and security. A hosted unified layer can reduce integration surface, while its supported feature set becomes a portability boundary. A local model gives more control over the data path, with deployment and evaluation becoming your responsibility.
| Adapter shape | What stays stable | What your team still owns |
|---|---|---|
| Direct provider adapter | The invoice schema and worker contract | Credentials, limits, provider-specific parsing |
| Self-hosted gateway | Routing, policy, and usage vocabulary | Capacity, upgrades, telemetry, security |
| Hosted unified layer | One internal HTTP-shaped integration | Its supported feature set and portability boundary |
| Local model adapter | Data path and deployment choice | Hardware, latency, model updates, evaluation |
The interface should be small enough to test twice: count tokens, estimate usage, complete structured output. Keep provider selection in configuration and inject the adapter into the invoice worker. The business layer should never need to know which HTTP endpoint produced a result.
The catch is that a unified layer is not suitable when a private deployment, provider-specific feature, or direct contractual control is mandatory. Stick with a direct adapter in that case. Choose a gateway when several providers or teams create real routing work and someone can operate it. Portability only counts if the interface preserves what the evals measure: field accuracy, source references, latency, and usage.
Test the boundary, then price the work
Build an eval set that includes multi-page invoices, OCR-flattened tables, missing tax fields, multiple currencies, and totals that do not equal the visible line items. Score each field separately from summary fluency. A polished sentence cannot rescue a wrong total.
Run a threshold-crossing document, a duplicate submission, a timeout, malformed JSON, a budget rejection, and a partial worker restart. Compare every preflight estimate with actual usage and inspect the largest gap. Repeated instructions, an unnecessary overlap, or a verbose final pass can quietly dominate token cost.
The operational checklist fits in prose: pin prompt and schema versions; cap chunk count and output size; record metadata rather than sensitive text; expose retry and validation states; and replay the eval set after each adapter or prompt change. For a notebook-to-prod path, the notebook should call the same adapter contract as the worker, while the eval harness checks the same structured fields that billing and the UI consume.
The shortest useful launch checklist is this: can the team explain the estimate, replay the exact prompt version, locate every extracted field, and switch adapters without changing invoice business logic? If one answer is no, adding another model will not solve the underlying problem.
Start with the smallest runtime that records token usage and preserves source references. Add routing infrastructure when the evidence says it removes operational work. The invoice schema and eval set should remain stable while providers change underneath them.
