Short answer: build invoice retrieval around a provider-neutral document contract, count tokens before every embedding or generation call, and keep extraction separate from semantic search. Batch only documents that share the same embedding configuration. This gives a Node.js service a credible cost estimate before indexing and a narrow boundary it can move later.
For supplier invoices, cheap ingestion is a weak goal by itself. The useful goal is a ledger that can explain which source page produced each extracted field, what was sent to each model operation, and what must be reprocessed after a parser, schema, or provider change. That constraint shapes the system more than a model leaderboard does.
What should a portable invoice RAG system preserve?
Preserve facts that survive a provider change: a stable document ID, the original file checksum, page and chunk coordinates, tokenizer identity, embedding configuration, extraction schema version, and processing status. Keep provider response objects outside this core record. They are evidence for an adapter log, not your domain model.
An invoice is also not ordinary prose. A supplier name near the top, a line-item table, a tax identifier, and a total on the last page have different structural roles. Flattening the whole PDF into equal character windows can place a quantity in one chunk and its unit price in another. Semantic similarity can't repair that lost relationship. Parse layout first, retain page coordinates, and form chunks along headers, table rows, and field groups.
The catch is that this design asks you to retain more metadata and operate a versioned ledger. It isn't suitable when the collection is tiny, disposable, and searched only once; direct extraction with a strict output schema is easier there. It is also a poor fit for exact reconciliation by invoice number or purchase-order ID. Use a normal indexed database lookup for exact identifiers, then reserve semantic search for vague questions such as "which invoices mention cold-chain handling?"
Compliance changes the boundary too. Invoice data can contain names, addresses, bank details, and tax identifiers. Classify those fields before indexing, minimize what enters retrieval text, and keep authorization outside the model prompt. Don't assume that a plausible answer is an authorized answer.
How should a Node.js RAG pipeline batch embeddings for semantic search?
Batching should be a transport optimization, not a change to document identity. First create deterministic chunks. Then count each chunk with the tokenizer associated with the selected operation, reject or split anything over the configured input limit, and pack chunks under both item-count and token-count ceilings. Record every attempted item before sending a batch, and mark each item complete only after its vector and metadata are durably stored.
Keep the batch manifest.
That small record prevents a nasty ambiguity: after an interrupted worker, the system can distinguish "planned but never sent" from "embedded and stored." Retry only non-complete item IDs, while an idempotency key derived from the source checksum, chunk coordinates, embedding configuration, and normalization version prevents duplicate logical records. This is especially important for invoice corrections, where a supplier may resend a PDF under the same filename with one line item changed. The checksum changes; the filename doesn't.
A practical manifest has four states: planned, submitted, stored, and rejected. Those are application states, not claims about any external API. A rejected item should carry a local reason such as TOKEN_LIMIT or EMPTY_TEXT. Rate-limit responses need bounded retries with jitter and a durable next-attempt time; malformed source data should go to review rather than cycling through the queue.
The cost ledger matters more than a sticker price
Estimate three workloads independently: indexing embeddings, query embeddings, and answer generation. If E is the total indexing token count, Q the expected query-embedding tokens, I the generation input tokens, and O the generation output tokens, the estimate is E × embedding_rate + Q × query_embedding_rate + I × input_rate + O × output_rate. Rates must use the billing units in the provider's published price sheet; normalize them before multiplication. I'm not sure what your real query and output totals will be until production traffic exists, so show assumptions and a low/base/high range instead of presenting one precise number as truth. Token counting belongs beside the exact text transformation it measures. Count after normalization and redaction, not before. Store totals per invoice, page, chunk, batch, and schema version. With that ledger, a team can test a proposed chunking change against a representative corpus without sending it anywhere: compare chunk count, token count, duplicated header text, and the number of table rows split across boundaries. Generation usually deserves the sharper limit — retrieve a modest candidate set, apply metadata filters, rerank if the evaluation shows a benefit, and pass only the evidence needed for the requested fields. Asking the model to reproduce a full invoice when the caller needs invoice_number, due_date, and total wastes context and makes validation harder. A strict function or tool schema can constrain the shape of model-produced arguments, but the application must still validate those values and reconcile them with source evidence.
A minimal provider boundary in Python
The production service may be Node.js; the boundary below is intentionally language-neutral in shape and shown in Python because the important artifact is the contract. An adapter owns tokenization, embedding transport, and generation details. Domain code owns invoice IDs, manifests, evidence, and validation. No commercial endpoint leaks into either record.
from dataclasses import dataclass
from decimal import Decimal
from typing import Callable, Protocol, Sequence
@dataclass(frozen=True)
class Chunk:
chunk_id: str
invoice_id: str
page: int
text: str
token_count: int
@dataclass(frozen=True)
class Rates:
embedding_per_token: Decimal
input_per_token: Decimal
output_per_token: Decimal
class Embeddings(Protocol):
def embed(self, texts: Sequence[str]) -> Sequence[Sequence[float]]:
...
def make_chunks(
invoice_id: str,
pages: Sequence[str],
count_tokens: Callable[[str], int],
) -> list[Chunk]:
chunks: list[Chunk] = []
for page_number, text in enumerate(pages, start=1):
normalized = " ".join(text.split())
if not normalized:
continue
chunks.append(
Chunk(
chunk_id=f"{invoice_id}:page:{page_number}",
invoice_id=invoice_id,
page=page_number,
text=normalized,
token_count=count_tokens(normalized),
)
)
return chunks
def pack_batches(
chunks: Sequence[Chunk],
max_items: int,
max_tokens: int,
) -> list[list[Chunk]]:
if max_items < 1 or max_tokens < 1:
raise ValueError("batch limits must be positive")
batches: list[list[Chunk]] = []
current: list[Chunk] = []
current_tokens = 0
for chunk in chunks:
if chunk.token_count > max_tokens:
raise ValueError(f"chunk exceeds token limit: {chunk.chunk_id}")
would_overflow = (
len(current) >= max_items
or current_tokens + chunk.token_count > max_tokens
)
if current and would_overflow:
batches.append(current)
current = []
current_tokens = 0
current.append(chunk)
current_tokens += chunk.token_count
if current:
batches.append(current)
return batches
def estimate_cost(
embedding_tokens: int,
generation_input_tokens: int,
generation_output_tokens: int,
rates: Rates,
) -> Decimal:
return (
Decimal(embedding_tokens) * rates.embedding_per_token
+ Decimal(generation_input_tokens) * rates.input_per_token
+ Decimal(generation_output_tokens) * rates.output_per_token
)
The example refuses an oversized chunk instead of silently dropping it, and it receives a tokenizer callback rather than pretending that characters or words equal tokens. In a real invoice parser, make_chunks should consume layout-aware blocks rather than one page string. Its deliberately small implementation makes that missing production concern visible.
Portability has limits. Token counts can change when the tokenizer changes, embedding vectors from different configurations shouldn't share an index, and structured-output behavior must be evaluated per adapter. If a provider-specific capability materially improves extraction quality, keep that adapter and accept the coupling; don't bury it behind an interface that falsely promises identical semantics.
Roll out with a shadow index, then migrate
Start with a fixed evaluation set of redacted invoices and expected fields. Include multi-page tables, repeated totals, credit notes, scanned pages, ambiguous dates, and invoices whose filename was reused. Score field accuracy and evidence location separately. A correct value with the wrong cited page is a warning, especially when downstream staff use the citation to approve payment.
Build a new index beside the current one whenever chunking, normalization, embedding configuration, or authorization metadata changes. Replay the evaluation set, compare retrieval candidates and extraction results, then send a small slice of read traffic to the new index. Keep writes dual-recorded only for the migration window, and make the ledger show which version owns each query. Rollback should be a routing decision, not an emergency re-embedding job.
Watch a few operational signals: tokens per invoice, rejected chunks by reason, queue age, retry count, retrieval hit rate on the evaluation set, missing required fields, evidence-page disagreement, and generation input tokens per successful extraction. Cost without quality is noise. Quality without traceability won't survive an invoice dispute.
