An AI product can become expensive without doing anything obviously wrong.
The prompts work.
The model answers correctly.
Users are getting value.
Then usage grows and the inference bill grows much faster than expected.
One common reason is architectural:
every task is being sent through roughly the same model path.
A document extraction step uses the same model as a difficult reasoning task.
A simple classification gets the same reasoning effort as a complex investigation.
A repeated 20,000-token workspace context gets sent again and again.
A model receives hundreds of tool results just to filter and sort them.
Nothing is technically broken.
The workflow is simply spending expensive model intelligence on work that does not always need it.
A cost-aware model router fixes that by deciding how each task should run before the request reaches the model.
Start with tasks, not models
A weak routing design usually begins like this:
const response = await ai.generate({
model: "strongest-model",
input
});
Every feature eventually calls the same helper.
That is easy to build.
It also hides the economics.
A better starting point is to describe what the task actually requires.
For example:
type AITask =
| "extract"
| "classify"
| "summarize"
| "support_answer"
| "research"
| "decision"
| "complex_agent";
Now the application has something meaningful to route.
The model choice becomes a consequence of the job instead of a hard-coded default.
Add a task profile
The task name alone is not enough.
Two extraction tasks may have very different requirements.
A short invoice and a 200-page legal document should not necessarily follow the same path.
Represent the requirements explicitly.
type TaskProfile = {
task: AITask;
complexity:
| "low"
| "medium"
| "high";
latency:
| "live"
| "interactive"
| "background";
reasoning:
| "minimal"
| "low"
| "medium"
| "high";
volume:
| "low"
| "medium"
| "high";
deterministicPostProcessing: boolean;
};
A simple extraction feature could then declare:
const profile: TaskProfile = {
task: "extract",
complexity: "low",
latency: "interactive",
reasoning: "minimal",
volume: "high",
deterministicPostProcessing: true
};
A difficult research workflow might look very different:
const profile: TaskProfile = {
task: "research",
complexity: "high",
latency: "background",
reasoning: "high",
volume: "low",
deterministicPostProcessing: false
};
The router now has product context.
Create a small number of execution tiers
Do not start with twenty routing combinations.
Three tiers are enough for many products.
type ModelTier =
| "economy"
| "balanced"
| "frontier";
Think about them by responsibility.
Economy
Useful for high-volume work with predictable output.
Examples:
- extraction
- classification
- tagging
- simple transformations
- repetitive structured decisions
Balanced
Useful when the task needs stronger interpretation but still runs frequently.
Examples:
- support responses
- document summaries
- workflow routing
- moderate tool use
- customer-facing assistants
Frontier
Reserve this for work where better judgment materially changes the outcome.
Examples:
- ambiguous research
- complex agent orchestration
- difficult coding
- multi-source reasoning
- consequential recommendations
The names can change.
The boundary is what matters.
Build the router around the task profile
A first version does not need machine learning.
Rules are often easier to inspect.
function chooseTier(
profile: TaskProfile
): ModelTier {
if (
profile.complexity === "high" ||
profile.reasoning === "high"
) {
return "frontier";
}
if (
profile.complexity === "medium" ||
profile.reasoning === "medium"
) {
return "balanced";
}
return "economy";
}
Then map each tier to the model configuration you currently prefer.
const MODEL_CONFIG = {
economy: {
model: "cost-optimized-model",
reasoningEffort: "minimal"
},
balanced: {
model: "balanced-model",
reasoningEffort: "low"
},
frontier: {
model: "frontier-model",
reasoningEffort: "high"
}
};
Keep product logic separated from provider configuration.
When pricing or model performance changes, you can update the mapping without rewriting every feature.
Do not assume maximum reasoning is better
Reasoning effort is another routing decision.
If the task is:
Extract the invoice number, customer name, and total.
high reasoning may add cost without adding useful product value.
For a task like:
Compare these five contracts, identify conflicting obligations, and explain which interpretation is best supported.
the extra reasoning may be justified.
So route reasoning independently when possible.
function chooseReasoning(
profile: TaskProfile
) {
if (profile.reasoning === "high") {
return "high";
}
if (profile.reasoning === "medium") {
return "medium";
}
return "minimal";
}
This gives you another economic control without changing the user experience.
Move deterministic work out of the model
This can remove surprising amounts of token usage.
Imagine an agent retrieves 200 records and needs to:
- remove records older than 30 days
- sort by transaction value
- keep the top 20
- group them by account
- ask the model which groups deserve attention
The model does not need to perform steps 1 through 4.
That work is deterministic.
Write code for it.
const relevant = records
.filter(isWithinLast30Days)
.sort((a, b) => b.value - a.value)
.slice(0, 20);
const grouped = groupByAccount(relevant);
Now send the smaller result into the model.
const judgment = await analyzeAccounts(grouped);
The model spends tokens on judgment.
Code handles filtering, sorting, counting, and aggregation.
OpenAI highlights this same separation in its GPT-5.6 guidance, describing workflows where programmatic tool calling processes deterministic intermediate data outside the model context so model tokens stay focused on reasoning.
Cache context that keeps repeating
Many SaaS AI features send large stable prefixes repeatedly.
Examples include:
- company instructions
- workspace policies
- product catalogs
- long system prompts
- tool definitions
- organization context
If the first 25,000 tokens are almost identical across several requests, repeatedly processing that prefix creates unnecessary cost.
Use prompt caching where the provider supports it.
Also structure prompts so stable content remains stable.
Bad:
Timestamp
Dynamic metadata
Large company instructions
Tool definitions
User request
Better:
Large stable company instructions
Tool definitions
Stable workspace context
Dynamic metadata
User request
The more stable the reusable prefix is, the more useful caching can become.
OpenAI says GPT-5.6 extends its prompt cache TTL to at least 30 minutes and supports deterministic cache breakpoints, specifically to improve reuse across repeated agent runs.
Treat multi-agent execution as a cost decision too
More agents do not automatically mean better architecture.
Suppose the primary agent creates six subagents.
Each receives context.
Each calls tools.
Each generates reasoning.
Then another model synthesizes the outputs.
That can be useful for work that genuinely benefits from parallel investigation.
It can also multiply token consumption very quickly.
Represent the decision explicitly.
type ParallelPolicy = {
allowed: boolean;
maxAgents: number;
minimumComplexity: "medium" | "high";
};
Then:
function canSpawnSubagents(
profile: TaskProfile,
policy: ParallelPolicy
) {
if (!policy.allowed) return false;
if (profile.complexity !== "high") {
return false;
}
return true;
}
The workflow should earn parallelism.
Do not make subagents the default simply because the API supports them.
Add a cost budget to every feature
Model routing becomes much more useful when the product has an economic boundary.
Define cost at the feature level.
type FeatureBudget = {
feature: string;
maxCostPerRunUsd: number;
warningThresholdUsd: number;
};
For example:
const budget: FeatureBudget = {
feature: "document_enrichment",
maxCostPerRunUsd: 0.15,
warningThresholdUsd: 0.10
};
The exact number should come from your own product economics.
The architecture now knows that cost is a requirement, not merely something observed at the end of the month.
Record what the router decided
Every AI run should leave enough information to explain its cost.
type AIRun = {
feature: string;
task: AITask;
selectedTier: ModelTier;
reasoningEffort: string;
inputTokens: number;
outputTokens: number;
cachedInputTokens?: number;
toolCalls: number;
estimatedCostUsd: number;
durationMs: number;
successful: boolean;
};
Now you can answer useful questions.
Which features are consuming the most AI budget?
Which tasks regularly escalate to the frontier tier?
Is the economy model producing acceptable results?
Did prompt caching reduce repeated input?
Are subagents improving results enough to justify their cost?
Without this data, model routing becomes guesswork.
Measure quality alongside cost
Do not optimize cost in isolation.
A cheaper request that creates more support tickets is not cheaper.
A small model that misclassifies 8% of requests may create expensive downstream failures.
Track a quality measure appropriate for the feature.
For extraction:
Field accuracy
Missing-field rate
Human correction rate
For support:
Resolution rate
Escalation rate
User correction rate
For agents:
Task completion
Tool failure rate
Retry rate
Human intervention
Now compare:
Cost per run
+
Quality
+
Latency
+
Failure rate
That is a much stronger routing signal than price per token alone.
Add fallback intentionally
A cost-optimized model will sometimes fail.
That does not mean every request must start with the expensive model.
Use escalation.
Economy model
↓
Quality check passes?
↙ ↘
Yes No
↓ ↓
Return Balanced model
↓
Still uncertain?
↙ ↘
No Yes
↓ ↓
Return Frontier model
This can keep the common path inexpensive while preserving a stronger path for difficult cases.
The quality gate might be:
- schema validation
- confidence threshold
- deterministic rule
- evaluation model
- missing evidence check
- human review
Choose it around the workflow.
Example: document processing
A document workflow might use several tiers.
PDF uploaded
↓
Extract text
↓
Economy model:
classify document
↓
Economy model:
extract known fields
↓
Validation
↓
Missing ambiguity?
↙ ↘
No Yes
↓ ↓
Save Balanced model:
resolve context
↓
Consequential decision?
↙ ↘
No Yes
↓ ↓
Save Frontier model
or human review
The workflow does not sacrifice intelligence.
It spends intelligence where ambiguity increases.
Example: an AI support product
The same pattern can apply to support.
Customer message
↓
Economy model:
intent classification
↓
Deterministic routing
↓
Retrieve account + docs
↓
Balanced model:
prepare response
↓
High-risk action requested?
↙ ↘
No Yes
↓ ↓
Reply Frontier reasoning
+ approval boundary
Again, the expensive path is available.
It is simply not the default for every message.
A practical rollout plan
Do not replace every model path at once.
Start with one high-volume feature.
Step 1: Baseline it
Record:
- request count
- model
- reasoning effort
- input/output tokens
- average cost
- latency
- quality metric
Step 2: Break the workflow into tasks
Identify which steps require:
- extraction
- classification
- deterministic processing
- interpretation
- difficult judgment
Step 3: Test cheaper paths offline
Run representative inputs through alternative configurations.
Compare quality before changing production routing.
Step 4: Introduce routing
Start with a narrow percentage of traffic.
Step 5: Add escalation
If the cheaper route is uncertain, move the request upward.
Step 6: Watch cost per successful outcome
Do not stop at token savings.
Measure whether the product still completes the job properly.
OpenAI's GPT-5.6 guidance points in this direction
OpenAI's recent builder guide describes several production teams reducing AI costs through smaller models, lower reasoning effort, prompt caching, and architectural changes.
It also argues that many workflows no longer need a frontier model at every stage.
The exact model choices will continue changing.
That makes the architecture behind selection more valuable than any single recommendation.
Build routing as product infrastructure
Model pricing will change.
New models will arrive.
Capabilities will overlap.
Reasoning controls will change.
Latency will improve.
If every feature directly chooses its own provider model, each change becomes a migration project.
A central task-aware router gives you a stable product boundary.
The feature says:
This is the job I need done.
The routing layer decides:
What is the lowest-cost path that can do it reliably?
That is a much healthier economic contract for an AI product.
Source
OpenAI, The builder's guide to GPT-5.6
