Not Every Prompt Deserves a Frontier Model: A Practical Router for LLM Traffic

Last quarter I pulled the request logs from three of my hobby projects and grouped the LLM calls by what they were actually doing. The result stung a little. Roughly seven out of ten calls were mechanical busywork — classifying inbound messages, reformatting dates, turning verbose logs into short status lines. All of it was hitting the most expensive model in my config, not because the work demanded it, but because I had set a default once and never questioned it.

The fix turned out to be embarrassingly simple: treat LLM calls like any other tiered resource. Give the trivial work to a free model, reserve the paid model for work where errors actually hurt, and put a tiny routing layer in between. This post covers the triage rule, the router code, and the small test harness that keeps the cheap lane from quietly rotting.

Triage by blast radius

I sorted every call type in my system with one question: when this output is subtly wrong, who notices, and when?

The rule underneath the table: a task qualifies for the free lane only when a mediocre answer is either harmless or gets intercepted by a person or a validator before it matters. That call is contextual — nobody's benchmark suite knows what a mistake costs inside your pipeline.

The routing layer

I kept the router as boring as possible. It's a static lookup, and it fails toward spending money, not toward saving it:

// router.mjs — deliberately unglamorous
import OpenAI from "openai";

// Lane 1: zero-cost tier. Mine runs on MonkeyCode's free model access
// via its free server option; at hobby traffic levels, these calls
// cost me nothing.
const freeLane = new OpenAI({
  baseURL: process.env.FREE_LANE_URL,
  apiKey: process.env.FREE_LANE_KEY,
});

// Lane 2: paid model, used only when the work justifies it.
const paidLane = new OpenAI({ apiKey: process.env.PAID_LANE_KEY });

const FREE_LANE_MODEL = process.env.FREE_LANE_MODEL;
const PAID_LANE_MODEL = process.env.PAID_LANE_MODEL;

const freeEligible = new Set([
  "classify_message",
  "normalize_format",
  "condense_log",
  "draft_alt_text",
]);

export async function route(taskKind, prompt) {
  // Unknown task kinds fall through to the paid lane on purpose:
  // an unclassified task hasn't earned anyone's trust yet.
  const useFree = freeEligible.has(taskKind);
  const client = useFree ? freeLane : paidLane;
  const model = useFree ? FREE_LANE_MODEL : PAID_LANE_MODEL;

  const res = await client.chat.completions.create({
    model,
    temperature: 0.1,
    messages: [{ role: "user", content: prompt }],
  });
  return res.choices[0].message.content;
}

Three decisions I'd argue for:

1. Default to the expensive lane. New task types get premium treatment until I've reviewed them. Overpaying for a log summary is a rounding error; under-powering a data cleanup is a liability.
2. No model picks the model. An LLM-based router adds a billable call and a fresh failure mode to every request. At this scale, a set you can audit in five seconds is better engineering.
3. The router knows nothing about providers. Two OpenAI-compatible clients and environment variables mean I can swap either endpoint by editing config, not code.

Don't trust the free lane — test it

"Seems fine" is how free tiers quietly degrade. I built a small harness around real samples instead. For each task type I pulled ~50 historical inputs from my logs and wrote a pass/fail check specific to that task: exact-match for classification, schema validation plus spot checks for formatting, and a short rubric for text drafts (names the right component, invents no facts, fits the length budget).

// eval.mjs — run weekly, takes a few minutes
import { route } from "./router.mjs";
import samples from "./eval_set.json" assert { type: "json" };

for (const group of samples) {
  let pass = 0;
  for (const item of group.cases) {
    const output = await route(group.kind, item.input);
    if (await eval(group.check)(output)) pass++; // or plain functions per task
  }
  const rate = pass / group.cases.length;
  console.log(`${group.kind}: ${(rate * 100).toFixed(1)}%`);
  if (rate < group.floor) {
    console.warn(`⚠ ${group.kind} fell below its floor — investigate`);
  }
}

I run this weekly against the free lane and log the scores. That habit has already caught one regression: after what I assume was a backend model swap, classification accuracy slid about nine points, and I temporarily moved that task back to the paid lane until it recovered. Casual usage would never have surfaced that drift.

One caveat about the numbers: my pass rates reflect my prompts and my checks. They tell you nothing about your workload. Build the eval set from your own logs — that's most of the value of the exercise anyway.

About the free endpoint

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

My free lane currently runs through MonkeyCode's free model access on its free server option. That made it cheap to prototype the whole routing design before I was sure it was worth keeping. But look at what the system actually depends on: an OpenAI-compatible interface and a routing table. MonkeyCode fills lane one today; any compatible endpoint could fill it tomorrow with two env-var edits. The durable idea here is the lane discipline, not the vendor.

Where this pattern breaks down

• You need logs first. Routing requires a catalogue of your actual call types. Capture a week or two of traffic before writing a single line of dispatch logic.
• Free tiers carry no promises. Expect rate limits, latency variance, and occasional silent model changes. Anything user-facing or latency-critical belongs on a paid tier with an SLA.
• Data sensitivity is a hard gate. Read the provider's data policy before sending anything containing user data or proprietary code through a free tier. My free lane only ever touches non-sensitive content.
• Homogeneous workloads don't benefit. If 95% of your calls are genuinely hard reasoning, you've built indirection for nothing. If 95% are trivial, skip the router entirely and just point the client at the free tier.
• Human review is structural, not optional. Half the rows in my table qualify for the free lane only because a person or a validator sees the output. Remove that checkpoint and re-sort the table.

Takeaway

The cheapest LLM call is the one that never reaches a paid model. A static routing table that defaults to safety, plus a weekly eval run built from your own logs, is enough to capture most of that saving in an afternoon — and none of it locks you to a provider. If you want a zero-cost endpoint to experiment with on the cheap lane, MonkeyCode's free model access is an easy starting point; just make sure the triage judgment stays yours.