There's a recurring ritual in every dev community I'm part of: a model drops, someone posts benchmark screenshots, and for two days we debate whether to rewire everything. Then the next model drops and we do it again.
A few months ago I opted out of the debate. Not because model quality doesn't matter — it does — but because I realized I was optimizing the wrong variable. The question that actually changed my week wasn't which model is strongest but which of my requests are strong-model requests at all. Turns out: maybe a third. The rest were getting premium answers to trivial questions.
This post is the habit I built instead. It has three parts: sort work by what a mistake costs, keep a frozen sanity suite of my own tasks, and write the routing rules down somewhere with a conservative default.
Sort work by blast radius, not by how clever it sounds
My old instinct was to send "hard" prompts to the good model. But difficulty and danger are different axes. A tricky regex is hard to write and trivial to verify. A two-line find | xargs rm is easy to write and capable of eating a directory.
So I classify by blast radius now:
- Instantly falsifiable. Formatting, renames, boilerplate, simple transformations. If it's wrong, my compiler or test suite screams within seconds. Any competent model will do — this is the natural home for free or cheap tiers.
- Reviewed by me anyway. Commit messages, changelogs, Slack summaries of a diff, first-draft comments. I'm the verification layer, so a mediocre draft still saves me typing.
- Verifiable only with effort. Anything touching the filesystem, package installs, CI config, database changes. Wrongness here is quiet and expensive. These get the strongest model I have and a sandboxed dry run before anything executes. I wrote a separate piece on dry-running assistant-written shell commands because this category keeps hurting people.
- Judgment calls. Data model design, API shape, splitting a service. There's no test for "good architecture," so the model's role is to produce options I'll interrogate, not answers I'll apply.
The first two buckets are where the money leaks. They were easily 60% of my request volume and were getting the same expensive treatment as the fourth bucket.
Keep a frozen sanity suite, and run it before believing any release notes
When something new trends, benchmark threads tell me how it performs on curated public tasks. What I need to know is how it performs on my tasks — the ones where I already know the correct answer because I shipped it.
So I keep a small frozen suite, maybe a dozen prompts pulled from my own history: a regex I once fumbled, a pytest file with a known correct assertion set, a diff that needs a specific explanation, one nasty quoting problem. Rough sketch (adapt freely):
# Sketch only — wire `ask()` to whatever client you use.
# The critical property: SUITE is append-only. Never edit a case
# because a model failed it.
SUITE = [
Case("quoting-bug", prompt=P1, verify=exact_match(KNOWN_FIX)),
Case("pytest-asserts", prompt=P2, verify=tests_still_pass_and_catch_bug),
Case("diff-summary", prompt=P3, verify=mentions_key_change),
]
for case in SUITE:
answer = ask(candidate_model, case.prompt, temperature=0)
log(case.id, model=candidate_model, ok=case.verify(answer))
The discipline matters more than the code:
- Frozen means frozen. Tuning a case after a failure turns evaluation into flattery.
- Failure shapes get remembered. A model that invents a plausible-but-wrong assertion in the pytest case gets banned from test-writing for me, whatever its total score says.
- It has to be cheap to run. A suite you hesitate to execute is a suite you skip.
That third point is where free tiers earn their keep. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode currently provides free model access plus a free server option, and I've been using that slot as my low-cost tier — it's genuinely handy at evaluation time, since I can rerun the suite against a candidate without watching a token counter. I can't speak to how long that access lasts or what limits apply, so I treat it as a current convenience rather than a foundation.
Write the routing down, and make the default paranoid
Rules I hold in my head evaporate the moment I'm in a hurry. So the policy lives in a file next to my assistant config, something like:
# Routing policy — example, rename buckets to match your work
policy:
low-stakes: # falsifiable or self-reviewed work
model: free-tier
side-effects: # shell, CI, package installs, migrations
model: strong-tier
gate: dry-run-in-sandbox
judgment: # design, cross-cutting refactors
model: strong-tier
gate: human-decides
unclassified: strong-tier # <-- the line that matters most
That last line is the whole ballgame. Routing failures don't look like "cheap model flubbed a rename." They look like a dangerous task getting confidently filed under cheap. When I'm unsure what a request is, it goes up-tier; tasks only move down after the sanity suite or real usage says it's safe. Demotion is earned, never assumed.
Where this breaks down
- Low volume makes it pointless. Two assistant sessions a week? Use the good model for everything and spend this energy elsewhere. The payoff scales with daily, varied usage.
- Free access is weather, not climate. Build so the cheap tier disappearing means "some tasks get pricier," not "workflow stops." If MonkeyCode's free tier went away tomorrow, my setup reroutes that bucket to whatever's next cheapest — annoying, not fatal.
- A dozen cases can't catch subtle drift. The suite filters out gross incompetence. Real regressions surface as real failures, at which point that failure becomes case thirteen.
- Task type isn't the only axis. A "trivial" summarization over a huge context window can exceed what a weaker model holds. If your work is long-context-heavy, classify by size too.
- Some people have no low-stakes bucket. If your day is auth systems, payments, and prod infrastructure, the honest output of this exercise is "everything routes strong-tier." That's a valid result, not a failed one.
The actual point
Model releases are inputs to a routing decision, not occasions for migration. Freeze a sanity suite from your own work, classify by what a mistake costs, write the rules down with a paranoid default, and let free or cheap capacity soak up the work where being wrong is boring. If you want a zero-billing environment to stand that habit up in, MonkeyCode's free models and free server are a reasonable place to start while you figure out your permanent setup.
What's in your low-stakes bucket? I'm curious whether others sort by task kind, by repo, or by whether they'd notice a mistake before it ships.
