Every few weeks the same post does the rounds. Someone's routing layer cut their LLM bill by 60%, "with no drop in quality", and here's a lovely chart to prove it.
Except it doesn't prove anything. The eval set is private. The grading is private. The actual model responses that produced that score are never published. You're being asked to trust a screenshot, and we've been in I.T. long enough to know roughly what a screenshot is worth.
So we built the thing that was missing. It's called OmnisBench, it's open source (Apache 2.0), and its whole job is to measure how good LLM routing actually is, in a way you can check for yourself.
What routing is, and why it should save money
If you're not knee deep in this yet: an LLM router looks at each request and sends it to the cheapest model that can still do the job, rather than firing everything at one expensive frontier model. Easy prompts go to something small. Hard ones go to the big model. In theory you keep most of the quality and pay a fraction of the bill.
That's the theory. OmnisBench measures the practice.
The number we care about most is what we call the oracle. For every task, the oracle picks the cheapest model that actually got it right. We know which ones did, because we ran them all. That is not a router you can ship. It's the ceiling. It's the best any router could possibly do on a given set of tasks, and it tells you how much money is genuinely on the table before anyone starts making claims.
The first run
We ran it over 364 tasks: HumanEval (164 coding problems, graded by their own unit tests) and GSM8K (200 maths problems, graded on the answer). The model pool was Claude Opus 5, GPT-5, Claude Haiku 4.5 and GPT-5-nano.
Here is what came out.
| Policy | What it does | Task success | Cost per 1,000 requests |
|---|---|---|---|
| oracle | cheapest model that actually solved each task | 99.7% | $0.62 |
| always-frontier | everything to Claude Opus 5 | 99.2% | $6.25 |
| random | pick a model at random | 96.2% | $4.01 |
| always-cheapest | everything to GPT-5-nano | 94.5% | $0.43 |
So the headline writes itself. Ideal routing hit 99.7% success for roughly 90% less than sending everything to the top model. Very nice.
Now let me talk you out of it a little.
The honest bit
Two things you will not see on most of those 60% charts.
First, the oracle is a ceiling, not a product. It cheats. It knows, after the fact, which model got each task right, and no live router has that luxury. The real score for a router you can actually deploy is how close it gets to that oracle line, and the gap is the interesting part. Anyone quoting an oracle-style number as their shipping product is selling you the ceiling.
Second, and this is the one that surprised us: the cheapest model on its own already scored 94.5%. On this particular set of tasks GPT-5-nano is just quietly good. So routing is not buying you a magic 60%. It's buying you the last few points of quality without paying frontier prices for all of it. That is still very much worth having. It just isn't the fairy tale.
We would rather tell you that up front than have you find out after you have quoted the shiny number to your boss.
The part that actually matters: you can check it
This is the whole reason OmnisBench exists. It does not ask you to trust the table above. It publishes the actual model response for every task, and ships a command that re-grades the lot offline.
pip install -e .
python scripts/prepare_datasets.py # HumanEval + GSM8K
python -m omnisbench.cli run --config configs/v0.yaml --run runs/mine
python -m omnisbench.cli report --run runs/mine
python -m omnisbench.cli verify --run runs/2026-08-19 # re-grades the published run, no API calls
verify re-runs the graders against the published responses and rebuilds the entire leaderboard, cost included, without calling a single API or needing a key. Tamper with one stored answer and it fails. That is the point. If our numbers are wrong, you can prove it in about a minute, and we would genuinely like you to try.
Have at it
The results, the method and the honest caveats are all on the site, and the code is on GitHub:
- Results: https://omnisbench.fortitude-omnis.group/
- Code: https://github.com/Fortitude-Group/OmnisBench
We are also building an actual open router, OmnisRouter, which this benchmark will grade on every release, so it can't quietly regress and hope nobody notices. More on that when it's ready.
If you can break the numbers, poke a hole in the method, or you just reckon we've measured the wrong thing, tell us. That is what open and verifiable is for.
OmnisBench is built by Fortitude Omnis. We make small, sharp tools, and we try not to lie in our own marketing.
