If you're building on someone else's LLM API, you've probably had this experience: your app has been routing tickets, scoring leads, or gating decisions on a model's answers for months, it's been fine — and then one day it isn't. No error. No changelog. The same input just started coming back with a different answer.
That's what happened to us with TypeSafe's Jev model. We don't control when Jev gets retrained or redeployed. TypeSafe does. So we built a small open-source CLI, jev-watch, to catch it the moment it happens instead of finding out from a support ticket.
The bug that made the case for this tool
Early in building the adapter, we mapped score-type questions onto Jev's noul (yes/no probability) type instead of its native score type. It ran. It returned answers. Nothing crashed. It just silently dropped confidence data and returned worse answers than the model was actually capable of.
We only caught it by comparing live output against a set of answers we knew were correct. That comparison loop — save a known-good answer, re-run later, flag anything that changed — is exactly what jev-watch automates.
How it works
Write a test case as JSON: a scenario, a question, and the answer you know is correct today.
{
"testName": "CRAToolkit refund eligibility",
"state": "Customer says: I was charged twice for my subscription this month and want a refund",
"questions": {
"department": {
"type": "choice",
"instruction": "Which team should handle this?",
"options": {
"billing": "Payment, invoices, subscription issues",
"technical": "Login, bugs, product errors",
"sales": "New purchases, upgrades, demos"
}
}
},
"expected": { "department": "billing" },
"tolerance": 0.1
}
Commit that as your baseline. Re-run it anytime — after a model update, on a schedule, or in CI on every deploy:
$ jev-watch examples
PASS CRAToolkit refund eligibility
PASS CRAToolkit sales lead qualification
PASS CRAToolkit angry customer detection
PASS CRAToolkit support routing
4/4 tests passed
If Jev's answer changes — a choice flips, a score moves past tolerance, confidence drops — you get told exactly what changed, not a vague "something's off":
$ jev-watch examples/support-routing.json
FAIL CRAToolkit support routing
drift [department] choice: expected sales, got technical (tolerance 0.1)
0/1 tests passed
Exit code 0/1, so it drops straight into CI.
Setup
npm install && npm run build
echo "JEV_API_KEY=sk-..." > .env.local # or OPENROUTER_API_KEY, no TypeSafe account needed
node bin/jev-watch.js examples
Works against TypeSafe's API directly, or through OpenRouter if you'd rather not hold a TypeSafe key.
Why this matters beyond Jev
This is really just regression testing applied to model behavior instead of your code. Your code didn't change, your tests still pass, your types still check — but the thing sitting behind the API call did, and none of your existing test suite is watching for that. If you depend on any hosted model you don't control the release cycle of, the same gap exists.
Try it
Repo: https://github.com/akanthed/jev-watch — MIT licensed, v0.1.0, still early. If a model update has ever quietly broken something for you, turn that case into a JSON file and send it as a PR — that's exactly the kind of regression case this project needs more of.
