Originally published at vinpatel.com
By the end of this you'll know how to build a private benchmark that tells you whether a coding model actually works on your codebase, not someone else's GitHub history.
This matters right now because a company called Specific just shipped Real-SWE, a benchmark that scores AI models against private, real-world, enterprise codebases instead of the public repos every other leaderboard scrapes. That distinction is the whole story: a model that tops a public benchmark has often seen those exact repos, those exact bugs, those exact pull requests during training. Your codebase is not on the internet. No amount of leaderboard glory tells you how a model performs against code it has never seen, written in your patterns, with your conventions, touching your test suite.
Here's the recipe Real-SWE's approach implies, adapted for your own repo:
- Pull a handful of real, already-merged pull requests from your private codebase.
- Strip each one down to just the issue description and the code state before the fix.
- Hand the model only that starting state and the issue, nothing else.
- Ask it to produce a patch, then run your actual test suite against it, no partial credit.
- Score pass or fail against your own tests, not against a rubric a lab wrote for itself.
- Re-run the same set every time you swap models or upgrade a version, so the comparison stays apples-to-apples.
Here's a starting prompt template you can paste into your eval harness today:
CONTEXT: [paste the repo state before the fix, relevant files only]
ISSUE: [paste the original issue description, unedited]
TASK: Produce a patch that resolves the issue above.
CONSTRAINT: Do not modify test files.
OUTPUT: A unified diff only.
Run that same block against every model you're evaluating, then run your real test suite against whatever diff comes back. That's the entire mechanism Real-SWE is built on, and you don't need their infrastructure to start doing it yourself this week.
The gotcha: a model that looks strong on public leaderboards can quietly underperform on your repo, and you will not find out until you test it against your own code. Public benchmark scores are not a proxy for your codebase. Treat them as marketing, not evidence, until you've run your own private set.
One AI signal a day. 90 seconds. No fluff. If you want the next one in your inbox, subscribe at vinpatel.com/subscribe/.
