Ranking Language Models by How Well They Spot Liars

Code: Megapixel99/social-deduction-bench

A seven-player game of Mafia is a good test of whether you can tell who's lying, and a terrible one to score by who won. The Mafia can lose with perfect play because the Detective's first coin-flip investigation happened to land, and win with incoherent play because two villagers fixated on each other. Rank models by win rate over any batch you can afford to run and you're mostly ranking luck.

So I built the harness to score something narrower. Every town player names one public suspect per statement. The engine knows the roles, and it records the exact probability of hitting a Mafia by chance for that turn's living roster. Accuracy minus that baseline is a signed number with a meaningful zero, and it gives you 15 to 25 samples per game instead of one bit. I called it detection lift.

Then I ran 522 games across 19 models and found out that the first two things the instrument measured were both properties of the instrument.

(The concept, the research questions and the design are mine; much of the implementation was AI-assisted, which the repo says on its front page and I'll repeat here. The findings are exploratory.)

The leaderboard reordered itself while I watched

Run 007 seated two gpt-oss:20b, one qwen3:4b, one qwen3.5:2b, one gemma3:1b and two copies of a rule-based control at seed 7, then played 60 games. I checked the interim table at 8 games, because that's what you do while a batch is running.

At 8 games, gpt-oss:20b led at +0.137 and qwen3:4b was last at −0.400. That's a spread of more than half a point, running in the direction you'd expect if parameter count buys detection. It would have made a perfectly readable chart.

At 60 games, same run, same seed, same code, nothing restarted:

qwen3:4b went from −0.400 to +0.078 without anything about it changing. Every model and the control now sit inside a tenth of a point, and the 20B model is sandwiched between a 2B and a 4B. The ordering that looked like a scaling result at 8 games is gone.

An earlier six-game batch on the same roster and seed had qwen3:4b on top at +0.600 and gpt-oss:20b at the bottom on −0.378, which is the reverse of the 8-game reading and the reverse of the reversal. I'll note that one had a different code version (a token-budget fix landed between them), so it isn't a clean sample-size comparison; the 8-versus-60 pair inside run 007 is, and it's enough on its own.

The control is a ruler, because it cannot learn

The useful thing about seating a rule-based policy at the table is that it's a fixed program. It doesn't have a temperature, it doesn't have a bad day, and its code is byte-identical in every batch. So any variation in its score is sampling noise by construction, which makes it a ruler you can hold against every other row.

Across five early batches the control scored +0.293, +0.133, −0.114, +0.125 and +0.146 on an unchanged policy. Pooled over all 27 batches it ever appeared in, its range is −0.125 to +0.416. That's a span of more than half a point, produced by a thing that does the same arithmetic every time. It is wider than the gap between any two models I've measured.

Split those 27 batches by size and the shape is completely ordinary:

Every extreme reading this project ever produced came from a batch of 12 games or fewer. That's 1/√n behaving as it should: reassuring about the harness, damning about the numbers I'd been reading off it. For a while I'd described the control's disagreement with itself as an irreducible measurement floor. That was wrong and flattering; it's just noise, and it prices a readable arm at around 30 games.

Day 1 is exactly chance, and most turns are day 1

The second finding is bigger, and I only found it because I stopped reading batch summaries and re-read the per-game JSON: 522 finished games, 3,368 individual town accusation records, broken out by which day of the game they happened on.

Day 1 is chance to within 0.005 on n=2,103. Not "models are bad at this"; there is provably nothing to know yet. Nobody has voted, nobody has contradicted themselves, and no one has died, so the only available evidence is prose style. Accuracy climbs steeply once a record exists to read.

Now put that next to the mean game length, which is 2.5 days. Most of the turns behind every pooled score in this project are day-1 turns where the correct answer was unobtainable. So a pooled detection number is diluted by unanswerable questions, and the dilution differs per model. A model whose games end quickly is judged mostly on its worst-information turns, and scores worse for a reason that has nothing to do with how well it reads people.

It also explains a structural result I'd been treating as a curiosity. Town won 88 of 522 games (16.9%), but that splits into 18.6% at seven seats (n=415) and 9.5% at ten (n=95). A bigger room means proportionally more day-1 turns, and day-1 turns carry no signal. The setup rather than the players is doing much of the work here, so any claim about town performance has to be read against a 16.9% base rate.

Two competences that pooling destroys

Reading detection lift per day rather than pooled turned up the one result that made the whole re-analysis worth it. The day curves don't just differ in level; they differ in shape.

The control starts at chance and compounds hard, which follows from what it is. Its policy reads an accumulating table of who accused whom and who voted how, and that table doesn't exist on day 1. gpt-oss:120b does the opposite. It has real cold-read signal where the control has none, and then it fails to compound.

Those are two different skills, and a single pooled number rates the two players as near-equal while they're doing opposite things. That's the sharpest argument I have for measuring per turn instead of per game. It arrived as a side effect of investigating a problem with my own reporting.

What actually survives

With the noise accounted for, here's the honest version, using a 95% cluster bootstrap over games rather than seats. That distinction matters. Seats within one game are not independent, since one player's hit is another player's miss, and resampling seats would narrow every interval in exactly the direction that manufactures findings.

gpt-oss:120b is the only model that clears zero on a real sample, and its interval still overlaps the control's. So it beats chance; it is not shown to beat a few dozen lines of voting-record policy. Three models sit significantly below chance, which is stranger than it sounds, and it isn't only protocol failure. qwen3:4b's invalid-move rate is 0.089, so it was making legal moves and choosing wrong more often than a coin would.

The one clean explanatory result in the whole log came from two anchored 30-game batches. mistral-small:24b (23.6B parameters, no reasoning mode) scored −0.001, which is chance to three decimals, at 18.3 seconds per call. gpt-oss:20b (20.9B, low reasoning effort) scored +0.113 at 4.2 seconds. The smaller model is both better and cheaper. Neither result is a formatting artefact (invalid rates 0.000 and 0.013), and the anchor seats common to both batches drifted by a mean of about −0.010, so the two batches really are comparable. What buys detection here is deliberate inference at generation time, not scale.

Check it in 11 seconds

The corpus itself isn't public (logs/ is gitignored), so the 522-game numbers above rest on my analysis of data you can't download. What you can do is regenerate the two structural findings from scratch. No API keys, no GPU and no model downloads, because the rule-based control plays itself:

for s in 1 2 3 4 5 6 7 8; do node src/index.js --test --games=12 --seed=$s; done

for s in 11 12 13 14; do node src/index.js --test --games=30 --seed=$s; done

node analysis/extract.js && node analysis/analyze.js

That's 216 games in 11.1 seconds on my machine. On my run, the eight 12-game batches of a policy that cannot learn spread from −0.065 to +0.239 (span 0.304), while the four 30-game batches spread from −0.003 to +0.090 (span 0.093). And the day breakdown across all 216: +0.011 on day 1 (n=905), +0.423 on day 2 (n=357), +0.518 on day 3 (n=55).

Those aren't the corpus numbers and shouldn't be; an all-scripted table is a far more predictable game than one with models in it, so the compounding is steeper. What reproduces is the shape, which is the part the argument rests on. A fixed policy scores differently every time you ask it, by an amount that shrinks with the batch size, and detection on day 1 doesn't exist.

The failure mode worth naming

Five of the defects in this project's log are the same bug wearing different clothes, and the last one is the clearest. A batch hit the Ollama Cloud free-tier quota, and the rate-limit-exhausted signal was caught by a generic catch before the batch runner could see it. The engine recorded an API error and substituted a seeded random move. So a quota refusal was being counted in the invalid-move rate exactly as though the model had failed to decide: an infrastructure outcome reported as a model result, inside the very number the run existed to measure.

The same class shows up in a design decision I nearly got wrong from the opposite direction. It's tempting to constrain the target of an accusation to an enum of living players, since that makes illegal moves impossible. It would also have silently zeroed the invalid-move rate and the dead-player-named counter, which are the metrics that measure whether a small model can track a shrinking roster. That's one of the clearest small-model findings here, and the constraint would have deleted it while presenting the result as a capability.

What I'd take from all of this is narrower than "use bigger samples", which everybody already knows and nobody's small-n blog post obeys. It's that a benchmark should contain something that provably cannot improve. The rule-based control went in as a baseline to beat, and it turned out to be worth more as an instrument than as an opponent. Because its policy is fixed, every point of spread it shows is a direct readout of the harness's own noise, in the same units as the thing I'm trying to measure. I did not have to reason about what my error bars should be. I could look up what they were.

Without that row I'd have published the 8-game table. It's a good table. It has a clean scaling story, the ordering makes sense, and it is entirely wrong.