I generate images locally in batches, and a vision model scores each one before anything ships. Theme, humour, wit, background, one composite number. Anything under the bar gets rebuilt.
That ran for weeks. Then I dumped the raw scores instead of the pass/fail summary and actually looked at them.
40 images, one slate, qwen2.5vl:7b
theme 9 on all 40
humor 7 on all 40
wit 8 on 37 of 40
score 8.38 to 8.82
Humour came back as 7 forty times in a row. The best image on that slate scored 8.82 and the worst scored 8.38, which is a range of 0.44 on a ten point scale. Nothing was ever going to fail that gate.
Why I didn't spot it sooner
Because the numbers looked fine, which is the whole problem.
They sat in a sensible range. They had decimals. The composite even wobbled between images, because bg had some genuine variance and dragged the average around by tenths, so no two images ever printed exactly the same final score. That wobble is what sold it. It looked like a measurement.
If your judge throws an exception you fix it that afternoon. If it hands back 8.6 for everything, you skim it, think "yeah, slate's decent", and run on it for a month.
So: go and take the standard deviation of each axis over your last run. Five lines. If an axis is flat it isn't measuring anything, and no individual score will ever tell you that.
Writing a better prompt didn't help
First thing I assumed was that my rubric was too woolly, so I rewrote it with hard numeric anchors and told the thing to be a harsh critic.
HUMOR anchors: 0-2 nothing amusing; 3-5 mildly charming; 6-7 a clear visual joke;
8-10 genuinely funny, a character is REACTING.
WIT anchors: 0-2 theme applied as texture; 3-5 sensible pairing; 6-7 real idea;
8-10 clever twist.
Same model, same 40 images. Humour loosened up and started using 2 through 5. Wit then collapsed instead, 7 on 38 of 40, standard deviation 0.22.
All I'd done was move the flat axis. That's when I gave up on prompt fixes and went looking for a different model.
What actually discriminates
Same 40 images, same anchored prompt, three models. I'm quoting standard deviation rather than min to max, because a single outlier makes a rubber stamp look like it's doing something.
| model | humour sd | wit sd | distinct wit scores | text defects found | speed |
|---|---|---|---|---|---|
| qwen2.5vl:7b | 0.99 | 0.22 | 2 | 0 of 40 | 1.0 s |
| qwen3-vl:30b-a3b-instruct | 0.82 | 1.12 | 6 | 0 of 40 | 2.6 s |
| qwen3-vl:32b-thinking | 2.54 | 2.33 | 10 | 4 of 40 | 164.7 s |
The histograms make it more obvious than the summary stats do.
qwen2.5vl:7b wit 7 on 38 of 40
qwen3-vl:30b-a3b humour 5 on 32 of 40
qwen3-vl:32b-thinking humour nine different values, 0 through 8
The 30b deserves better than I first gave it, though. It isn't a rubber stamp. It reliably drops the images that ignored the brief to a 2 while everything else gets a 5, so there's real pass/fail signal in there. It just can't rank anything that passed, which was the bit I actually wanted.
Only the 32b-thinking used the range it was handed.
The text check
The other thing I want from a judge is catching mangled text, because generated images produce garbled shop signs constantly and it's the fastest tell there is.
The 32b flagged 4 of 40. I opened all four. A garbled banner strung over a street scene, a shopfront reading "FRAME D", another row of nonsense signage, and a sports jersey with a melted logo on the chest. All four are real.
The other two models flagged nothing, on the same images, with the same instruction in the prompt.
Worth saying I only checked the four it flagged, not the thirty six it didn't, so that's precision and I have no idea about recall.
qwen3-vl:32b returns an empty string if you ask it for JSON
This one cost me an afternoon and I can't find it documented anywhere.
Ollama takes a format: "json" parameter that constrains the output to valid JSON, which is the obvious thing to set when you want a score sheet back. On qwen3-vl:32b you get a zero length response. No error, no exception, nothing in the logs, just an empty string where your JSON should be.
Four runs, same image:
qwen3-vl:32b format:json=True -> len=0 22.9s
qwen3-vl:32b format:json=False -> len=103 121.8s {"humor":8,"wit":9,...}
qwen3-vl:32b-thinking format:json=True -> len=0 6.2s
qwen3-vl:32b-thinking format:json=False -> len=106 146.6s {"humor":6,"wit":8,...}
Drop the format hint and both give you clean JSON first try.
I'd originally filed this under "thinking models are weird about structured output", figuring the constraint was fighting the reasoning block. Wrong. The dense non-thinking 32b does exactly the same thing. And qwen3-vl:30b-a3b-instruct handles format: "json" without complaint, so it isn't an ollama-wide thing either. It follows the model.
Two other bits if you go down this road. Parse the last {...} in the response, not the first, because a thinking model will happily emit braces while it reasons. And give it a real num_predict, around 3072, or it burns the whole budget thinking and never gets to the answer.
That 30b isn't a 30b
qwen3-vl:30b-a3b-instruct sits directly next to qwen3-vl:32b in the library, and the names suggest two roughly comparable models.
The a3b is doing a lot of quiet work in that tag. It's a mixture of experts with about 3B parameters active per token. The dense 32b runs all 32B. For "is this actually funny", which needs the model to hold a whole image against a rubric and then commit to an opinion, that difference is not subtle.
If you're picking local models off a list by the number in the name, check whether it's total or active first.
What I'm not claiming
I measured spread, not correctness. All this tells you is whether a model can tell your images apart. It says nothing about whether it's right. I have no human labelled ground truth for this set, so a model could hand me a lovely wide spread of confidently wrong numbers and my method would call it a winner. Discrimination is necessary, not sufficient.
One rubric, one prompt, one domain, my hardware.
The rubber stamp finding is the solid one. That's 40 rows of real output and the flat axes stay flat regardless of what the model comparison says.
What I changed
The good judge is slow. 164.7 seconds an image is nearly two hours for a 40 image slate, which isn't a gate, it's a second render pass. So it runs as a cascade now. The cheap model screens the lot and throws out the obvious failures, which is the one thing it's reliably good at, and only the survivors go to the expensive one. Puts a slate at roughly 25 minutes.
The smaller change is the one I'd actually recommend. The gate prints the standard deviation of every axis next to the scores now, so if an axis comes back flat I find out on the first run instead of the fortieth.
I use Claude for a lot of this work, including a hand with the writing. The measurements are mine and they're reproducible, every number above came off my own machine.
