When a 4B Local LLM Beats 26B on One Task

The smaller model won my latest local test. That sentence is true, but it needs a tight boundary.

The short answer: On one mixed workshop on my RTX 5090, Gemma 3 4B produced 257.1 tokens per second while Gemma 4 26B produced 218.5. Both passed the same four fixed code checks. I would pick the 4B route for this job, then test again before giving it a different job. Canonical URL: https://bmdpat.com/blog/local-llm-4b-vs-26b-task-quality-2026

The measured rows are in Reports/5090/benchmarks/workshop-discoveries.csv, rows 2 through 5. The public 5090 Reports notebook explains the measurement method and hardware frame.

Can a 4B local model beat a 26B model?

Yes, on a named workload with a named pass condition. Model size is not a score.

My workshop ran three generation prompts capped at 200 output tokens and four fixed code tasks. The runtime was Ollama on an RTX 5090. Both rows used a 2,048-token context and num_batch=512.

Gemma 3 4B generated 475 output tokens at 257.1 tokens per second. Gemma 4 26B generated 482 output tokens at 218.5 tokens per second. Both passed all four code checks.

That is a useful result for this workshop. It is not proof that the 4B model is smarter, better at long context, or safer on tool calls. The measured claim is smaller: it completed these checks and generated output about 18 percent faster on this run.

Why does task quality come before tokens per second?

A fast wrong answer is wasted GPU time. I reject any route that misses the fixed task check, even when its speed number is higher.

The same workshop recorded a Gemma 4 26B run at 156.9 tokens per second with zero of four code checks passing. A later setting for that model reached 215.7 tokens per second and passed all four. The model name stayed the same. The usable result changed with the tested configuration.

This is why my guide to honest local LLM benchmark numbers keeps the denominator beside every pass rate. Four of four is a smoke test, not a universal grade. It earns the next test.

What should a local model comparison hold fixed?

Start with the workload. Use the same prompts, output cap, context size, runtime path, and verifier. Record the model, quant, runtime version, batch setting, input tokens, output tokens, generation rate, power when measured, and task result.

Do not merge cold and resident requests. Do not compare a short chat prompt with a long code file. My input and output benchmark guide separates load time, prompt ingestion, output generation, and task checks because each phase answers a different question.

This workshop did not produce a complete model ranking. It produced one deployable choice for one narrow lane. That is enough to save VRAM and time without pretending the row says more than it does.

When would I still choose the 26B model?

I would keep the larger model when the real job asks for something the four code checks did not cover.

Longer context may change the result. A harder edit may expose planning errors. Structured tool calls may fail even when plain code passes. Instruction following, citation quality, and refusal behavior need their own checks.

The next test should match production work. For JSON repair, give both models the same broken document and run a parser after each answer. For code edits, run the same focused test suite. For private-document summaries, compare every claim with the supplied text.

Pick the smallest model that passes the real gate. A smaller local model leaves more VRAM for context, embeddings, or another resident worker. The task check decides whether that saved memory matters.

What model-selection rule did I keep?

I use four steps.

First, name the job. Second, set a deterministic pass condition. Third, compare speed and resource use only among passing routes. Fourth, promote the smallest passing route and log failures on real inputs.

The RTX 5090 has 32 GB of VRAM, so both models fit this test. Fit did not pick the winner. Four fixed checks did. I can rerun those checks after a model or runtime change.

Fit narrows the list. Runtime support narrows it again. A task check picks the route. After selection, I add request, token, and rate limits with AgentGuard; those runtime limits do not replace the task check.

Accompanying prompt

What the prompt does: compares local model benchmark rows and picks the smallest route that passed the named task.

Copy/paste this prompt:

Role:
You are reviewing local LLM candidates for one production task.

Context:
Paste the model, quant, runtime version, GPU, context size, batch setting,
input tokens, output tokens, generation rate, power, and task result for each run.

Task:
1. Reject every run that failed the task check.
2. Compare speed and resource use only among passing runs.
3. Pick the smallest passing route for this named job.
4. Name the next production-shaped test before the route gets more work.

Output:
- A compact comparison table.
- The chosen route and the exact reason it won.
- Claims the measurements do not support.
- The next test and its pass condition.

Constraints:
- Keep measured values separate from estimates.
- Keep the pass-rate denominator visible.
- Do not turn one workload result into a general model ranking.
- Mark missing values as unknown.

Copy the block above.

Get measured local AI notes by email: https://bmdpat.com/5090-reports

Get the local AI lab notes (benchmark rows, VRAM fit, quant choices, what runs on consumer GPUs), M-F only when there is something worth sending: https://bmdpat.com/newsletter?utm_source=blog_md&utm_medium=aeo&utm_campaign=local-llm-4b-vs-26b-task-quality-2026

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.