How I Test a 30B Local Model Before I Load It

A 30B local model can fit on a product page and still fail the job. I test memory, tool use, speed, and repeatability before I add one to a local agent path.

Short answer: I do not add a new model to an agent path until it loads, calls one fixed tool correctly, handles one tool error, and repeats the result. The current Ollama Nemotron 3 Nano page lists a 30B model with 3.5B active parameters. That makes it a test case, not a production default.

What should I check before I download a 30B model?

I start with the runtime that will serve the request. In my case, that is Ollama on owned hardware. I do not swap in a separate benchmark binary and call the result a production test. A different loader can use different memory rules, quantization, and settings.

The NVIDIA model page lists coding, reasoning, long context, and tool calling as model features. Those claims come from the model publisher. They do not show that my Ollama request can call a tool correctly.

Then I record the exact model tag, runtime version, GPU, VRAM, context size, and generation limit. This record makes the next run comparable. It also stops a common mistake: comparing a cold load on one card with a warm request on another card.

Does active parameter count tell me the VRAM need?

No. Active parameters can help explain compute work during generation. They do not remove the need to load the model weights and runtime state. I use the model's published size as a starting point, then I check the actual memory report from the target runtime.

My first stop is the VRAM calculator. It gives me a memory estimate before I pull a large file. The real gate comes after the download: can the runtime load the model without a memory error or slow system-memory offload?

I test the RTX 5090 and the 8 GB RTX 3070 as separate cases. A result on the card with more memory does not transfer to the smaller card. If the small card cannot hold a useful quantization locally, I record that as a no-go. I do not call a slow offloaded run a win just because it returned text.

What tool test should I run first?

I use a small fixed task with a clear answer. The model receives a local tool schema, selects the tool, sends valid arguments, and returns the expected result. The tool can query a small local data file or call a fake endpoint.

The test rejects malformed JSON, a wrong tool name, invalid arguments, and an answer that ignores the tool result. I keep the prompt and expected result fixed across runs. The row says pass or fail. It does not depend on a demo answer that looks good.

I also test a failure case. The fake endpoint returns a 503 error. I pass the run only when the model states that the request failed and does not invent data. My earlier local security task report records completed tasks, refusals, and timeouts as separate outcomes.

Should I set runtime limits during the test?

Yes. I set a small request limit before I let a new model repeat a tool loop. A bad tool schema can cause retries and long output. AgentGuard can set a budget and rate limit around that loop. The limit does not grade the model. It stops a bad test from using more time and tokens than the test needs.

How do I measure speed without fooling myself?

I record the time to first token, tokens per second, total request time, peak VRAM, and task result in one row. A high token rate is useful, but it cannot hide a long load, a failed tool call, or a repair pass.

I run the same task three times after the first model load. The first request can include download, allocation, or cache work. Warm runs make the serving path easier to compare. I keep the slowest, middle, and fastest times. I do not publish only the best number.

This rule came from an earlier test where generation speed looked better while the full request took longer. Tokens per second and wall-clock time answer different questions. I need both before I change a local default.

When is a new local model ready for an agent path?

I need four receipts. The target runtime must load it. The fixed tool task must pass. The error case must stay honest. The warm runs must stay inside a range I can use.

If one receipt fails, the model stays in the test list. That result does not reject the model. It shows that I cannot add it to this hardware and runtime on 2026-08-14. I put these limits in The 5090 Reports. A failed load or tool test tells me more than a chart with one fast number.

The useful order is simple. First, test the request path. Second, test the task. Third, compare the full time. Only then compare generation speed.

Accompanying prompt

What the prompt does: This prompt creates a repeatable local test plan before a large model enters an agent path.

Copy/paste this prompt:

Role:
You are a local model test reviewer.

Context:
I will give you a model tag, runtime version, GPU and VRAM,
context size, fixed tool task, expected result, and error case.

Task:
1. Make a preflight memory check for the target runtime.
2. Define a valid tool call and expected result.
3. Define one tool error test and the correct model response.
4. Make a table for warm-run time, tokens per second, VRAM, and pass or fail.

Output:
- A test checklist.
- A result table.
- A clear go, no-go, or test-again result.
- The facts that the test cannot show.

Constraints:
- Use the same runtime as the agent path.
- Keep the task and expected result fixed across runs.
- Do not select a model from speed alone.

Copy the block above.

Get measured local AI notes by email: https://bmdpat.com/5090-reports

Get the local AI lab notes (benchmark rows, VRAM fit, quant choices, what runs on consumer GPUs), M-F only when there is something worth sending: https://bmdpat.com/newsletter?utm_source=blog_md&utm_medium=aeo&utm_campaign=testing-nvidia-nemotron-3-5-lightning-on-local-gpus-2026

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.