Local Agent Bakeoff: Qwen Remains on Top, But Muse Makes a Splashy Debut

Qwen 3.6 has been my daily driver for months. I run it through OpenClaw; my wife runs it through Hermes Agent. Between the two of us, it handles a typical homelab mix: Home Assistant, a shared calendar, a running to-do list, and we're toying with the idea of financial trading. May you live in interesting times.

Here's the problem. Local AI at the ~30B-parameter tier isn't ready for prime time. The agent gets things wrong. It times out. It's flaky enough, often enough, that neither of us fully trusts it to run unattended. Especially with money. That's not a complaint. It's why I started this blog. Nothing about local AI is static, and the models keep advancing.

So when Meta entered the chat, it motivated me to go find a fresh slate of contestants and actually test them. Not against a coding benchmark, but against the job a local "personal assistant" agent would do. Getting there meant rebuilding the testing harness from the ground up. That's its own adventure and I'll detail it later.

For now: Why Qwen won, but I'll daily Muse for a month anyway.

The Contestants

Five models, one RTX 5090, 32GB of VRAM, one llama-swap-managed endpoint swapping between them:

Specs for all five models as configured on the homelab's RTX 5090.

Qwen 3.6 won its spot the hard way, documented over several prior posts. Nemotron Lightning and Muse Glimmer are both brand new — Muse Glimmer shipped six days before this bakeoff ran, Meta's first open model since it pivoted toward closed frontier releases. Neither has independent benchmark verification yet, only vendor numbers. Hermes 4.3 is Nous Research's largest local model and the namesake of the agent framework my wife runs — it earned a fair fight at real scale instead of the smaller Hermes-4-14B that was previously the default. Qwen3.8-27B is the latecomer: Alibaba shipped it a day after the other four had already run, Apache 2.0, dense instead of Qwen's usual MoE line, and recent enough that skipping it felt like the wrong call.

Four of the five share a context window on purpose. Muse Glimmer hard-caps at 131,072 regardless of available VRAM — that's its trained ceiling, not a config choice. Qwen 3.6, Qwen3.8-27B, and Nemotron Lightning could all go higher — Qwen3.8-27B's native ceiling is 262,144, Nemotron Lightning tested clean to 524,288 — but logs showed a max of a few thousand tokens per request. Nowhere near the limit, so we capped every model that could exceed Muse Glimmer's number at Muse Glimmer's number, to keep context length a controlled variable instead of a confound. Hermes 4.3 is the real outlier: its dense 36B weights leave only ~2.5GB of VRAM headroom even at a quarter of the others' context. That's a structural disadvantage on this card, not a benchmarking artifact.

The Test

Five real task areas, scored automatically with Inspect AI. We purposefully deviated from the manual rubric, no eyeballing transcripts for a 1-5 score the way past bakeoffs on this blog have done:

• Home Assistant — 80 device-control samples, the framework's original scope
• Calendar — 16 samples: list, create, update, delete, cancel, find-next
• Portfolio — 15 samples, deliberately read-only: holdings, quotes, performance, drift — no trade-execution tool exists
• To-do — 11 samples across personal and work task lists
• Coding — 13 samples split across two tasks: a portfolio-drift-flagging script and a calendar-conflict-detector, both scored by actually running the generated code

135 individual samples, five models, zero manual scoring. Every tool call gets checked against six independent dimensions — right tool, right arguments, right call count, valid structured output, no hallucinated tools, right response type (act vs. answer vs. ask vs. refuse). Building this out from an existing open-source harness is the whole next post. Let's get to the scores.

The Scores

The first pass through this battery was a single run per model. Then Qwen3.8-27B showed up and needed to be added, and running its numbers next to the original four raised an obvious question: how much would these scores move if we just... ran it again? At non-zero sampling temperature, an LLM doesn't necessarily make the same tool call twice on the same question, so "run once, rank, done" is an assumption worth checking rather than trusting by default.

So all five models went through the full six-domain battery three times each, not once. Every number below is a 3-run mean, not a single sample:

Accuracy by domain and by scoring method, 3-run mean, all five models.

Qwen 3.6 wins on equal-weighted average — but it's essentially a tie. The 0.010 gap to Qwen3.8-27B in second place is smaller than either model's own run-to-run standard deviation (0.020 and 0.019 respectively). Muse Glimmer sits close behind in third at 0.799, also within range of the top two. Three models are bunched at the top; only Nemotron Lightning and Hermes 4.3 are clearly separated from that cluster, and Hermes 4.3 finishes last on both scoring methods, same as before.

A single run would have told a cleaner but less true story. Run 1 alone had Qwen 3.6 winning by what looked like a full point over Muse Glimmer. Run 2 alone had Qwen3.8-27B in first and the incumbent down in third. Neither snapshot was wrong, exactly. Each was a real result from a real run, but neither was stable enough to hang a verdict on by itself. The domain-by-domain breakdown of exactly how much these scores moved between runs, plus a couple of real findings that only showed up once we looked. That's the deep dive in the next post.

Why equal-weighted is the real number

Sample-pooled accuracy — total correct divided by all 135 samples — lets Home Assistant's 80 samples dominate the result. That's 59% of the total sample count deciding most of the ranking, even though Home Assistant is one of five equally important jobs this assistant does. Under sample-pooling, "best overall" mostly just means "best at device control."

Equal-weighted averages the five domain scores, treating Home Assistant, calendar, portfolio, to-do, and coding as five co-equal responsibilities regardless of how many test cases happen to exist for each. That matches how I plan to use the assistant.

Every conclusion in this post uses equal-weighted. Sample-pooled is reported for transparency, and this round it disagrees more than last time: Qwen3.8-27B wins sample-pooled outright, in every one of the three runs, not just on average. That's not noise — it's a real, consistent Home Assistant advantage for the new model, and it's worth its own section near the end.

Speed: MoE vs. Dense

Accuracy isn't the whole picture. We logged wall-clock time and token counts on every single run too, and turning those into tokens-per-second tells a story the scores table doesn't.

Inspect AI doesn't track time-to-first-token through this harness. It logs total round-trip time per request, not the prompt-processing/generation split llama.cpp's native API exposes. What it does track cleanly is output tokens and wall-clock time per sample, which is enough to compute real throughput:

Output tokens per second, by domain and in aggregate, from run 1 (single-run figures; not re-measured across all three rounds).

The split is architectural, not incidental. Qwen 3.6 and Nemotron Lightning are both MoE with roughly 3B active parameters per token. They run 4-5x faster than Muse Glimmer and Hermes 4.3, both dense models where every parameter fires on every token. Speed is the other half of the VRAM story above: the same density that costs Hermes 4.3 its context headroom also costs it throughput, and it's the slowest model in the field by a wide margin. Muse Glimmer pays the same dense-model tax, just from a smaller base — 30B dense instead of 36B — which is why it lands faster than Hermes 4.3 but nowhere near the two MoE models. Qwen3.8-27B is dense too, and its reasoning traces ran long enough on the coding tasks that a single request sometimes took 15-30+ minutes end to end (a real cost this table doesn't fully capture yet, since throughput was only measured on the original run). None of this shows up in the accuracy tables. It matters anyway: a model that's right 80% of the time but takes 5x longer per response is a very different daily-driver proposition than one that's right 75% of the time and answers almost instantly.

What the Dimension Breakdown Reveals

The single most important number in this whole bakeoff isn't in the table above. Across all four original models, on all four tool-calling domains, two of the six scoring dimensions came back at a flat 1.000: format_valid and no_hallucinated_tools. Not one of these models — not even last-place Hermes 4.3 — ever emitted malformed tool-call JSON or invented a tool that doesn't exist.

That reframes the whole result. Every point of separation between these models is judgment, not mechanics. They're all equally competent at the tool-calling plumbing. What differs is whether they pick the right tool, extract the right arguments, call it the right number of times, and correctly decide whether to act, answer, ask, or refuse.

Per-dimension accuracy across the four tool-calling domains — Home Assistant, calendar, portfolio, to-do — from run 1, original four models.

Two specific patterns stood out on close reading of the transcripts.

Qwen's weak spot is argument completeness, not accuracy. Across the Home Assistant transcripts, Qwen repeatedly picked the right tool and the right entity name, then dropped the optional domain hint the test expected:

Expected: HassTurnOn({"name": "Kitchen Thermostat", "domain": ["climate"]})
Actual:   HassTurnOn({"name": "Kitchen Thermostat"})

That's not a wrong answer in practice. Home Assistant can resolve the entity from the name alone. But it's an incomplete one by the test's schema, and it happened often enough to be Qwen's single lowest dimension.

Hermes 4.3 under-calls on simple state queries and over-promises on things it can't do. It answered "what's the kitchen thermostat set to," "what's the status of the front door lock," and "what's today's date" with zero tool calls at all. It's just plain text, no lookup, on requests with an unambiguous matching tool. On the flip side, asked to "order more paper towels from amazon," it replied: "Sure, I can help with that. I'll place an order for more paper towels from Amazon for you." No such capability exists. That's a worse failure mode than a refusal. It's a fabricated commitment. The same pattern showed up on a calendar case: asked to add an ambiguous reminder with no time given, Hermes 4.3 invented a start time, an end time, a description, and a location, none of which the user provided, instead of asking a clarifying question.

The Portfolio Refusal Test — a Real Nuance, Not a Clean Story

Three test cases asked each model to buy stock, sell stock, or rebalance an account. No trade-execution tool exists in this harness by design. I don't trust a local model to place real trades unattended no matter how many YouTube influencers do. I'm including it to track because one day we will. Anyway, for this test all three cases expect a plain-text refusal, zero tool calls.

Qwen 3.6 and Hermes 4.3 refused cleanly on all three. Nemotron Lightning and Muse Glimmer did not:

Refusal outcome per model across the three trade-execution test cases, original four models.

Reading the actual reasoning traces changes the story, though. Asked to sell Apple stock, Nemotron Lightning's own chain of thought read: "I can only report information, cannot execute trades... I need to explain that I cannot place trades, but I can compute drift and show current allocation vs target." It then called PortfolioGetHoldings — a read-only lookup — clearly intending to follow up with exactly the refusal the test expected. Muse Glimmer's trace on the rebalance case was nearly identical, ending in a call to PortfolioComputeDrift instead.

Neither model attempted anything resembling a trade. No such tool exists for them to call. What actually happened is a single-turn capture limitation in this harness: it records the tool call a model makes and stops there, so it never sees the natural-language refusal that comes next in a real conversation. That's a fair criticism of the test, not evidence these two models will place unauthorized trades. It also raises a real design question: gathering current account data before explaining a limitation isn't reckless behavior — it's what a careful advisor does before answering. We'll have to adjust this part of the test harness.

Two Coding Tasks, Mostly Zero Differentiation

Every model scored a perfect 1.000 on both coding tasks in the first two runs — a portfolio-drift-flagging script and a harder calendar-conflict detector with real edge cases (back-to-back events that must not flag, fully nested events that must, a three-event chain designed to catch a model that incorrectly treats overlap as transitive). The apparent ceiling cracked on the third run: Nemotron Lightning and Hermes 4.3 each dropped to 0.875 on the conflict detector after two straight perfect scores, while Qwen 3.6, Muse Glimmer, and Qwen3.8-27B stayed perfect across all three runs. That's a thin but real signal — 2 of 5 models have a non-zero failure rate on this task — that only a third independent run surfaced.

Combined with the perfect format_valid/no_hallucinated_tools scores above, this mostly closes the loop anyway: at this parameter tier, basic-to-intermediate coding and tool-call mechanics are close to uniformly solved. These models don't differentiate much on raw capability. They differentiate mostly on judgment under ambiguity. That's exactly the dimension the per-domain scores above measure.

The Verdict

Qwen 3.6 holds — barely. Four months as the daily driver, and it's still the best all-around choice on the equal-weighted number that actually matches how this assistant gets used, three runs averaged instead of one. But the margin over 2nd place is 0.010, smaller than the model's own run-to-run noise. Call it what it is: a statistical tie at the top, not a clean win. Its weakest dimension — argument completeness — is a minor, fixable pattern, not a reliability problem.

Qwen3.8-27B is a legitimate co-leader, not just a fast follower. It showed up a day after the other four had already run, went straight into the same battery, and landed close enough to the incumbent on equal-weighted that three runs of averaging still couldn't cleanly separate them. On sample-pooled — where Home Assistant's 80 samples carry the most weight — it wins outright, every run.

Muse Glimmer is still the real story of this bakeoff. A model six days old, with zero independent benchmarks going in, lands in a tight cluster with both Qwens at the top of equal-weighted (0.799, within noise of 1st and 2nd) and comes within a hair of Qwen3.8-27B on sample-pooled too. That's a serious debut for Meta's first genuinely open local release in a long time.

Nemotron Lightning lands off the top cluster. A solid, unremarkable, no standout weakness but no standout strength either. Now clearly separated from the three-way tie above it once averaged across runs.

Hermes 4.3 finishes last, on both scoring methods, across every run. Last on the scoreboard, and last on VRAM headroom (~2.5GB free at a quarter of the other models' context window). Both point the same direction: it's the weakest fit for this card and this job. It is, however, one of the two most consistent models tested. More on that in the next post.

What's Next

Muse Glimmer earns the win, for me. Qwen 3.6 wins on paper, by a margin too small to trust on its own. I'm running Muse Glimmer anyway. Here's why. A model this new, this balanced, this fast out of the gate deserves soak time. Is it really that good? And more importantly, it scored well on Home Assistant tasks. And that's my biggest takeaway. That's all I really trust these agents to do at this stage. Replace "Hey, Siri" and "OK Google" in my smarthome.

Here's the one number that almost changed that decision. For Home Assistant specifically — the actual job, not the aggregate — Muse Glimmer averages 0.705 across three runs, and Qwen3.8-27B averages 0.708. That's a dead heat, well inside each other's run-to-run noise, so it didn't move my pick. But Qwen3.8-27B was technically the stronger number, it wins sample-pooled outright, and it's the newest thing on this list. I'm going with Muse Glimmer anyway, because tinkerers gonna tinker and a six-day-old model from Meta's first genuinely open release in years is the more interesting thing to live with for a month. I'll be tracking Qwen3.8-27B closely in the background, though. If this soak test doesn't hold up, it's the obvious next thing to try.

None of this would exist without extending an open-source testing harness to cover calendar, portfolio, and coding domains it never supported before. That extension surfaced a real bug in the harness itself, plus the full run-to-run variance study behind the numbers above. That's the next post.

By the Numbers

• 5 models tested, 135 individual samples per run, 3 full runs per model, 0 manually scored
• 1.000 — the score on format_valid and no_hallucinated_tools, for every one of the original four models
• 0.814 vs. 0.746 — Qwen 3.6's equal-weighted score vs. last-place Hermes 4.3, both 3-run means
• 0.010 — the equal-weighted gap between 1st-place Qwen 3.6 and 2nd-place Qwen3.8-27B, smaller than either model's own run-to-run standard deviation
• 6 days old — Muse Glimmer's age at test time
• 1 day — how long after the original four ran that Qwen3.8-27B shipped and got added
• 2 of 5 — models that refused all 3 trade-execution requests cleanly (Qwen 3.6 and Hermes 4.3); the other 3 called a read-only lookup instead of a pure refusal on at least one case
• 13 coding samples, 5 models, cracked to 0.875 for 2 of 5 models only on the third run
• ~2.5GB — Hermes 4.3's VRAM headroom at its (already-reduced) context window
• 131,072 — the token context window shared by 4 of the 5 models, by design
• 0.705 vs. 0.708 — Muse Glimmer vs. Qwen3.8-27B on Home Assistant alone, a dead heat that didn't change which model I'm actually running