Needle 2 Crams a Tool-Calling AI Agent Into 14MB. It Forgets Everything After 256 Tokens.

There's a category of GitHub repo that never gets the "10,000 stars overnight" treatment, because its pitch doesn't fit in a tweet: not "chat with your codebase," not "replace your SRE team," just — a language model small enough to live inside a firmware image. Needle 2, from a small outfit called Cactus Compute, spent this week climbing GitHub's trending page anyway. The whole model — weights included — is a 14MB binary. It runs a full inference session in about 28MB of RAM. And according to the team's own comparisons, it "trades wins" with function-calling models 5 to 70 times its size.

That claim is worth taking apart, because it's simultaneously more impressive and more limited than it sounds. Needle 2 is not a small chatbot. It's a purpose-built tool-calling engine that gave up almost everything a general LLM does well in exchange for fitting on a device that has no cloud connection, no GPU, and sometimes no more than a coin-cell battery. Whether that trade is one you want depends entirely on what you're building — and the README is honest about the limits in a way that's rare enough to be worth calling out.

What Needle 2 actually does

Strip away the marketing and Needle 2 is an agent runtime with the model weights baked directly into the binary. There's no separate .gguf file to download at runtime, no model registry to hit, no network call needed for inference. You pip install cactus-needle, decorate a Python function as a tool, and the 45-million-parameter model decides when and how to call it:

import needle

@needle.tool
def get_weather(city: str):
    "Get current weather for a city."
    return {"city": city, "temp_c": 27, "sky": "clear"}

agent = needle.Needle(tools=[get_weather])
agent.run("what's it like in Lagos right now?")

That's the entire surface area the project cares about: structured tool invocation, JSON-schema-constrained output, and data extraction into Pydantic models. It is explicitly not trying to be a general conversational assistant. The target runtime is a phone, a wearable, a smart-home hub, or a robot — devices where sending every user utterance to an API endpoint costs latency, battery, connectivity, and privacy, and where a 7B-parameter model simply will not fit in memory regardless of quantization.

Beyond the basic tool-call loop, three features do most of the practical work. Confidence gating attaches a calibrated confidence score to each response, so a caller can set a threshold and fall back to a human, a cloud model, or a "didn't understand" state rather than silently acting on a guess — important on a device where a wrong tool call might mean the wrong appliance turns on. Tool retrieval narrows a larger catalog down to the five most relevant tools for a given turn before the model has to choose, which matters because a 256-token context can't hold fifty function signatures at once — it's a cheap form of retrieval-augmented tool selection rather than brute-force context stuffing. And grammar-constrained decoding compiles a Pydantic schema directly into a byte-level grammar the decoder is forced to follow, so "the model returned malformed JSON" stops being a runtime failure mode you have to write a retry loop around.

How it gets to 14MB

The architecture, described in the team's paper on arXiv, is where the real engineering is. Cactus calls it a "Simple Attention Network," and it stacks a handful of choices that each buy a few percentage points of size or speed:

• Hadamard MLP instead of a standard FFN. Replacing the feed-forward block's dense projections with Hadamard-structured transforms cuts parameter count in the part of a transformer that usually dominates it, at the cost of some representational flexibility.
• Grouped Query Attention (GQA). By now a standard efficiency trick — sharing key/value heads across multiple query heads — but essential when every megabyte is a line item.
• Engram key-value memory. This is the more unusual piece: n-gram hash tables that fire at two layers, acting as a cheap associative memory the model can consult instead of encoding every fact in its weights. It's a way of borrowing some of what retrieval-augmented systems do, without an actual vector database attached.
• Multi-lane hyper-connections with Sinkhorn-normalized routing. Instead of a single residual stream, the model routes information across several parallel "lanes," with the routing matrix kept doubly stochastic via Sinkhorn iteration — a trick more commonly seen in optimal-transport and mixture-of-experts literature than in tiny models.
• CQ2-bit quantization ("Cactus Quants"). The default release quantizes down to roughly 2 bits per weight, with a 4-bit option for when you can spare the RAM. This is the single biggest lever: dropping from fp16 to 2-bit is close to an 8x size reduction on its own, before any architectural changes.
• A 256-token sliding window with tools pinned as KV sinks, and byte-level grammar compilation from schemas to force outputs into valid JSON without a separate constrained-decoding library bolted on top.

None of these ideas are individually novel — Hadamard-structured layers, GQA, and 2-bit quantization all have prior art. What's notable is the combination, tuned specifically for a workload (tool calling, not open-ended generation) that tolerates a much smaller effective vocabulary of "things the model needs to express well." A general chat model has to be good at everything from poetry to Python; a tool-calling model has to reliably pick the right function and fill its arguments. That narrower job is what makes 45M parameters plausible in the first place.

The arithmetic is worth spelling out, because it explains why 14MB is the specific number and not, say, 30MB or 5MB. 45 million parameters at 2 bits each is roughly 11.25MB of raw weight data — 45,000,000 × 2 bits ÷ 8 bits/byte ÷ 1,000,000. The remaining ~2.75MB is tokenizer tables, the grammar-compilation logic, Engram's hash tables, and runtime code, all bundled into the same binary. That also explains the RAM figure: 28MB for a full session is roughly double the on-disk size, which is a normal ratio once you account for the KV cache, activation buffers, and the sliding window's working set — and it's a ratio that holds up much better at 45M parameters than it would at 7B, where the same 2x overhead means gigabytes rather than megabytes.

It's also worth being clear-eyed about what 2-bit quantization costs conceptually, even without a published benchmark to cite: at 2 bits per weight you have four representable values per parameter, versus roughly 65,000 at fp16. Cactus's "Cactus Quants" scheme is presumably doing more than naive rounding — grouped/blockwise quantization with per-block scale factors is the standard way to make 2-bit weights usable at all — but any 2-bit model is trading a meaningful amount of representational precision for size, and the effects of that trade tend to show up unevenly: worse on tasks with many similar-looking options, better on tasks with one clearly correct answer. Structured tool selection from a short, curated list is closer to the second category than the first, which is probably why this architecture was aimed at tool calling rather than open-ended text generation in the first place.

What's actually new here

On-device small language models aren't a new idea — Google's Gemma 3 270M, Liquid AI's LFM2 line, and Apple's on-device foundation model behind Apple Intelligence all target the same "runs locally, doesn't phone home" niche. What Needle 2 is testing is how far down the parameter count can go before a model stops being useful for a specific task, rather than how far it can go before it stops being a good generalist.

Cactus's own comparison — again, their numbers, not independently reproduced here — puts Needle 2 in the same performance band as FunctionGemma 270M (a function-calling-tuned Gemma variant), LFM2.5 230M, and Apple's on-device model, while being 5 to 70x smaller in parameter count and running at 2-bit precision against their fp16 baselines. If that holds up under third-party testing, it says something less about Needle 2's cleverness and more about how much of a modern SLM's weight budget is spent on capabilities a tool-calling agent never uses — long-form reasoning, broad world knowledge, multi-turn conversational nuance. Cut those out deliberately and 45M parameters may simply be enough.

The other genuinely new piece is distribution: shipping the weights inside the pip package and the binary, rather than as a separate download fetched on first run. That sounds cosmetic, but it changes the deployment story for embedded and robotics teams who currently have to bundle model files into firmware images, manage checksums, and handle the failure mode of a corrupted or missing weight file on a device with no user-facing error console. A single 14MB binary with the model already inside it is a much smaller ops surface.

It's also useful to place Needle 2 against the runtimes people already use to get small models onto devices, because they're solving an adjacent but different problem. llama.cpp and MLC-LLM are inference engines — they'll happily run a quantized Gemma or LFM2 checkpoint on a phone or a Raspberry Pi, but you still have to source the model, manage the weight file separately from the binary, and the smallest models that run well on them still tend to land in the 200M-plus parameter range if you want general-purpose behavior. Ollama solves the same class of problem for a developer's laptop, not a wearable with a coin-cell battery. None of these projects are trying to answer "how small can a model be while still doing one job reliably" — they're trying to answer "how do I run an existing model locally." Needle 2 is further upstream: it's a purpose-trained model plus a minimal runtime, co-designed together, rather than a general-purpose engine pointed at whatever checkpoint you hand it.

Trying it: what the developer experience actually looks like

The installation story is a single pip install cactus-needle, and the model is already present afterward — there's no second step where you go fetch weights from a hub, which is the detail that most differentiates this from the llama.cpp-plus-GGUF workflow. Tool definitions are ordinary Python functions with a docstring the model reads as the tool's description:

import needle

@needle.tool
def set_thermostat(room: str, temp_f: int):
    "Set the target temperature for a room, in Fahrenheit."
    hardware.set_temp(room, temp_f)
    return {"room": room, "temp_f": temp_f, "status": "ok"}

@needle.tool
def get_room_status(room: str):
    "Get current temperature and occupancy for a room."
    return hardware.read_sensors(room)

agent = needle.Needle(tools=[set_thermostat, get_room_status])
result = agent.run("it's chilly in the office, warm it up a bit")

That's a realistic smart-home example: the model has to resolve "the office" to a room identifier, infer a reasonable target temperature from "a bit," and call the right function — all without a network round trip. For structured extraction rather than action-taking, the same engine accepts a Pydantic model as the target schema and returns a populated instance instead of free text, which is the pattern most production extraction pipelines want regardless of model size.

Customizing the model to a new domain goes through LoRA fine-tuning with adapter merging, run via a browser-based playground rather than a separate training script — you point it at examples, it synthesizes additional training data through an LLM API, trains a low-rank adapter, and merges it back into the base weights. That's a genuinely low-friction path for a team that wants "the same 14MB footprint, but tuned to our specific set of twelve tools" without standing up their own training infrastructure.

Why this matters if you build for constrained devices

For a typical web or backend developer, this project is mostly interesting as a data point. For anyone shipping firmware, embedded Linux, or battery-powered hardware, it's a genuine option in a category that's had almost no good ones:

• No inference cost per call. Once the binary's on the device, tool-calling doesn't hit an API meter. That matters for a wearable making dozens of small decisions a day, where cloud LLM pricing turns "always-on assistant" into a subscription with a variable-cost tail.
• No network dependency for the loop that matters. A smart-home hub that has to reach an LLM API to decide whether "turn off the lights" matched a light-control tool is one Wi-Fi outage away from being useless. Local tool-calling removes that single point of failure from the interaction that has to work every time.
• Latency measured in milliseconds, not round-trips. Tool selection and argument extraction are exactly the kind of small, frequent decisions where a 200ms+ API round trip is disproportionately expensive relative to the work being done.
• Privacy by construction. Nothing about "what did the user just say to their thermostat" leaves the device. For health wearables or anything touching regulated data, that's not a nice-to-have, it's often the only viable architecture.
• Battery. Radio is one of the most power-hungry components on a small device. Cutting a network round trip out of every interaction has a real effect on battery life that a cloud-first team rarely has to think about.

None of this is a reason to run Needle 2 instead of GPT-5.6 or Claude for a task that actually needs broad reasoning. It's a reason to stop routing "was that a request to turn on the porch light" through a data center.

Practical use cases

The README's target list — phones, wearables, smart-home systems, robots — maps to a fairly concrete set of jobs: intent classification and slot-filling for voice assistants that need to work without connectivity; structured extraction from sensor or log data on an edge device before anything gets uploaded; a robot's local decision layer for "which of my 12 known actions does this instruction map to," with the tool-retrieval feature narrowing a larger catalog down to the top five candidates per turn; and lightweight data extraction pipelines where a Pydantic model defines the shape you want and Needle fills it in without a network call. It's a worse fit for anything that needs actual conversation, long-context reasoning, or knowledge the model wasn't fine-tuned on — which is most of what people mean when they say "chatbot."

The limitations the pitch doesn't dwell on

A few things are worth flagging plainly, because the project's own materials mention them but don't emphasize them:

256 tokens is genuinely small. The sliding window keeps recent context and pins tool definitions as KV sinks so they don't fall out of the window, but this is not a model you hand a long document or a multi-turn conversation with real history. It's built for short, transactional exchanges — which is fine for its stated use case, but it's worth being explicit that "sliding window" here is doing a lot of the work to make a very short context feel workable rather than eliminating the limit.

Fine-tuning and data synthesis require an OpenRouter API key. For a project whose entire value proposition is "runs fully offline, no cloud dependency," it's a little ironic that customizing the model to your domain currently routes through a cloud LLM API for synthetic data generation. That's a build-time dependency rather than a runtime one, so it doesn't undermine the deployed product, but teams evaluating this for air-gapped or fully offline development pipelines should know the tooling isn't fully self-contained yet.

2-bit quantization has a real accuracy cost, and the README doesn't show the number. Cactus's own writeup states the performance comparisons but, per the project's documentation, doesn't publish a benchmark table with concrete accuracy figures alongside them — the comparison is described in prose ("trades wins... at 5x to 70x smaller, and 2 bits against their f16") rather than backed by a reproducible table in the README. That's not disqualifying, but it means the "matches models 70x larger" claim currently rests on the vendor's word until someone runs an independent eval.

This is a young, single-vendor project, and the business model is still implicit. Cactus Compute is a small team (the README credits Henry Ndubuaku, Karen Mosoyan, and others), the repo is fresh enough to still be climbing GitHub's trending page rather than sitting on it, and there's no visible funding round or public production-deployment case study to point to. The README's invitation to reach out for "partnerships and production deployments" reads like the standard open-core pattern — free MIT-licensed model and runtime, paid support or custom fine-tuning for companies that want to ship it — but that's an inference, not a stated plan, and it's worth knowing before you build a dependency on a company whose revenue model you're guessing at. MIT licensing does mean you're not contractually locked in if Cactus Compute disappears or pivots: you keep the weights, the code, and the right to fork and maintain it yourself. What you don't keep is the team's accumulated tuning knowledge if they stop working on it, which for a niche architecture like this one is a real form of soft lock-in even under a permissive license.

How it stacks up

The honest read of that table: Needle 2's size advantage is real and architecturally earned, but it's buying that size by being narrower in scope and context than its comparison set, and by being the newest, least-resourced entry in the field.

Where this lands, independently

The engineering is credible — Hadamard MLPs, Sinkhorn-normalized routing, and Engram memory aren't the kind of thing you bolt onto a project for a marketing bullet point, and the paper trail on arXiv suggests this is a genuine research effort rather than a repackaged existing model. The distribution model (weights inside the binary, pip install and go) solves a real pain point for embedded teams that today cobble together model-file bundling by hand.

But the headline comparison — competitive with models 5 to 70x larger — is a vendor claim about a model that's days old on GitHub's trending page, evaluated against undisclosed methodology, from a company with no visible track record. That doesn't mean it's wrong. It means "verify before you bet a product on it" is the correct posture, not "verify before you're mildly interested." The 256-token window and the OpenRouter dependency for fine-tuning are the kind of details that show up in the README but not in the pitch, which is a point in the project's favor on honesty and a point of caution on readiness.

Who should try it, and who should wait

If you're building a voice interface, agent loop, or structured-extraction pipeline for a phone app, wearable, or smart-home device, and your task genuinely fits inside "classify the intent, call the right function, fill the arguments correctly" — Needle 2 is worth a weekend to prototype against. The pip install is trivial, the tool-decorator API is about as low-friction as agent tooling gets, and the failure mode of "it doesn't work well enough" costs you almost nothing to discover.

If you're building anything that needs multi-turn memory beyond a couple of exchanges, general knowledge the model wasn't tuned on, or actual conversational fluency, this isn't your model — not because it's bad, but because it was deliberately built not to be that. And if you're deciding what to put in a shipping product with a support burden and a multi-year lifecycle, wait for either an independent benchmark or a track record longer than "trending on GitHub this week." A 14MB binary is a wonderful thing to prototype with. It's a much bigger commitment to promise a customer.

What's the smallest model you've actually shipped in production, and what broke first — the accuracy, the context window, or the tooling around it?

Sources:

• Needle 2 — GitHub repository
• Needle 2 model weights — Hugging Face
• Needle 2 architecture paper — arXiv
• Cactus Compute
• Introducing Gemma 3 270M — Google Developers Blog
• Liquid AI
• Apple Intelligence — on-device foundation models
• GitHub Trending