A team at Percepta built a transformer whose weights were calculated rather than trained, and encoded a complete WebAssembly virtual machine into them. The result runs arbitrary C programs inside the model itself, at roughly 30,000 tokens per second. No training data, no gradient descent, no learning of any kind: the computation was compiled directly into the parameters.
Key facts
- The anchor number: about 30,000 tokens per second, running arbitrary compiled programs.
- The method: compile C to WebAssembly, solve a mixed-integer-programming schedule, construct the weights analytically, then execute with a purpose-built C++ engine.
- What it is: a standard softmax-ReGLU transformer, not a custom architecture.
- Primary sources: the Percepta write-up "Constructing an LLM-Computer" and the transformer-vm repository.
Almost everything anyone knows about what transformers can do comes from watching trained ones behave. That is an empirical approach with a permanent blind spot: when a model succeeds, you cannot tell whether the architecture made the capability possible or whether the training data happened to contain it. This project attacks the question from the other side, by asking what can be built into the weights deliberately.
The pipeline runs in one direction, from source code down to parameters. A C program is compiled to WebAssembly, a compact instruction format designed to run untrusted code safely. A scheduling problem is then solved using mixed-integer programming, which lays out how the computation must be arranged across the model's layers and attention heads. From that schedule the weights are computed directly. Finally, a C++ inference engine executes the resulting model. The repository's own description is precise: "a standard softmax-ReGLU transformer whose weights are computed analytically" that simulates a WebAssembly virtual machine on arbitrary programs.
The mechanism inside is the elegant part. A virtual machine needs to track state: a stack, a region of memory, local variables, a cursor marking the current instruction, and a call depth for nested function calls. A transformer has no variables and no memory in the ordinary programming sense. It has attention, which lets each position look up information from other positions, and it has arithmetic over sequences. The project maintains all of that interpreter state through attention lookups and cumulative sums, with conditional branches and control-flow instructions handled inside the interpreter layer.
The analogy that fits is a music box. A music box has no processor, only a cylinder with carefully placed pins that strike tuned teeth in a fixed order. Nothing in it is programmable in the usual sense, yet the arrangement of the pins encodes a complete piece of music. What Percepta built is a compiler that turns programs into pin placements, where the pins are attention weights.
The author, Christos Tzamos, described the required stack in the release announcement: an abstract computation model for transformers, a programming language targeting that model, a WebAssembly interpreter written in that language, and an integer-programming-based compiler that lowers the whole thing into weights. The blog post's subtitle is the cleanest statement of the goal: "From programs into transformer weights."
Why this matters is a question worth answering carefully, because it is easy to overclaim. The honest reading is that this is a constructive existence proof about representational capacity. When people argue about whether a transformer could in principle implement some algorithm, the answer for a large class of algorithms is now demonstrably yes, with a working artifact and a compiler rather than a proof sketch. That is genuinely useful grounding for the interpretability research program, which spends much of its effort trying to identify algorithms hidden inside trained weights. Knowing what a hand-built implementation of a known algorithm looks like gives that search something to compare against.
What it does not show is anything about learning. Gradient descent does not search the space of hand-compiled programs, and there is no reason to expect a trained model to arrange itself the way a compiler would. The gap between "the architecture can represent this" and "training will find this" is where nearly all the hard questions live, and this work does not close it.
A large amount of nonsense attached itself to this project on the way around the internet. The most persistent version claimed a Doom renderer running inside a 21-billion-parameter transformer at a specific frame rate. None of it appears in any primary source. There is no Doom demo, no parameter count of that kind, and no frame-rate figure in the repository, the blog post or the release announcement. The documented examples are ordinary test programs, things like Collatz sequences, Fibonacci and Sudoku solvers. The original discussion was also modest: the Hacker News front-page snapshot for the post shows nine points, so it was noticed rather than viral. A Tildes thread covers it accurately, repeating the verified claim that the model executes arbitrary C code for millions of steps while streaming at more than 30,000 tokens per second.
The honest caveat is practical. Nobody should run programs this way. A conventional processor executes the same work many orders of magnitude faster and more cheaply, and the value here is entirely conceptual. What you get is a clean answer to a question that used to be argued about with intuitions.
Background: transformers, the engine inside almost every modern AI, and looped transformers, which explores a related question about reusing layers to compute deeper.
Originally published on Ground Truth, where every claim is checked against the primary source.
