TL;DR — Speculative decoding accelerates autoregressive inference by using a small draft model to propose token sequences that a larger verifier model accepts or rejects in parallel. The speedup is real and theoretically grounded, but the gains are highly sensitive to draft-target alignment, sequence length, and workload distribution. This essay breaks down the mechanics, the failure modes, and what the technique actually implies for how we should think about inference architecture.
Every token an autoregressive language model generates requires a full forward pass through the entire network. That is the original sin of transformer-based generation: the architecture that made large models so capable also made them serially expensive to run. Batching helps on the throughput axis, but latency — the wall-clock time a user waits for a response — is stubbornly tied to the number of sequential decode steps.
Speculative decoding attacks this constraint without retraining anything. The core idea is deceptively simple: use a cheap model to guess several tokens ahead, then use the expensive model to verify all those guesses in a single parallel forward pass. If the guesses are good, you get multiple tokens for roughly the cost of one. If they are bad, you discard them and fall back to normal decoding. The expensive model's output distribution is preserved exactly — this is not an approximation.
That last point deserves emphasis. Speculative decoding is not quantization, not pruning, not distillation. It is a lossless acceleration scheme. The outputs are statistically identical to what the large model would have produced on its own. That is a remarkable claim, and it is actually true.
How the Verification Step Works
The verification logic is the intellectual core of the technique. When the draft model produces a candidate sequence of k tokens, the target model scores all k+1 positions simultaneously — the original context plus each draft token as a prefix for the next. This is the key: a transformer can process an entire sequence in one forward pass, so verifying k draft tokens costs roughly the same as generating one token normally.
Each draft token is then accepted or rejected via a carefully designed sampling procedure. If the draft model assigned probability q(x) to a token and the target model assigns p(x), the token is accepted with probability min(1, p(x)/q(x)). Rejected tokens are resampled from a corrected distribution derived from the difference between p and q. The result is that the final token sequence is drawn exactly from the target model's distribution, regardless of how many draft tokens were accepted.
The expected number of accepted tokens per verification step — often called the acceptance rate — determines the practical speedup. If the draft model agrees with the target model most of the time, you get close to k tokens per step. If it disagrees frequently, you get barely more than one. The technique lives or dies by this number.
Where the Speedup Actually Comes From
It is worth being precise about the source of the gain, because it is easy to misattribute it. The speedup does not come from the draft model being fast in isolation — a small model generating tokens sequentially is still sequential. The speedup comes from converting sequential decode steps into a parallel verification step on the large model.
Modern accelerators — GPUs and TPUs alike — are dramatically underutilized during single-token decode because the operation is memory-bandwidth-bound rather than compute-bound. The weights must be loaded from HBM for every token, but the actual arithmetic is trivial. Verifying a batch of k draft tokens loads the weights once and does k times the arithmetic, improving arithmetic intensity and making better use of the hardware's compute capacity.
This is why speculative decoding works better on larger models. The larger the target model, the more memory-bound its single-token decode, and the more headroom there is to exploit parallelism. On a small enough model, the verification overhead can actually slow things down.
The Draft Model Problem
Choosing or constructing a good draft model is where the engineering gets hard. The draft model needs to be fast enough that running it k times sequentially is cheap, but aligned enough with the target model that acceptance rates stay high. These two requirements pull in opposite directions.
Several strategies have emerged in the literature and in practice:
Independent small models: Use a separately trained smaller model from the same family. Acceptance rates are reasonable when the models share training data and tokenization, but alignment degrades on distribution shift.
Draft heads on the target model itself: Attach lightweight prediction heads to intermediate layers of the target model. This guarantees architectural alignment but requires modifying or fine-tuning the target.
Self-speculative decoding: Use early exit from the target model itself as the draft. Elegant in theory, but the implementation complexity is non-trivial and the speedup is sensitive to layer selection.
Retrieval-augmented drafting: Look up likely continuations from a datastore rather than running a neural model. Works well on repetitive or templated text; degrades badly on creative or novel outputs.
There is no universally dominant approach. The right choice depends on the target model size, the serving infrastructure, and the distribution of prompts you actually receive.
Production Failure Modes
The academic framing of speculative decoding tends to report acceptance rates on benchmarks with clean, representative prompts. Production workloads are less cooperative.
Acceptance rate is not a fixed property of a model pair — it is a property of a model pair on a specific input distribution. Prompts that are highly constrained (code completion, structured data extraction, templated responses) tend to produce high acceptance rates because the target model's distribution is sharply peaked and the draft model tracks it well. Open-ended generation, multilingual inputs, or prompts that fall outside the draft model's training distribution can collapse acceptance rates toward zero, at which point you are paying the overhead of running two models and getting no benefit.
Latency variance is another underappreciated problem. Average speedup looks good in benchmarks. But the distribution of speedup across requests can be wide. A system that is fast on average but occasionally falls back to slow single-token decode on unpredictable inputs creates a tail-latency problem that is difficult to reason about in SLA terms.
Memory pressure is real too. Running a draft model alongside a large target model on the same device requires fitting both sets of weights in HBM simultaneously. On memory-constrained deployments this can force a smaller batch size on the target model, partially or fully erasing the throughput gains from speculation.
What This Implies for Inference Architecture
Speculative decoding is best understood not as a drop-in optimization but as an architectural commitment. Adopting it well means instrumenting acceptance rates per request type, building routing logic that can fall back gracefully, and treating the draft model as a first-class component of the serving stack rather than an afterthought.
The technique also has an interesting implication for model families. There is now a concrete engineering incentive to train small and large models jointly, or at minimum to fine-tune small models specifically to track the distribution of their larger siblings. The draft model is not just a convenience — it is a load-bearing part of the inference system, and its quality directly determines serving cost.
More broadly, speculative decoding is a reminder that inference optimization is not just a hardware problem or a quantization problem. There is still significant headroom in algorithmic improvements that operate entirely at the systems level, without touching model weights or sacrificing output quality. That is a rarer combination than it sounds, and it is worth taking seriously.
The technique has been in the literature long enough that the basic theory is settled. The open questions are all operational: how to maintain draft-target alignment under fine-tuning, how to route requests intelligently based on predicted acceptance rate, and how to make the failure modes visible enough that engineers can act on them. Those are hard problems, and they are not solved yet.
