Glossary/S/speculative decoding
Model interfaces Also called: assisted generation, assisted decoding, speculative sampling, draft model decoding

What is speculative decoding?

Speculative decoding is an inference technique where a small draft model proposes several next tokens for a large target model, and the target model verifies them all in a single forward pass. Tokens that match what the target would have sampled are accepted; the first mismatch is replaced from the target distribution and the rest are discarded. The output distribution is identical to sampling from the target model alone, so you get the same text with fewer target-model forward passes.

Written by Derek Wonk · last updated September 2026

Ask ChatGPT Ask Claude View as Markdown

How speculative decoding actually works

Normal autoregressive decoding runs the target model once per output token. Each forward pass reads the whole KV cache, produces logits for one position, samples a token, and appends it. Latency is bounded by the number of tokens times the per-token cost of the target, and on a single request the GPU spends most of its time waiting on memory bandwidth for weights it will reuse on the next step.

Speculative decoding turns those wasted cycles into throughput. A cheap draft model, often a smaller checkpoint from the same family, samples k candidate tokens in sequence at low cost. The target model then runs a single forward pass over the current prefix plus those k proposals and returns logits at every position at once, because attention over a fixed context is parallel across positions. A verification step walks the proposals left to right: each token is accepted or rejected against the target's distribution at that position. On the first rejection the algorithm resamples that position from a corrected distribution and throws the rest of the draft away.

The number of tokens produced per target call is between 1 and k+1. If the draft agrees with the target on all k proposals, the verification pass also produces one bonus token from the target's logits at position k+1, and the request advances k+1 tokens for the price of one target forward pass. If the draft is wrong on the first token, the request advances one token, which is the same as plain decoding.

Why the output distribution matches the target model

The key property, established in the Leviathan et al. paper and the parallel DeepMind result by Chen et al., is that the accepted sequence is distributed exactly as if it had been sampled from the target model token by token. The acceptance rule is a rejection-sampling step: a draft token x proposed with probability q(x) is accepted with probability min(1, p(x)/q(x)), where p is the target distribution. On rejection the position is resampled from the residual distribution max(0, p(x) - q(x)), renormalised. Nothing about temperature, top-p or top-k has to change, and no fine-tuning is required to make the draft compatible.

The practical consequence is that speculative decoding is not an approximation. It changes the arithmetic of how many target-model calls a completion needs, not the tokens the user sees. That is why providers and inference servers can turn it on without a quality regression: greedy runs stay identical to greedy runs, and sampled runs draw from the same distribution as before.

When does speculative decoding pay off?

The speed-up depends on two numbers: the acceptance rate of the draft, and the cost ratio between draft and target. If the draft agrees with the target 70% of the time and costs a tenth of the target per token, a request finishes noticeably faster. If the draft agrees only 30% of the time, or the draft is close in size to the target, verification overhead eats the win and plain decoding is faster. Alignment between draft and target matters more than raw draft quality: a small model from the same family and tokenizer usually beats a strong but unrelated model.

Workload shape matters too. Speculative decoding helps most on single-request, memory-bound generation, which is the common case for chat and coding assistants where one user waits on one stream. At high batch sizes the target model is already compute-bound and its forward pass is close to saturated, so the extra proposals do not buy much. vLLM's documentation calls out that speculative decoding can reduce inter-token latency in low-QPS regimes but may not help, and can hurt, once the server is running many concurrent requests.

Variants have grown up around the same core idea. Medusa adds extra decoding heads to the target instead of a separate draft model. EAGLE reuses the target's own hidden states to propose tokens. Prompt lookup decoding skips the draft model entirely and proposes n-grams copied from the input, which works well when the output quotes the input verbatim, as in code edits or retrieval-augmented answers. All of them keep the same accept-or-resample verification loop.

Assisted generation in Hugging Face Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

target_id = "meta-llama/Llama-3.1-70B-Instruct"
draft_id  = "meta-llama/Llama-3.2-1B-Instruct"

tok    = AutoTokenizer.from_pretrained(target_id)
target = AutoModelForCausalLM.from_pretrained(target_id, device_map="auto")
draft  = AutoModelForCausalLM.from_pretrained(draft_id,  device_map="auto")

inputs = tok("Explain KV cache reuse in one paragraph.", return_tensors="pt").to(target.device)

out = target.generate(
    **inputs,
    assistant_model=draft,     # turns on assisted (speculative) decoding
    num_assistant_tokens=5,    # k: tokens the draft proposes per round
    do_sample=True,
    temperature=0.7,
    max_new_tokens=256,
)

print(tok.decode(out[0], skip_special_tokens=True))

Pass a smaller model from the same family as assistant_model and generate() runs the target only to verify batches of five draft tokens at a time. The sampled distribution is unchanged; only the number of target forward passes drops.

Common questions

Speculative decoding: frequently asked

Does speculative decoding change the model's output?

No. The acceptance and resampling rule is constructed so that the sequence returned is distributed identically to plain sampling from the target model at the same temperature and top-p. Greedy decoding with a draft returns the same tokens as greedy decoding without one. That is why inference servers can enable it by default without a quality regression.

How do I choose a draft model?

Pick the smallest model from the same family and tokenizer as the target: Llama 3.2 1B for a Llama 3.1 70B target, Qwen 2.5 0.5B for a larger Qwen, and so on. Sharing the tokenizer is what lets verification align positions cheaply. Acceptance rate on your own prompts matters more than any benchmark, so measure the average tokens accepted per target call on a representative trace before committing.

Does speculative decoding help under heavy load?

Usually less than you would hope. The technique trades draft-model work for saved target-model calls, which pays off when the target is memory-bound waiting on weights, as in a single user's stream. At high concurrency the target is compute-bound because many requests share each weight load, so the marginal saving per request shrinks and the draft's cost is added on top. vLLM notes that speculative decoding can hurt throughput once the server is saturated.

How is speculative decoding different from Medusa or EAGLE?

The verification math is the same. What differs is where the draft tokens come from. Classic speculative decoding runs a separate small model. Medusa attaches extra prediction heads to the target so one forward pass emits several candidate tokens directly. EAGLE trains a lightweight module on the target's own hidden states to propose the next few tokens. Prompt lookup decoding skips learning entirely and copies n-grams from the prompt when they match. All of them keep the accept-or-resample loop that guarantees an unchanged output distribution.

Sources

Where these facts come from

latency streaming sampling model batch inference quantization