Key takeaways

  • Speculative decoding has a small draft model guess several tokens that the large target model then checks in a single pass.
  • Its accept-or-reject rule keeps the output distribution the same as the target model's, so speed rises without a change in quality.
  • Google reported 2x to 3.4x speedups on T5-XXL and DeepMind about 2x to 2.5x on Chinchilla 70B, and this week's MMFlash paper reports up to 3.58x for vision-language models.

Speculative decoding is a way to make a large language model write two to three times faster without changing what it writes. A small, fast "draft" model guesses the next few tokens, and the big model checks all of the guesses in a single pass, keeping the ones it agrees with. The idea comes from two papers written independently: Leviathan, Kalman and Matias at Google Research, posted in November 2022, and Chen and colleagues at DeepMind, posted in February 2023. It is now built into inference engines such as vLLM and llama.cpp, and researchers are still extending it.

Why LLMs write slowly

A language model writes one token, a word or piece of a word, at a time. Each token takes a full pass through the network, and the next pass can't start until the last one ends. A 500-token answer is 500 passes in a row.

Each pass is slow because the chip spends its time reading billions of weights out of memory, not doing arithmetic. The DeepMind paper notes that because decoding is bound by memory bandwidth, scoring a few tokens in one pass takes about as long as producing one. Speculative decoding uses that spare capacity. For the bigger picture, see our explainer on AI inference.

How speculative decoding works

There are two models: the target, the big model whose answers you want, and the draft, a much smaller one that is fast but less accurate.

  1. The draft writes the next few tokens on its own. The number of guesses is a setting, called γ in the Google paper and K in the DeepMind one.
  2. The target reads all the guesses in one pass, which gives its own probability for every guessed position at once.
  3. Guesses are checked left to right. If the target gives a token at least as much probability as the draft did, it is kept. If not, it is kept with a chance equal to the target's probability divided by the draft's.
  4. At the first rejection, the rest of the draft is thrown away and the target picks a replacement from an adjusted distribution that corrects for the draft's mistake.
  5. If every guess survives, the target adds one more token for free.

Each round produces at least one token, as normal decoding does, and up to γ + 1 when the draft guesses well.

Why the output stays the same

The accept-or-reject rule is a form of rejection sampling. Both papers prove that text produced this way follows the same probability distribution as text from the target model alone. The draft only decides how fast the answer arrives, never what it is.

In practice, the DeepMind authors note that a single token changed by floating-point rounding can send two runs in different directions. vLLM's documentation calls its implementation "theoretically lossless up to the precision limits of hardware numerics."

How much faster it is

Paper Target model Draft model Task Speedup
Leviathan et al., Google T5-XXL (11B) T5-small (77M) Translation 3.4x greedy, 2.6x sampling
Leviathan et al., Google T5-XXL (11B) T5-small (77M) Summarization 3.1x greedy, 2.3x sampling
Chen et al., DeepMind Chinchilla (70B) 4B, 8 layers XSum summarization 2.01x greedy, 1.92x nucleus
Chen et al., DeepMind Chinchilla (70B) 4B, 8 layers HumanEval coding 2.46x nucleus

Google measured on one TPU v4, one request at a time. DeepMind served Chinchilla on 16 TPU v4 chips with K = 4: time per token fell from 14.1 to 5.73 milliseconds on HumanEval, and the score stayed close (47.0% against 45.1%). The DeepMind paper says the two teams worked "concurrently and independently."

What decides the speedup

The key number is the acceptance rate, α: how often the target agrees with the draft. The Google paper gives the expected tokens per round as (1 − α^(γ+1)) / (1 − α). With α at 0.8 and γ at 5, that is about 3.7 tokens per pass of the big model.

  • Predictable text gains more. DeepMind's biggest gain was on code.
  • Longer drafts aren't always better. Each extra guess costs time and is less likely to survive. DeepMind's latency was lowest at K = 3 on one task.
  • It trades compute for time. Google's paper says latency improves "at the cost of an increased number of arithmetic operations." On a server already busy with many requests, there is less to spare. vLLM aims it at "medium-to-low QPS (queries per second), memory-bound workloads."

Which draft model is best

There is no single best one. A draft helps only if it predicts its own target well, so released drafts are built for one specific model. The engines support several kinds:

  • A separate small model, the original approach, which llama.cpp's docs call the most widely used.
  • n-gram lookup, which needs no model and reuses text already in the context. llama.cpp suggests it for repetitive output such as rewriting code.
  • EAGLE-style drafters, which read the target model's internal states.
  • Multi-token prediction (MTP), extra heads in the main model that predict several tokens ahead. So "speculative decoding vs MTP" is a false choice: MTP is a way to make the draft, and the target still checks it.
  • Block drafters such as DFlash, which propose a whole block of tokens in one pass.

Apple has its own, the Recurrent Drafter (ReDrafter), a small recurrent network fed the big model's hidden states. Apple reports up to 2.8x faster inference on Nvidia H100 GPUs and up to 2.3x on Apple silicon with its MLX framework.

This week's research: MMFlash for vision-language models

Drafts struggle when a model is looking at an image. MMFlash, posted to arXiv on November 13, 2026 by researchers at the National University of Singapore, measures this. For tokens that depend least on the image, drafts were accepted 46% of the time. For tokens that depend most, 22%.

MMFlash builds on the DFlash block drafter and trains it to focus on the image-dependent tokens it still gets wrong. For long videos, sparse attention cuts the draft's context from over 40,000 tokens to about 1,600.

The authors report lossless speedups of up to 3.58x, about 3x on average on image benchmarks with Qwen3.5 4B and 9B on one Nvidia H200, and 2.33x to 2.71x on video. They say it beats EAGLE-3 by up to 195.4% and DFlash by up to 17.1%, and that the lead narrows on a busy server. These numbers are the authors' own. The code is MIT-licensed, with drafts for Qwen3.5, Qwen3.8-27B and Gemma 4 E4B that the README says run on stock SGLang and vLLM.

How to turn it on in vLLM and llama.cpp

In vLLM, pass --speculative-config (or speculative_config in Python). Its method field picks the drafter, such as draft_model, ngram, mtp, eagle3 or dflash, and num_speculative_tokens sets how many tokens it guesses per step.

In llama.cpp, llama-server takes a draft model with -md, the method with --spec-type, and the longest draft with --spec-draft-n-max, default 3. The server prints acceptance statistics, the number to watch. vLLM's docs say gains depend on the model, traffic, hardware and sampling settings, and recommend measuring on your own setup.

Frequently asked questions

Is speculative decoding lossless?

In theory, yes. Both original papers show that the accept-or-reject rule gives the same output distribution as the target model alone. In practice vLLM describes it as lossless up to the precision limits of hardware numerics, so rounding can cause small differences.

What is the difference between speculative decoding and MTP?

Multi-token prediction (MTP) is one way to make the draft: the main model has extra heads that predict several tokens ahead. Speculative decoding is the overall draft-and-verify loop, and vLLM and llama.cpp both list MTP as one of its methods.

Does llama.cpp support speculative decoding?

Yes. llama-server takes a draft model with -md or --model-draft, picks the method with --spec-type, and sets the longest draft with --spec-draft-n-max, which defaults to 3. It also supports n-gram drafting that needs no extra model.

What is the speculative decoding formula?

Leviathan et al. give the expected tokens per round as (1 - α^(γ+1)) / (1 - α), where α is the acceptance rate and γ the number of drafted tokens. With α at 0.8 and γ at 5, that is about 3.7 tokens per pass of the large model.

Sources

  1. Fast Inference from Transformers via Speculative DecodingarXiv (Leviathan, Kalman, Matias, Google Research)
  2. Accelerating Large Language Model Decoding with Speculative SamplingarXiv (Chen et al., DeepMind)
  3. MMFlash: Faster Speculative Decoding for Vision-Language ModelsarXiv (Feng et al., National University of Singapore)
  4. MMFlash code and modelsGitHub
  5. Speculative decodingvLLM
  6. Speculative decodingllama.cpp
  7. Recurrent Drafter for Fast Speculative Decoding in Large Language ModelsApple Machine Learning Research

Add ClaudeGPT Si as a preferred source on Google