Key takeaways
- Distillation trains a small student model on the outputs of a large teacher model, so it runs cheaper and often keeps most of the teacher's skill on the taught tasks.
- DistilBERT is 40% smaller and 60% faster than BERT-base and retains 97% of its GLUE performance, according to its authors.
- DeepSeek reports that distilling R1 into Qwen2.5-32B scored 72.6% on AIME 2024, against 47.0% for the same base trained with reinforcement learning alone.
Distillation in AI is a way to train a small model by having it copy a big one. Instead of learning only from the right answers in a dataset, the small model (the student) learns from the outputs of a larger model (the teacher). The student ends up much cheaper to run and, on the tasks it was taught, often close to the teacher. The idea comes from a 2015 paper by Hinton, Vinyals and Dean, and it now sits behind many of the small, cheap models people use. That is the short answer to what distillation AI is.
That is why the term keeps coming up. On November 9, 2026, Meta launched Muse Ember, its smallest and cheapest Muse model, at $0.10 per million input tokens. Nothing Meta has published says Ember was distilled, so this article does not claim it. But small, cheap models are the reason readers want to know what distillation is, and it is one of the main ways such models get built.
How distillation works
A classifier does not just output an answer. It outputs a probability for every possible answer. Hinton's paper calls these probabilities "soft targets". They carry more information than a single correct label. A model looking at a handwritten 2 might give a small chance to 3 and almost none to 7. That tells the student which wrong answers are nearly right, which a plain label never says.
To make those small probabilities visible, the paper raises the "temperature" of the softmax, the final step that turns raw scores into probabilities. A higher temperature flattens the distribution. The student is trained at that same high temperature to match the teacher, then run at a temperature of 1. The authors found it works best to combine this with the true labels, with the soft part weighted up to compensate for its smaller gradients.
On MNIST, a digit-recognition benchmark, a large regularized network made 67 test errors. A smaller net trained on labels alone made 146. The same small net trained to also match the large net's soft targets at a temperature of 20 made 74. The paper also reports a significant gain for the acoustic model of a heavily used commercial speech system.
DistilBERT: 40% smaller, 60% faster
The technique reached language models with DistilBERT in 2019. Its authors distilled BERT-base (110 million parameters) into a 66 million parameter student. The student starts from the teacher's weights, keeping one layer out of every two, and trains with three losses: one on the teacher's soft probabilities, the usual masked language modeling loss, and one that aligns the two models' hidden states.
The paper says DistilBERT is 40% smaller, 60% faster at inference and retains 97% of BERT's performance on the GLUE benchmark. The macro-score is 77.0 against 79.5 for BERT-base. The gap is uneven. On the RTE task DistilBERT scores 59.9 against 69.3. Distillation gives up some accuracy, and it gives up more on some tasks than others.
Distillation in reasoning models: DeepSeek-R1
Modern distillation often works on text, not probabilities. In the DeepSeek-R1 paper, DeepSeek used R1 to generate about 800,000 training samples and fine-tuned open models from the Qwen and Llama families on them. The students got supervised fine-tuning only, with no reinforcement learning stage.
DeepSeek reports that its 1.5 billion parameter student scored 28.9% on AIME 2024, a competition math test, against 9.3% for GPT-4o and 16.0% for Claude 3.5 Sonnet in the same table. The 32 billion parameter student scored 72.6%.
The paper also compares routes. DeepSeek trained Qwen2.5-32B directly with large-scale reinforcement learning for over 10,000 steps and got 47.0% on AIME 2024. The distilled version of the same size got 72.6%. DeepSeek concludes that distillation is cheaper and works better for smaller models, while pushing past the teacher's ability probably still needs larger models and more reinforcement learning. These are DeepSeek's own results, on its own evaluation setup.
How companies offer distillation today
You can do this without a research team. OpenAI's fine-tuning guide describes it as using a large model's results as training data for a smaller one. The steps: tune a prompt on the large model until it passes your evals, capture its outputs, keep only the responses that meet your criteria, then fine-tune a smaller model on them. The guide says the goal is a small model that performs similarly on one specific task, not on everything. It also advises building evals first and holding out data to compare against the base model.
What distillation can and can't do
The student is usually good at what the teacher showed it and weaker elsewhere. DistilBERT's uneven GLUE scores show the pattern, and OpenAI's guide frames the result as task-specific. A distilled model also inherits whatever the teacher gets wrong, because the teacher's outputs are its only lesson.
For readers, the practical point is cost. A small model that does one job nearly as well as a large one can run for a fraction of the price, which is why model makers and developers both use the technique. When a new cheap model appears, the useful questions are what it was trained from and what it was tested on. Ember's launch post, for one, does not say how it was trained.
Frequently asked questions
What is model distillation?
Model distillation trains a small model to reproduce the behavior of a larger one, using the large model's outputs as the training signal. The small model is cheaper to run.
Is distillation the same as fine-tuning?
Distillation is a use of fine-tuning. OpenAI's guide describes fine-tuning a smaller model on a larger model's filtered outputs. Ordinary fine-tuning can use any data, such as human-written examples.
Does a distilled model perform as well as the original?
Usually not on everything. DistilBERT retains 97% of BERT's GLUE score overall but drops from 69.3 to 59.9 on the RTE task, so the loss varies by task.
Is Meta's Muse Ember a distilled model?
No source says so. Meta describes Ember as its smallest and most cost-efficient Muse model, and its launch material does not mention distillation.
Sources
- Distilling the Knowledge in a Neural NetworkarXiv
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighterarXiv
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement LearningarXiv
- Supervised fine-tuning: distilling from a larger modelOpenAI
- Meta Muse Ember launchClaudeGPT Si


