ResearchExplainer
What is distillation in AI: how big models teach small ones
Distillation trains a small model to copy a large one. Here is how it works, from Hinton's 2015 paper to DistilBERT and DeepSeek-R1, and why it explains today's cheap models.