What is distillation?
Distillation trains a smaller student model to imitate a larger teacher model, so the student inherits most of the teacher's behaviour at a fraction of the inference cost. The supervision signal is the teacher's per-token probability distribution, or its full generations, in place of the original hard labels. The student's architecture, tokenizer and training data are your choice; what makes it distillation is that another model supplies the targets rather than ground truth. The process only queries the teacher; it does not change its weights.
How the student learns from the teacher
Standard supervised training gives the model a one-hot label: token 4217 is correct, every other token is wrong. Distillation replaces that with the teacher's full probability distribution over the vocabulary, which Hinton, Vinyals and Dean called soft targets in the 2015 paper that named the technique. The distribution says token 4217 is probable, but so are three near-synonyms, and the ratios between those tail probabilities carry information about what the teacher actually knows. The student is trained to match that distribution, usually by minimising KL divergence between student and teacher probabilities on the same input.
Two knobs make it work. A temperature above 1 is applied to both sides of the softmax during training, which flattens the distribution and exposes the relative weight of the runner-up tokens; the same temperature is used on the student so the gradients line up. The loss is often a weighted sum of the soft-target KL and the ordinary cross-entropy against the hard label when one is available. At inference the temperature drops back to 1 and the student behaves like any other model of its size.
Response distillation vs logit distillation: what is the difference?
Logit distillation, the original form, needs the teacher's full next-token distribution at every position, which is only possible when you host the teacher yourself or the provider exposes logprobs for the whole vocabulary. DistilBERT (Sanh et al., 2019) is the canonical example: 40% fewer parameters than BERT-base, roughly 60% faster, and 97% of BERT's GLUE score, trained by matching soft targets on the same corpus.
Response distillation, sometimes called sequence-level or behavioural distillation, only needs the teacher's generated completions. You prompt the teacher, log the outputs, and fine-tune the student on those (prompt, completion) pairs with ordinary cross-entropy. This is what OpenAI's distillation product does with stored completions, and it is how most closed-model teachers get used in practice, because you cannot read GPT-4's or Claude's logits. On-policy variants such as TRL's GKD trainer sit between the two: the student samples its own completions and the teacher scores them, which reduces the exposure bias that pure offline distillation suffers from.
Distillation vs fine-tuning vs quantization
All three make inference cheaper, and they attack different parts of the cost. Quantization keeps the same model and stores its weights in fewer bits, so the file shrinks and the arithmetic gets cheaper, but the parameter count is unchanged. Fine-tuning keeps the same model and updates its weights to specialise it on a task, using labels you supply. Distillation changes the model: a different (usually smaller) architecture is trained from scratch or from a checkpoint, using a bigger model as the source of truth.
They compose. A common production pattern is to distil a 70B teacher into a 7B student for a narrow task, fine-tune the student on a small labelled set to sharpen the edges, then quantize the result to 4 bits for serving. Each step targets a different bottleneck: distillation cuts the parameter count, fine-tuning corrects task-specific mistakes, quantization cuts memory and bandwidth. Skipping distillation and only fine-tuning a small base model is often good enough, and it is the cheaper first experiment.
When distillation is the right call
Reach for distillation when you have a narrow task, a bigger model that solves it, and volume that makes the per-call cost of the bigger model painful. Classification, extraction, routing and structured rewriting are the cases where a 1B to 8B student can land within a point or two of a frontier teacher on your own eval, because the task is bounded and the teacher's outputs cover it. You pay the teacher's inference cost once during data generation, then serve the student at its lower cost from then on.
Skip it when the task is open-ended and the teacher's edge comes from broad world knowledge; a student two orders of magnitude smaller cannot store what it does not have room for. Skip it when the teacher's terms of service forbid using its outputs to train competing models, which many hosted APIs do. And be honest about the eval: a distilled student that scores well on the teacher's outputs has learned to imitate the teacher, including the teacher's mistakes, so measure against ground truth wherever you have it.
DistilBERT vs BERT-base on the numbers the paper reports
# From Sanh et al., 2019 (arXiv:1910.01108), Table 2 and Section 3
model params GLUE macro inference (CPU)
---------------------------------------------------------
BERT-base 110M 79.5 1.00x (baseline)
DistilBERT 66M 77.0 1.60x faster
# Trained with a weighted loss:
# L = a * L_ce(student, hard_label)
# + b * L_kl(softmax(student/T), softmax(teacher/T)) * T^2
# + c * L_cos(student_hidden, teacher_hidden)
# T = 2.0 during training, 1.0 at inference.
#
# Same tokenizer, same corpus, half the layers, no next-sentence head.
# The student sees the teacher's soft distribution on every masked token,
# so it learns which wrong answers are close and which are absurd.
40% fewer parameters and 60% faster on CPU for a 3-point GLUE drop. The KL term on temperature-softened logits is what carries the teacher's dark knowledge into the smaller network.
Common questions
Distillation: frequently asked
Is distillation the same as fine-tuning?
No, though response distillation looks like fine-tuning from the outside. Fine-tuning updates a model on labelled examples you supply, keeping the architecture fixed. Distillation trains a different (usually smaller) model to imitate a teacher, and the teacher is the source of the training targets. The two are often combined: distil a big model into a smaller one to get the base behaviour, then fine-tune the smaller one on a labelled set for the task-specific corrections.
Do I need the teacher's logits, or are completions enough?
Completions are enough for the response-distillation case that most closed-model teachers force you into. You prompt the teacher, save the outputs, and fine-tune the student on those pairs with standard cross-entropy. Logit distillation, which matches the full next-token distribution with a KL loss, is stronger per training example but requires access to per-token probabilities over the whole vocabulary, so it is practical mainly for open-weights teachers you host yourself.
Can a distilled student match its teacher?
On the slice the training data covers, often within a small gap; across the teacher's full range, no. DistilBERT keeps 97% of BERT's GLUE score at 60% of the size, and modern task-specific distillations frequently land within a point or two of the teacher on the target eval. Off-distribution prompts are where the gap opens: a 7B student cannot store the breadth a 70B teacher does, so it will regress on tasks that were not represented in the distillation corpus.
Are there licence issues with using a hosted model as the teacher?
Often, yes. Several major providers prohibit using their model outputs to train models that compete with theirs; the exact wording sits in the API terms of service and changes over time. Check the current terms before you build a product around a distilled student, and prefer open-weights teachers (Llama, Qwen, Mistral, Gemma, and similar) when the plan is to ship the student externally.
Sources
Where these facts come from
- Distilling the Knowledge in a Neural Network (Hinton, Vinyals, Dean, 2015): the paper that introduced soft targets, temperature-scaled softmax and the KL-divergence training objective that define knowledge distillation
- DistilBERT, a distilled version of BERT (Sanh et al., 2019): the canonical worked example: 40% smaller, 60% faster, 97% of BERT's GLUE score, with the exact composite loss used to train it
- Hugging Face Transformers: DistilBERT model documentation: reference implementation and training recipe for the DistilBERT student
- TRL: Generalized Knowledge Distillation (GKD) trainer: on-policy distillation trainer where the student samples completions and the teacher scores them, addressing the exposure-bias problem in offline distillation
- OpenAI docs: model distillation with stored completions: the response-distillation workflow for closed-weights teachers: log teacher completions, fine-tune a smaller model on them