Which settings apply where
| Setting | Off-policy (teacher writes) | On-policy (teacher grades) |
|---|---|---|
| Teacher model | Yes | Yes |
| Student model | Yes | Yes |
| Max answer length | Yes, on some services | No |
| Max practice answer length | No | Yes |
| Divergence β | No | Yes |
| Epochs, learning rate | Yes | Yes |
The settings
The model whose behaviour is passed on. Its quality on your questions caps the student's, and its mistakes, length and style are learned too, so evaluate it against the student before distilling. Off-policy, any teacher works because only text crosses over, but managed services allow only their listed teacher → student pairs. On-policy, the teacher must share the student's vocabulary, so it is a larger model of the same family, and it must fit on the GPU beside the student. Thinking teachers, which reason at length before answering, are valuable for hard multi-step problems; for short chat replies a non-thinking teacher usually suits a small student better.
The smaller model being trained, and the one you will deploy, so it sets the running cost and speed. Smaller is cheaper and faster but has less room to absorb the teacher's behaviour, and a student rarely beats its teacher. In Scalata the student is the base model you pick; models that can't be students for the chosen teacher or provider are greyed out with the reason.
Off-policy, on services that expose it: the longest answer, in tokens, the teacher may write per prompt. Longer answers cost more teacher output tokens and more student training tokens, and the student learns that length. Too short, and teacher answers are cut off and the student learns truncated text.
On-policy only: the longest answer the student writes per prompt in each step. Generation is the slowest part of the step, so this is the main lever on GPU time. Answers longer than the limit are cut off, and the student gets feedback only on the part it wrote.
On-policy only: which divergence compares student and teacher at each token. 0 is forward KL: the student tries to cover everything the teacher might say, which gives broader, more varied answers that can blur between options. 1 is reverse KL: the student concentrates on the teacher's most likely answer, sharper and more decisive. 0.5 is the Jensen–Shannon divergence, a symmetric mix, and values in between interpolate. Unlike KL β in reinforcement fine-tuning, this β doesn't anchor the model to its start; it chooses the measure itself.
How to choose
- Candidates: a 7B model that answers your evaluation questions well but costs too much to host, and 0.5B, 1.5B and 3B models of the same family.
- Evaluate the 7B against each. Say it beats the 0.5B on most questions, the 1.5B on many, and mostly ties the 3B.
- The 3B has little to learn; the 0.5B has the most to learn but the least room to absorb it. The 1.5B is the natural first student.
- Same family, so on-policy distillation is possible: practice answer length a little above your typical reply, β = 1, and a GPU that holds both the 7B and the 1.5B.
- The teacher, continuing an answer, splits its belief: “Yes” 50%, “Sure” 40%, anything else 10%.
- Under forward KL (β = 0) the student is pushed to keep both “Yes” and “Sure” likely, and is punished hard for giving either almost no probability.
- Under reverse KL (β = 1) it is punished hard for anything the teacher finds unlikely, and is content to put most of its weight on “Yes” alone.
- Over thousands of tokens the first gives a more varied student and the second a more consistent one. For short, factual replies, consistency is usually what you want.
What drives the cost
- Off-policy: the teacher's inference (input and output tokens for every answer it writes) plus the student's training (prompts plus teacher answers, × epochs). The teacher's answer length isn't known in advance, so estimates are a range. Prompt synthesis by the service grows both parts.
- On-policy: GPU time. Each step generates answers and runs both models, so it is slower than supervised training on the same data, and the GPU must hold both.
Common mistakes
- Choosing the teacher by reputation instead of by an evaluation on your questions.
- Choosing a student too small to hold what the teacher knows.
- Leaving the teacher's max answer length far above your typical reply: you pay for the length and the student learns it.
- Mixing up the two βs. Distillation β picks the divergence; RFT's KL β is a penalty for drifting.
- Not budgeting GPU memory for the teacher in an on-policy run.
References and further reading
External sources open in a new tab.
- 01Distillation TrainerHugging Face TRL docs · huggingface.coDefines β (0 forward KL, 1 reverse KL, 0.5 JSD) and the default completion length.
- 02On-Policy Distillation of Language Models: Learning from Self-Generated MistakesAgarwal et al., 2024 · ICLR / arXiv · arxiv.orgExperiments on how the divergence choice trades quality against diversity.
- 03Kullback–Leibler divergenceWikipedia · en.wikipedia.orgThe definition of KL divergence and why its two directions differ.
- 04Jensen–Shannon divergenceWikipedia · en.wikipedia.orgThe symmetric divergence that β = 0.5 corresponds to.