Coming soonModel Training is launching soon in Scalata Enterprise. Teach a model to answer the way your team does.See what's coming

Resources/Model training/Training methods

Method

Off-policy distillation: the teacher writes answers

In off-policy distillation the teacher answers your prompts and the student is fine-tuned on those answers like ordinary supervised training. Teacher and student can come from any model families.

Chapter 73 min read4 references

How it works

  1. Prompts. The user side of your chats. One conversation with three of your replies becomes three prompt rows, each holding the conversation up to that reply, so the teacher answers the way the conversation actually went.
  2. Teacher answers. The teacher writes a full answer to every prompt. Your own replies can be included as golden examples, so the teacher sees how you answer. Some services also generate similar prompts to widen coverage (data synthesis).
  3. Student training. The student is fine-tuned on prompt → teacher answer, exactly like supervised fine-tuning, with the loss on the answer tokens only.
Way 1 · teacher writes answers
your prompt→teacher (large)→full answer text→student learns prompt → answer

Any model families. Only text crosses over. Trains exactly like supervised fine-tuning.

Way 2 · teacher grades the student's own answer
your prompt→student writes→teacher scores every token→student moves toward teacher

Same tokenizer required, so the teacher is usually a bigger model of the same family. Learns from its own mistakes.

Off-policy distillation passes finished text from teacher to student. On-policy distillation lets the student write, then shows it, token by token, what the teacher would have said instead.
One row sent to the service: the prompt only, ending with the user's message
{"messages": [
  {"role": "system",    "content": "You are Scalata's support assistant."},
  {"role": "user",      "content": "How do I share a report with a client?"},
  {"role": "assistant", "content": "Open the report → Share → pick the client."},
  {"role": "user",      "content": "Does the client need an account?"}
]}

In the research literature this is sequence-level knowledge distillation (Kim & Rush, 2016): instead of matching the teacher's probabilities token by token, the student learns from the single sequence the teacher actually produced. It gives up the soft-target detail, but needs nothing from the teacher except text. That is why managed services use it: the teacher can be a hosted model that is only reachable through an API.

Why “off-policy”

Policy is reinforcement learning's word for a model's way of choosing tokens. Here the training answers come from a different policy, the teacher's, so the student learns off-policy. That has a known cost, called exposure bias or train–inference mismatch. In training, the student always continues from the teacher's flawless prefix. In real use it continues from its own earlier tokens, mistakes included, and it has never practised recovering from them. A small early slip can lead it somewhere the teacher's answers never went.

On-policy distillation exists to fix exactly this. Off-policy remains the practical choice whenever teacher and student come from different model families, or the teacher is only available as text.

Worked example

What one conversation becomes
  1. A chat: the user asks how to share a report, you reply; they ask whether the client needs an account, you reply; they ask when the link expires, you reply.
  2. Three prompt rows: (1) the first question; (2) the first question, your first reply and the second question; (3) everything up to the third question.
  3. The teacher writes three answers. The earlier replies inside rows 2 and 3 are yours, kept as context, so the teacher continues your conversation rather than its own.
  4. The student trains on three examples: each prompt row → the teacher's answer to it.
  5. Cost: the teacher's inference for three answers (input and output tokens), plus the student's training tokens (prompts plus teacher answers, × epochs).

Getting the most from it

  • Pick the teacher by evaluation. Compare teacher and student on your held-out questions first. If the student is already close, there is little to gain.
  • Volume matters. Each teacher answer is one example, and a small model needs many to change its habits. A few dozen prompts rarely move it; plan for hundreds or more, or use the service's prompt synthesis.
  • Mind the teacher's length and style. The student learns them. A teacher that writes 300 words where your team writes 50 produces a student that does the same. Golden examples help; a supervised pass on your replies afterwards helps more.
  • Thinking teachers write long reasoning before the answer. That is valuable for hard multi-step problems, and costly noise for short support replies.

Common mistakes

  • Too few prompts for the student to change its habits.
  • Forgetting that every teacher error becomes a training example.
  • Expecting your style without golden examples or a later supervised pass.
  • Comparing the distilled student only with the teacher. Compare it with the plain student too: that shows what distillation added.

References and further reading

External sources open in a new tab.

  1. 01Sequence-Level Knowledge DistillationKim & Rush, 2016 · arXiv · arxiv.orgThe method: train the student on the teacher's generated sequences.
  2. 02Distilling the Knowledge in a Neural NetworkHinton, Vinyals & Dean, 2015 · arXiv · arxiv.orgThe original soft-target distillation that the sequence-level version simplifies.
  3. 03DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement LearningDeepSeek-AI, 2025 · arXiv · arxiv.orgA large-scale example: 800,000 R1 samples distilled into small Qwen and Llama models.
  4. 04On-Policy Distillation of Language Models: Learning from Self-Generated MistakesAgarwal et al., 2024 · ICLR / arXiv · arxiv.orgExplains the train–inference mismatch that off-policy distillation suffers from.
MethodDistillationDistillation trains a small, cheap student model to behave like a large teacher model on your tasks, either by learning from the teacher's answers or from the teacher's judgement of its own.MethodOn-policy distillation: the teacher grades the studentIn on-policy distillation the student writes its own answers and the teacher scores every token of them, so the student learns exactly where it goes wrong. Teacher and student must share a vocabulary.SettingDistillation settingsTeacher, student, the two answer-length limits and the divergence β: what each distillation setting does, which route it applies to, and how to choose it.MethodSupervised fine-tuning (SFT)Supervised fine-tuning teaches a model by example: it is shown a conversation and the reply you want, and its weights are nudged so that reply becomes more likely, token by token.ConceptWhy tokens drive costHow token counts turn into the price of training and of using a model, with worked numbers, and which levers actually lower the bill.MethodContinuing from a trained modelA training run can start from the weights of one of your earlier runs instead of the base model, so the model keeps what it learned and adds what the new data teaches. This is how multi-step recipes such as supervised then preference tuning are built.

Turn weeks of expert work into minutes

See how Scalata fits your team, your data, and your controls.

Contact Sales