Preference tuning (DPO)

What it is

Many things you want from a model are easier to judge than to write down. Which of two replies is clearer, more specific, more on-brand, less likely to invent a feature? A reviewer can answer that in seconds, even when neither reply is perfect. Preference tuning turns those judgements into training data: each record is a pair, one prompt with a preferred and a non-preferred answer.

Direct Preference Optimization (DPO), introduced by Rafailov and colleagues in 2023, is the most widely used way to train on such pairs. The subtitle of their paper sums up the idea: “your language model is secretly a reward model.”

Where it comes from: RLHF

To see what DPO simplifies, start with reinforcement learning from human feedback (RLHF), the recipe behind InstructGPT and the first wave of chat assistants. After pre-training it has three stages:

1. Supervised fine-tuning on demonstrations, to get a model that answers in the right shape.
2. Train a reward model: a separate network that reads a prompt and an answer and outputs a score, fitted so that preferred answers score higher than non-preferred ones.
3. Reinforcement learning, usually PPO: the model writes answers, the reward model scores them, and the model is updated to score higher, with a penalty for drifting too far from the supervised model.

It works: in the InstructGPT paper, people preferred the outputs of a 1.3B-parameter model trained this way to those of the 175B-parameter GPT-3. But it is heavy. The reinforcement stage generates text inside the training loop, keeps several models in memory at once and is sensitive to its settings.

How DPO works

Both RLHF's reward model and DPO rest on the same model of human choice, the Bradley–Terry model. It says that if two answers have hidden quality scores, the chance a person prefers the first is the sigmoid of the difference between the scores:

P(A preferred over B) = σ(r_A − r_B) = 1 / (1 + e^−(r_A − r_B))

r_A − r_B = 0  →  0.50
r_A − r_B = 1  →  0.73
r_A − r_B = 2  →  0.88

Only differences matter: add the same amount to both scores and the probability does not change. A gap of 0 is a coin flip; a gap of 2 means the first answer wins 88% of the time.

RLHF's reinforcement stage maximises reward while paying a penalty, weighted by β, for moving away from a reference model (the penalty is the KL divergence between the two). The DPO authors showed this objective has an exact solution: the best possible model is the reference re-weighted by the exponential of reward ÷ β. Turned around, every model defines an implicit reward: β times the log of how much more likely it finds an answer than the reference does. Put that into Bradley–Terry and the parts that are hard to compute cancel out, leaving a loss that needs only four numbers per pair:

loss = −ln σ( β × [ (log π(chosen)   − log π_ref(chosen))
                  − (log π(rejected) − log π_ref(rejected)) ] )

implicit reward of an answer  =  β × (log π(answer) − log π_ref(answer))

• log π(chosen) and log π(rejected): how likely the model being trained finds each whole answer, as the sum of its token log-probabilities.
• log π_ref(…): the same numbers from the reference model, a frozen copy of the model as it was when training began.
• β: how strongly the model is held to that reference. Lower lets it move further.
• σ: the sigmoid. Its output reads as “the probability that the model's implicit reward agrees with the human preference”. The loss is minus its log, so it is large when the model disagrees and near 0 when it clearly agrees.

Training lowers the loss by raising the chosen answer's likelihood relative to the reference, lowering the rejected one's, or both. The gradient also weights each pair by how wrong the model currently is: pairs already ranked correctly by a wide margin contribute almost nothing, pairs ranked the wrong way contribute most. The paper reports that without this weighting the model can degenerate.

What that means in practice

• Cheaper and more stable than RLHF. No reward model to train and no text generated during training. It runs like a supervised job on pairs, with a frozen copy of the starting model alongside.
• Offline. It learns from a fixed set of pairs. The answers need not come from the model being trained, but pairs close to what it actually writes teach it most.
• Relative, not absolute. It widens the gap between answers. Research has found that the chosen answer's own likelihood often falls during training (“likelihood displacement”), as long as the rejected one falls faster.
• It cannot add knowledge. DPO re-ranks behaviour the model can already produce. If neither answer is something the model could write, there is little to learn, which is why it normally follows supervised fine-tuning.

Data formats

Providers name the fields differently, but every format stores the same three things: the prompt, the better answer and the worse answer. A hosted format:

{
  "input": {"messages": [
    {"role": "system", "content": "You are Scalata's support assistant."},
    {"role": "user", "content": "How do I reset my password?"}
  ]},
  "preferred_output": [
    {"role": "assistant", "content": "Open Settings → Security and choose Reset password. You'll get an email link valid for 30 minutes."}
  ],
  "non_preferred_output": [
    {"role": "assistant", "content": "Please contact support to reset your password."}
  ]
}

Open-source trainers such as Hugging Face TRL use prompt, chosen and rejected, and accept the whole earlier conversation as the prompt:

{"prompt": [
   {"role": "user", "content": "I uploaded a CSV yesterday."},
   {"role": "assistant", "content": "Thanks — it's in Local Storage."},
   {"role": "user", "content": "Can I use it in a dashboard?"}],
 "chosen":   [{"role": "assistant", "content": "Yes: open Dashboards → Add widget → pick the file."}],
 "rejected": [{"role": "assistant", "content": "Maybe."}]}

Some hosted implementations train on one turn only, so earlier turns have to be folded into the last user message as context. Others keep the conversation as separate messages.

Common mistakes

• Running DPO on a plain base model that knows nothing about your product. Both answers in each pair are often equally uninformed, so there is little signal.
• “Confident but wrong”: with few pairs the model copies the style of the preferred answers (specific names, numbered steps, more length) without the knowledge behind them, and invents specifics.
• Pairs whose answers differ only in wording. The model learns noise, or learns that length is what you prefer.
• Treating a falling DPO loss as proof of better answers. It shows the model separates your pairs; only an evaluation shows whether answers improved.
• Too many epochs or too high a learning rate. Preference tuning usually needs 1–2 epochs and a lower learning rate than supervised training.

References and further reading

External sources open in a new tab.

1. 01Direct Preference Optimization: Your Language Model is Secretly a Reward Model↗Rafailov et al., 2023 · arXiv · arxiv.orgThe original paper: the derivation from RLHF, the loss, and experiments against PPO.
2. 02Training language models to follow instructions with human feedback↗Ouyang et al., 2022 · arXiv · arxiv.orgInstructGPT: the three-stage RLHF pipeline that DPO simplifies.
3. 03Deep reinforcement learning from human preferences↗Christiano et al., 2017 · arXiv · arxiv.orgThe paper that introduced learning a reward from human comparisons, first for games and robotics.
4. 04Bradley–Terry model↗Wikipedia · en.wikipedia.orgThe statistical model of paired comparisons behind both reward models and DPO.
5. 05DPO Trainer↗Hugging Face TRL docs · huggingface.coA widely used open-source implementation: data format, loss variants and logged metrics.

Related concepts