Preference accuracy

What it is

Some trainers report, next to the DPO loss, a number usually called preference accuracy (Hugging Face TRL logs it as rewards/accuracies). For each pair it asks a yes-or-no question: does the model now rank the preferred answer above the non-preferred one, once each is compared with the reference model? The reported value is the share of pairs where the answer is yes.

How it is computed

Recall DPO's implicit reward: β times the log of how much more likely the trained model finds an answer than the reference does. Each pair has two, one per side, and the margin is the chosen reward minus the rejected one.

margin   = reward(chosen) − reward(rejected)
correct  = margin > 0
accuracy = (pairs with margin > 0) ÷ (all pairs)

At the very first step the trained model equals the reference, so every margin is exactly 0. In the first updates the margins are tiny and nearly random in sign, so accuracy hovers around 50%: a coin flip. As training proceeds the margins spread out, and if the pairs are teaching something, most of them move to the right of zero.

Reading it with the loss

Expect the curve to be jumpy. Accuracy is often logged per batch, and a batch of 8 pairs can only score 0%, 12.5%, 25% and so on, so single readings swing widely. Look at the trend over many steps, or at the per-epoch value, rather than any one point. With very few pairs overall, a handful of pairs flipping sign moves the number a lot without meaning much.

What it does not measure

Preference accuracy is computed on your pairs, whose answers were already written. It never looks at what the model writes now. A 2024 study measured a related number (whether a model gives the better answer the higher likelihood outright, without a reference) and found that most state-of-the-art preference-tuned models do so less than 60% of the time on common preference datasets, partly because DPO struggles to fix even mild ranking errors the starting model already made. A high number on your training pairs shows the run worked mechanically; it is not a measure of answer quality.

If accuracy is also reported on held-back pairs, that number is more informative than the training one, because it shows whether the preference carries over to pairs the model was not trained on. The final word still belongs to an evaluation in which the model writes its own answers.

Common mistakes

• Reading 100% training accuracy as success. With few or easy pairs it is expected, and can mean memorising.
• Reading accuracy without the loss or margin: right by a whisker and right by a mile both count as correct.
• Comparing accuracy across datasets. Harder, subtler pairs naturally give lower numbers.
• Treating it as an evaluation. It never looks at answers the model writes.

References and further reading

External sources open in a new tab.

1. 01DPO Trainer↗Hugging Face TRL docs · huggingface.coDefines rewards/accuracies and rewards/margins as logged during DPO training.
2. 02Direct Preference Optimization: Your Language Model is Secretly a Reward Model↗Rafailov et al., 2023 · arXiv · arxiv.orgWhere the implicit reward behind this metric comes from.
3. 03Preference Learning Algorithms Do Not Learn Preference Rankings↗Chen et al., 2024 · arXiv · arxiv.orgHow often preference-tuned models actually rank the better answer higher, and why DPO rarely fixes existing ranking errors.
4. 04Zephyr: Direct Distillation of LM Alignment↗Tunstall et al., 2023 · arXiv · arxiv.orgPerfect training accuracy after one DPO epoch, and what it did and did not mean for the final model.

Related concepts