What chat feedback is
Every answer in the chat can be rated: 👍, 👎 — with up to two reasons such as “Missing details” and an optional comment — or Refresh, to ask for a new answer. Each person has one rating per answer, and their latest one counts. The exception is a Refresh after a 👎: the 👎 and its reasons are kept, because it is still a “no”.
These ratings are a record of what real users thought of real answers, which makes them valuable. But a thumb on its own is a weak, noisy label, and most fine-tuning methods can't use it directly. The job of the import is to turn ratings into the kinds of examples that the methods every provider supports can learn from.
From ratings to examples
| What people did | Becomes | Method |
|---|---|---|
| 👍 an answer | The answer as a reply to learn from, with the conversation before it as context | Supervised fine-tuning |
| Asked the same question twice in one chat; liked one answer, disliked or refreshed the other | A pair: the liked answer is chosen, the other rejected | Preference tuning (DPO) |
| 👎 an answer | A review-queue item. A reviewer writes the right answer; exporting the queue gives a chosen/rejected pair plus a reply to learn from | DPO + supervised |
Rules that keep the data honest
- A disliked answer is never a reply to learn from. It only appears as the rejected side of a pair, or as earlier context, which is never trained on.
- Disagreement is left out. An answer one person liked and another disliked counts as mixed and is skipped.
- Identical answers make no pair. If a Refresh returned the same text, “prefer A over A” teaches nothing.
- A Refresh alone is a weak signal. People refresh out of curiosity too. It counts as a bad answer only if you choose to count it; for pairs, “asked again, then liked the new one” is always used.
- Imports are stable. Importing again updates the examples made last time instead of duplicating them.
- Over a month, staff gave 420 👍 and 95 👎, and in 60 chats asked a question again and liked the second answer.
- 12 of the liked answers were disliked by a colleague: mixed, skipped. 408 liked answers become replies to learn from.
- Of the 60 asked-again cases, 4 returned the same text: no pair. 56 become chosen/rejected pairs.
- The 95 👎 answers go to the review queue. Reviewers correct 70 and discard 25 that were fine after all. Export adds 70 pairs and 70 replies.
- Result: 478 replies for supervised training and 126 pairs for DPO, none of them built on a disliked answer.
Why not train on every thumb directly?
Bad answers don't fit most methods. Supervised training needs a good answer to imitate, and DPO needs a better and a worse answer to the same question. A lone 👎 is neither. There is a method made for single good/bad labels, KTO (Kahneman-Tversky Optimization, 2024), whose authors report it matching or beating preference-based methods at scales from 1 to 30 billion parameters while learning only from whether each answer was desirable. Open-source trainers such as Hugging Face TRL support it, but most hosted fine-tuning services don't offer it. A 👎 becomes useful for them only once there is a better answer to compare it with.
Raw thumbs can teach the wrong thing. People like answers that agree with them and sound confident. Anthropic's study of sycophancy (2023) found that both people and preference models sometimes prefer a convincingly written, agreeable answer over a correct one. A 2024 study that trained models directly on simulated user feedback went further: the models reliably learned manipulative or deceptive tactics to win thumbs up, and even when only 2% of users were vulnerable, learned to single them out. Routing 👎 answers through a reviewer keeps a person deciding what the right answer is.
The order that usually works
- Supervised fine-tuning on the liked answers — and the reviewers' corrections — so the model learns what good answers look like.
- DPO on the pairs, starting from the supervised model, so it learns which of two plausible answers is better.
OpenAI's DPO guidance recommends the same workflow: supervised fine-tuning on preferred responses first, then a DPO job from the result.
Settings
When on, an answer that was refreshed without any other rating counts as a bad answer. Leave it off unless your users only refresh when an answer is wrong. Pairs where the new answer was liked are used either way.
Common mistakes
- Training on every answer that wasn't disliked, as if silence meant approval.
- Turning on “also count Refresh” when users refresh mostly to see alternatives.
- Running DPO on feedback pairs without supervised training on good answers first.
- Letting a small group of raters' tastes define “good” without checking the result against an evaluation.
- Leaving the review queue unworked, so the most informative feedback — what went wrong — never becomes data.
References and further reading
External sources open in a new tab.
- 01KTO: Model Alignment as Prospect Theoretic OptimizationEthayarajh et al., 2024 · arXiv · arxiv.orgThe method built for single desirable/undesirable labels.
- 02KTO TrainerHugging Face TRL docs · huggingface.coWhat KTO needs as data, and how it differs from DPO in practice.
- 03Towards Understanding Sycophancy in Language ModelsSharma et al., 2023 · arXiv · arxiv.orgHow human preferences can reward agreeable answers over correct ones.
- 04On Targeted Manipulation and Deception when Optimizing LLMs for User FeedbackWilliams et al., 2024 · arXiv · arxiv.orgWhat models learn when trained directly to win users' thumbs up.
- 05Direct preference optimizationOpenAI API docs · developers.openai.comThe recommended order: supervised fine-tuning on preferred answers, then DPO.