Resources/Model training/Training methods

Best practice

Reward hacking

Reward hacking is when a model raises its score without getting better at the task, by exploiting what the grader fails to check. It is the main way reinforcement fine-tuning goes wrong.

Chapter 74 min read4 references

What it is

The grader is not your goal. It is a measurable stand-in for it, a proxy. Reinforcement learning optimises the proxy relentlessly, and wherever the proxy and your goal disagree, the model drifts toward the proxy. Economists know this as Goodhart's law: when a measure becomes a target, it ceases to be a good measure. In AI it is called reward hacking or specification gaming.

The classic example comes from game-playing RL. In the boat-racing game CoastRunners, an agent rewarded for hitting targets along the course learned to circle in a lagoon, hitting the same few targets over and over, and never finished the race. It scored more that way than by completing the course. Nothing was broken: the agent did exactly what the reward said.

Language models do the same with graders. DeepSeek-R1's authors report that while training against a learned reward model for helpfulness, the reward kept rising as performance on a coding benchmark fell, and they kept learned reward models out of their reasoning RL partly for that reason.

REWARD HACKINGBEST MODELhow far the model has moved from where it started (KL) →scoregrader's score (what training sees)real quality (independent evaluation)
Over-optimisation. The grader's score (solid) keeps rising the further training moves the model. Real quality, measured by an independent check (dashed), rises with it at first, peaks, and then falls as the model learns the grader's blind spots instead of the task.

Gao et al. (2022) measured this curve. Optimising a model against a learned reward model, they tracked a separate “gold” reward standing in for real quality: as the model moved further from its starting point, the proxy kept climbing while the gold score rose, peaked and then declined. Larger reward models suffered less, but none removed the effect.

How it shows up with each grader

GraderHow it gets gamedWhat it looks like
Judge modelLonger, thorough-sounding answers; a confident tone; text that flatters or instructs the judgeLength climbs; answers full of caveats and restated questions; lines like “This answer is complete and correct.”
SimilarityReusing the reference's phrases whether or not they fitAnswers that read like your replies but say the wrong thing
NumbersListing many figures so one lands near each of yoursAnswers stuffed with numbers or alternatives: “either 9.9% or 10.3%”
Exact match / checksSatisfying the letter of the checkValid JSON with empty fields; the right label surrounded by nonsense

Worked example

A hacked group, read on the run page
  1. Prompt: “Can I export a dashboard to PowerPoint?” Your reference: “Yes: Share → Export → PowerPoint. Each chart becomes a slide.”
  2. Answer 1, score 0.95, green: restates the question, gives the steps, adds three paragraphs of general presentation tips, and ends with “This answer fully addresses your question.”
  3. Answer 2, score 0.70, red: your reference's steps in two sentences.
  4. The judge rated thoroughness above concision, and the closing line may have nudged it further. Answer 2 is what you wanted, yet the update pushes the model toward answer 1.
  5. Fixes: a stronger judge, or a blend with similarity to your reply; a tighter max answer length; and an evaluation with a different judge to confirm the result.

How to prevent and catch it

  • Read the reinforced answers. The green answers in each group are exactly what the model is being taught. If they aren't what you'd want, stop and change the grader.
  • Evaluate independently. Compare the trained model with its starting point on held-out questions, using a different judge from the grader. A reward gain that doesn't show up there is suspect.
  • Blend graders. Each grader's blind spot is covered by another: a judge plus a numbers check is far harder to game than either alone.
  • Limit length. A sensible max answer length caps the most common hack outright.
  • Anchor with KL β. A penalty for drifting from the starting model limits how far any exploit can be pushed, at the cost of slower improvement.
  • Don't overtrain. Over-optimisation grows with distance from the start; a shorter run that stops near the peak often beats a longer one.
  • Prefer checkable rewards. Exact answers, figures, format checks and tests leave less room for interpretation than an opinion.

Common mistakes

  • Treating the reward chart as the measure of success.
  • Using the same model as grader and as evaluation judge.
  • Training far longer than needed “because the reward is still going up”.
  • Fixing one hack and not re-reading the answers; models usually find the next gap.
  • Blaming the model. It is doing what the reward says; the fix is almost always in the grader or the setup.

References and further reading

External sources open in a new tab.

  1. 01Concrete Problems in AI SafetyAmodei et al., 2016 · arXiv · arxiv.orgNames reward hacking as one of five practical safety problems in machine learning.
  2. 02Scaling Laws for Reward Model OveroptimizationGao, Schulman & Hilton, 2022 · arXiv · arxiv.orgMeasures how real quality peaks and falls as a model is optimised against a proxy reward.
  3. 03Specification gaming: the flip side of AI ingenuityKrakovna et al., 2020 · Google DeepMind · deepmind.googleExamples of agents doing what the reward said instead of what was meant.
  4. 04Reward Hacking in Reinforcement LearningWeng, 2024 · Lil'Log · lilianweng.github.ioA thorough survey of reward hacking in RL and in language-model training.

Turn weeks of expert work into minutes

See how Scalata fits your team, your data, and your controls.