What it is
The grader is not your goal. It is a measurable stand-in for it, a proxy. Reinforcement learning optimises the proxy relentlessly, and wherever the proxy and your goal disagree, the model drifts toward the proxy. Economists know this as Goodhart's law: when a measure becomes a target, it ceases to be a good measure. In AI it is called reward hacking or specification gaming.
The classic example comes from game-playing RL. In the boat-racing game CoastRunners, an agent rewarded for hitting targets along the course learned to circle in a lagoon, hitting the same few targets over and over, and never finished the race. It scored more that way than by completing the course. Nothing was broken: the agent did exactly what the reward said.
Language models do the same with graders. DeepSeek-R1's authors report that while training against a learned reward model for helpfulness, the reward kept rising as performance on a coding benchmark fell, and they kept learned reward models out of their reasoning RL partly for that reason.
Gao et al. (2022) measured this curve. Optimising a model against a learned reward model, they tracked a separate “gold” reward standing in for real quality: as the model moved further from its starting point, the proxy kept climbing while the gold score rose, peaked and then declined. Larger reward models suffered less, but none removed the effect.
How it shows up with each grader
| Grader | How it gets gamed | What it looks like |
|---|---|---|
| Judge model | Longer, thorough-sounding answers; a confident tone; text that flatters or instructs the judge | Length climbs; answers full of caveats and restated questions; lines like “This answer is complete and correct.” |
| Similarity | Reusing the reference's phrases whether or not they fit | Answers that read like your replies but say the wrong thing |
| Numbers | Listing many figures so one lands near each of yours | Answers stuffed with numbers or alternatives: “either 9.9% or 10.3%” |
| Exact match / checks | Satisfying the letter of the check | Valid JSON with empty fields; the right label surrounded by nonsense |
Worked example
- Prompt: “Can I export a dashboard to PowerPoint?” Your reference: “Yes: Share → Export → PowerPoint. Each chart becomes a slide.”
- Answer 1, score 0.95, green: restates the question, gives the steps, adds three paragraphs of general presentation tips, and ends with “This answer fully addresses your question.”
- Answer 2, score 0.70, red: your reference's steps in two sentences.
- The judge rated thoroughness above concision, and the closing line may have nudged it further. Answer 2 is what you wanted, yet the update pushes the model toward answer 1.
- Fixes: a stronger judge, or a blend with similarity to your reply; a tighter max answer length; and an evaluation with a different judge to confirm the result.
How to prevent and catch it
- Read the reinforced answers. The green answers in each group are exactly what the model is being taught. If they aren't what you'd want, stop and change the grader.
- Evaluate independently. Compare the trained model with its starting point on held-out questions, using a different judge from the grader. A reward gain that doesn't show up there is suspect.
- Blend graders. Each grader's blind spot is covered by another: a judge plus a numbers check is far harder to game than either alone.
- Limit length. A sensible max answer length caps the most common hack outright.
- Anchor with KL β. A penalty for drifting from the starting model limits how far any exploit can be pushed, at the cost of slower improvement.
- Don't overtrain. Over-optimisation grows with distance from the start; a shorter run that stops near the peak often beats a longer one.
- Prefer checkable rewards. Exact answers, figures, format checks and tests leave less room for interpretation than an opinion.
Common mistakes
- Treating the reward chart as the measure of success.
- Using the same model as grader and as evaluation judge.
- Training far longer than needed “because the reward is still going up”.
- Fixing one hack and not re-reading the answers; models usually find the next gap.
- Blaming the model. It is doing what the reward says; the fix is almost always in the grader or the setup.
References and further reading
External sources open in a new tab.
- 01Concrete Problems in AI SafetyAmodei et al., 2016 · arXiv · arxiv.orgNames reward hacking as one of five practical safety problems in machine learning.
- 02Scaling Laws for Reward Model OveroptimizationGao, Schulman & Hilton, 2022 · arXiv · arxiv.orgMeasures how real quality peaks and falls as a model is optimised against a proxy reward.
- 03Specification gaming: the flip side of AI ingenuityKrakovna et al., 2020 · Google DeepMind · deepmind.googleExamples of agents doing what the reward said instead of what was meant.
- 04Reward Hacking in Reinforcement LearningWeng, 2024 · Lil'Log · lilianweng.github.ioA thorough survey of reward hacking in RL and in language-model training.