Why loss isn't enough
In supervised training you watch the loss fall. In reinforcement fine-tuning the loss is a poor guide: the data changes every step because the model writes new answers, and the advantages are re-centred on zero in every group, so the reported loss can hover near zero while the model improves, or while it doesn't. The useful signals are about the answers: how they score, how much they differ, and how long they are.
The metrics
The grader's mean score over the step's answers, on 0–1. It should trend upward, noisily, since each step sees different prompts. Flat or falling means the model isn't finding better answers: the task may be beyond it, the grader may be too noisy, or the learning rate too low. Rising fast toward 1 deserves suspicion until you have read the answers. TRL logs it as reward; with several reward functions, each also gets its own line.
The share of answers that hit the max answer length and were cut off (completions/clipped_ratio in TRL). A cut-off answer is usually incomplete and scores badly, so a high ratio wastes much of the step and can push the model to be short for the wrong reason. Raise the limit, or ask for shorter answers in the system prompt. Not to be confused with the clipping of the update in GRPO's objective, which is logged separately.
How much scores differ. The reward standard deviation is the spread of scores (TRL computes it over the whole batch). The zero-std fraction (frac_reward_zero_std) is the share of prompts whose N answers all got the same score; those prompts produce no update. A spread near zero, or a zero-std fraction close to 1, means the run is starving: every answer is equally good or equally bad. Scalata shows the complement as prompts with a signal.
The average answer length in tokens (completions/mean_length). Some movement is normal: on hard reasoning tasks answers legitimately grow as the model learns to work longer, as DeepSeek-R1-Zero's did. On chat tasks, length rising in step with reward usually means the grader rewards length itself, a form of reward hacking. Length shrinking toward nothing with a flat reward can mean the model is avoiding penalties by saying little.
When KL β is above 0, TRL also logs kl, the average divergence from the starting model. A KL that keeps climbing while the reward has stopped improving says the model is still changing, but no longer for the better.
When the provider runs the grader itself (hosted reasoning models), it doesn't share the answers it scored during training. Instead, every few steps it asks a handful of validation questions the model never trains on and grades the answers with the same grader. Step 0 is the model before training, so each question becomes a small timeline. A low score at step 0 that rises is the clearest sign of learning that carries over; a question at 1.00 from step 0 can't show improvement; a dip in one check is sampling noise. Their mean, the validation reward, deserves more trust than the training reward, for the same reason validation loss beats training loss. Providers usually save checkpoints at the same steps, so if the checks stop improving halfway, the matching checkpoint may be as good as the final model.
Reading the charts together
| What you see | Likely meaning | What to do |
|---|---|---|
| Reward up, spread healthy, length steady | Learning | Let it run; confirm with an evaluation at the end |
| Reward flat near the floor, most prompts without a signal | Every answer is equally bad: nothing to reinforce | Start from a supervised run or a larger model; check the grader works |
| Reward flat near the top, most prompts without a signal | Prompts are too easy: every answer already scores well | Use harder prompts, or stop: little is left to gain |
| Reward and length rising together | The grader likes long answers | Read the reinforced answers; change or blend the grader; lower the length limit |
| High clipped ratio | Answers cut off at the limit | Raise max answer length, or ask for shorter answers |
| Reward up, but the evaluation is worse than the start | Reward hacking or drift | Train for fewer steps, raise KL β, change the grader |
- Steps 1–10: reward 0.31 → 0.38, 80% of prompts with a signal, clipped ratio 2%, length about 180 tokens. Healthy.
- Steps 10–30: reward 0.38 → 0.71, length 180 → 420 tokens, clipped ratio 2% → 18%. Reward and length are rising together, and answers are starting to hit the limit.
- Three groups on the run page: the green answers restate the question, list every possible caveat and repeat phrases from the reference. The judge liked thoroughness, and nothing penalised padding.
- Action: rerun with fewer steps and a lower length limit, or switch to a grader that checks content more strictly, then confirm with an evaluation that uses a different judge.
Common mistakes
- Judging an RL run by its loss. Watch reward, spread and length instead.
- Celebrating a reward near 1 before reading the top-scoring answers.
- Ignoring the clipped ratio. Cut-off answers spoil the scores of a whole step.
- Comparing rewards across runs with different graders or judges. The scales are not the same.
- Stopping at the charts. The final word is an evaluation on held-out questions, ideally with a different judge.
References and further reading
External sources open in a new tab.
- 01GRPO TrainerHugging Face TRL docs · huggingface.coDefinitions of reward, reward_std, frac_reward_zero_std, clipped_ratio and mean_length.
- 02Reinforcement fine-tuningOpenAI API docs · developers.openai.comReading train and validation reward in a hosted RFT job.
- 03DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement LearningDeepSeek-AI, 2025 · arXiv · arxiv.orgResponse length grew as R1-Zero learned to reason: a case where longer was genuinely better.
- 04Understanding R1-Zero-Like Training: A Critical PerspectiveLiu et al., 2025 · arXiv · arxiv.orgHow the original GRPO loss can inflate length, especially for wrong answers.