Resources/Model training/Training methods

Metric

RFT metrics to watch

Reward, clipped ratio, reward spread and completion length: the four charts that tell you whether a reinforcement run is learning, starved of signal, or gaming its grader.

Chapter 75 min read4 references

Why loss isn't enough

In supervised training you watch the loss fall. In reinforcement fine-tuning the loss is a poor guide: the data changes every step because the model writes new answers, and the advantages are re-centred on zero in every group, so the reported loss can hover near zero while the model improves, or while it doesn't. The useful signals are about the answers: how they score, how much they differ, and how long they are.

HealthyReward climbs, then levels off. Answers still differ within groups, and length stays steady.
Flat at the floorEvery answer scores about the same, so there is nothing to compare. Most prompts carry no signal.
Too good to be trueReward shoots up together with answer length. Read the answers: the grader may be rewarding padding.
reward (grader's mean score)completion length
Three shapes of the reward chart (solid) next to answer length (dashed). Only the first is a model getting better; the other two call for a change before more training.

The metrics

RewardTypical Up, then levelling off#

The grader's mean score over the step's answers, on 0–1. It should trend upward, noisily, since each step sees different prompts. Flat or falling means the model isn't finding better answers: the task may be beyond it, the grader may be too noisy, or the learning rate too low. Rising fast toward 1 deserves suspicion until you have read the answers. TRL logs it as reward; with several reward functions, each also gets its own line.

Clipped ratioTypical Near 0, a few percent at most#

The share of answers that hit the max answer length and were cut off (completions/clipped_ratio in TRL). A cut-off answer is usually incomplete and scores badly, so a high ratio wastes much of the step and can push the model to be short for the wrong reason. Raise the limit, or ask for shorter answers in the system prompt. Not to be confused with the clipping of the update in GRPO's objective, which is logged separately.

Reward std and zero-std fractionTypical Most prompts with a signal#

How much scores differ. The reward standard deviation is the spread of scores (TRL computes it over the whole batch). The zero-std fraction (frac_reward_zero_std) is the share of prompts whose N answers all got the same score; those prompts produce no update. A spread near zero, or a zero-std fraction close to 1, means the run is starving: every answer is equally good or equally bad. Scalata shows the complement as prompts with a signal.

Completion lengthTypical Stable, close to your replies' length#

The average answer length in tokens (completions/mean_length). Some movement is normal: on hard reasoning tasks answers legitimately grow as the model learns to work longer, as DeepSeek-R1-Zero's did. On chat tasks, length rising in step with reward usually means the grader rewards length itself, a form of reward hacking. Length shrinking toward nothing with a flat reward can mean the model is avoiding penalties by saying little.

When KL β is above 0, TRL also logs kl, the average divergence from the starting model. A KL that keeps climbing while the reward has stopped improving says the model is still changing, but no longer for the better.

Validation reward (held-out checks)Typical Rising from step 0, then levelling off#

When the provider runs the grader itself (hosted reasoning models), it doesn't share the answers it scored during training. Instead, every few steps it asks a handful of validation questions the model never trains on and grades the answers with the same grader. Step 0 is the model before training, so each question becomes a small timeline. A low score at step 0 that rises is the clearest sign of learning that carries over; a question at 1.00 from step 0 can't show improvement; a dip in one check is sampling noise. Their mean, the validation reward, deserves more trust than the training reward, for the same reason validation loss beats training loss. Providers usually save checkpoints at the same steps, so if the checks stop improving halfway, the matching checkpoint may be as good as the final model.

Reading the charts together

What you seeLikely meaningWhat to do
Reward up, spread healthy, length steadyLearningLet it run; confirm with an evaluation at the end
Reward flat near the floor, most prompts without a signalEvery answer is equally bad: nothing to reinforceStart from a supervised run or a larger model; check the grader works
Reward flat near the top, most prompts without a signalPrompts are too easy: every answer already scores wellUse harder prompts, or stop: little is left to gain
Reward and length rising togetherThe grader likes long answersRead the reinforced answers; change or blend the grader; lower the length limit
High clipped ratioAnswers cut off at the limitRaise max answer length, or ask for shorter answers
Reward up, but the evaluation is worse than the startReward hacking or driftTrain for fewer steps, raise KL β, change the grader
Reading a run (illustrative numbers)
  1. Steps 1–10: reward 0.31 → 0.38, 80% of prompts with a signal, clipped ratio 2%, length about 180 tokens. Healthy.
  2. Steps 10–30: reward 0.38 → 0.71, length 180 → 420 tokens, clipped ratio 2% → 18%. Reward and length are rising together, and answers are starting to hit the limit.
  3. Three groups on the run page: the green answers restate the question, list every possible caveat and repeat phrases from the reference. The judge liked thoroughness, and nothing penalised padding.
  4. Action: rerun with fewer steps and a lower length limit, or switch to a grader that checks content more strictly, then confirm with an evaluation that uses a different judge.

Common mistakes

  • Judging an RL run by its loss. Watch reward, spread and length instead.
  • Celebrating a reward near 1 before reading the top-scoring answers.
  • Ignoring the clipped ratio. Cut-off answers spoil the scores of a whole step.
  • Comparing rewards across runs with different graders or judges. The scales are not the same.
  • Stopping at the charts. The final word is an evaluation on held-out questions, ideally with a different judge.

References and further reading

External sources open in a new tab.

  1. 01GRPO TrainerHugging Face TRL docs · huggingface.coDefinitions of reward, reward_std, frac_reward_zero_std, clipped_ratio and mean_length.
  2. 02Reinforcement fine-tuningOpenAI API docs · developers.openai.comReading train and validation reward in a hosted RFT job.
  3. 03DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement LearningDeepSeek-AI, 2025 · arXiv · arxiv.orgResponse length grew as R1-Zero learned to reason: a case where longer was genuinely better.
  4. 04Understanding R1-Zero-Like Training: A Critical PerspectiveLiu et al., 2025 · arXiv · arxiv.orgHow the original GRPO loss can inflate length, especially for wrong answers.

Turn weeks of expert work into minutes

See how Scalata fits your team, your data, and your controls.