What they are
On open models, Scalata can show you the real groups of answers the grader scored. On hosted reasoning models, the provider runs the grader itself and doesn't share the answers it scored during training. What it does share is a set of held-out checks: every few steps it asks a handful of validation questions the model never trains on, and grades those answers with the same grader.
Because the first check runs at step 0, before any training, each validation question becomes a small timeline: how the answer to the same question changed as training went on. That makes the checks the best available evidence that the gains carry over to questions the model hasn't seen, and not just to the ones it practised on.
Reading the timeline
- Low at step 0, then rising: the clearest sign of learning that carries over to new questions.
- 1.00 from step 0 onward: a ceiling. The model already answered well enough, so this question can't show improvement. Many of these means the grader is too easy, or the questions are.
- A dip in the middle: normal noise. Each check is one sampled answer, and one unlucky sample can score low. Look at the trend across checks and across questions, not at one card.
- Flat everywhere below the ceiling: the model isn't finding better answers. The usual cause is a starting model that rarely gets these right; a supervised run first helps.
| Question (illustrative) | Step 0 | 5 | 10 | 15 | 20 | Reading |
|---|---|---|---|---|---|---|
| How do I export a dashboard to PowerPoint? | 0.25 | 0.38 | 0.56 | 0.81 | 0.81 | Learning, levels off after 15 |
| Can a client open a shared link without an account? | 0.50 | 0.69 | 0.75 | 0.94 | 0.94 | Learning |
| Which file types can I upload? | 0.75 | 0.94 | 0.32 | 0.88 | 0.94 | One bad sample at 10 |
| How do I reset my password? | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | Ceiling: says nothing |
| Validation reward (mean) | 0.63 | 0.75 | 0.66 | 0.91 | 0.92 | Real gain; flat after 15 |
- The first two questions start low and climb: the answers moved toward the facts in your reference replies. That is the learning you trained for.
- The third drops to 0.32 at step 10 and recovers to 0.88 and 0.94. One poor sample, not a trend, but it pulls the step-10 mean down from where it would otherwise be.
- The fourth scores 1.00 throughout. It was already answered well before training, so it can't show anything.
- Validation reward rises from 0.63 to 0.92, but 0.91 at step 15 and 0.92 at step 20 are almost the same. The step-15 checkpoint is likely as good as the final model.
Validation reward vs training reward
The provider reports two rewards. Training reward is the grader's mean score on the questions the model is training on at that step; those questions change from step to step, so the number jumps around, and a model can raise it by getting better at exactly those prompts. Validation reward is the mean of the held-out checks: the same questions every time, never trained on. Trust it more, for the same reason validation loss beats training loss. OpenAI's documentation describes it the same way: training reward values “can fluctuate drastically from step to step”, while validation reward is “a more stable metric”.
Checks and checkpoints
Providers usually save a checkpoint at the same steps as the checks. So if the checks stop improving halfway, the matching checkpoint may be as good as the final model, and a shorter run would have cost less. Hosted reinforcement fine-tuning is billed per training hour and reasoning models write long answers, so each step is slow: noting where the checks flatten is a direct saving on the next run.
What the model trains on
Reinforcement fine-tuning uses the same chats as supervised training, turned into one row per reply, ending at that reply. The conversation up to the reply is the prompt; the reply itself becomes a reference that only the grader sees. A chat with three replies gives three rows, and each row's reference is only its last reply, with earlier replies kept in the prompt as context.
chat: system · user 1 · reply 1 · user 2 · reply 2 · user 3 · reply 3
row 1 prompt: developer · user 1 reference: reply 1
row 2 prompt: developer · user 1 · reply 1 · user 2 reference: reply 2
row 3 prompt: developer · user 1 · reply 1 · user 2 · reply 2 · user 3 reference: reply 3Hosted reasoning models (the o-series) don't accept a system message in reinforcement examples. The same instructions go in a developer message, which these models treat the way other models treat system; a training file with a system message there fails validation.
{"messages": [
{"role": "developer", "content": "You are Scalata's support assistant. Be brief."},
{"role": "user", "content": "How do I share a report with a client?"}],
"reference": "Open the report → Share → pick the client. They get an email link."}Metrics
The mean grader score over the held-out checks at one step. The same questions every time, never trained on, so it is a steadier and more honest signal than training reward. A rising validation reward is learning that carries over; one that flattens marks where a shorter run would have done.
The check before any training: the starting model's answers to the validation questions. Every later check is read against it. A question that is already high at step 0 has little room to show improvement.
A question that scores 1.00 from step 0 onward. The grader can't score higher, so it can't show any change. A few are normal; many mean the grader or the validation questions are too easy to measure progress.
Common mistakes
- Reading one card. A single check is one sampled answer; judge the trend across checks and questions.
- Trusting a rising training reward when validation reward is flat. The model may be fitting the training prompts, or gaming the grader.
- Ignoring ceilings. Questions at 1.00 throughout inflate the average and hide whether anything changed.
- Always using the final model. If the checks flattened halfway, the matching checkpoint is as good and the next run can be shorter.
- Writing a system message in hand-built reinforcement files for o-series models. Use a developer message instead.
References and further reading
External sources open in a new tab.
- 01Reinforcement fine-tuningOpenAI API docs · developers.openai.comThe hosted RFT service: validation files, training and validation reward, and checkpoints.
- 02GradersOpenAI API docs · developers.openai.comHow the provider's graders are defined and what they read from each row.
- 03Fine-tune a model with reinforcement fine-tuning in Microsoft FoundryMicrosoft Learn · learn.microsoft.comReinforcement fine-tuning of reasoning models on Azure, including validation data.
- 04Scaling Laws for Reward Model OveroptimizationGao, Schulman & Hilton, 2022 · arXiv · arxiv.orgWhy a score the model is optimised against can rise while true quality falls.