What a grader is
In reinforcement fine-tuning the model writes several answers per prompt and needs a number for each. The grader produces it: a function, a model, or a mix of both, that reads the prompt, the answer and, in Scalata, your own reply as a reference, and returns a score from 0 to 1. GRPO only uses the differences between answers to the same prompt, so a grader's most important property is that it ranks answers the way you would.
Graders come in two families. Rule-based graders compute the score: exact match, word overlap, comparing numbers, validating a format. They are fast, free and perfectly consistent, but they only measure what they were written to measure. Model-based graders, an LLM acting as judge, read the answer as a person would and can weigh correctness, helpfulness and tone, at the cost of tokens, time and their own biases. DeepSeek-R1 used only rule-based rewards for its reasoning RL, because its authors found learned reward models open to reward hacking at scale.
The graders
An LLM reads the conversation, your reference reply and all N answers together, and scores each from 1 to 5 on correctness, helpfulness, tone and format; the marks are scaled to 0–1. Because it understands meaning, a correct answer in different words still scores well. It costs one judge call per prompt per epoch, and a weak judge gives noisy scores.
Word overlap between the answer and your reply, measured as ROUGE-L F1 (0–1). It needs no judge model, so it costs nothing. It rewards wording, not meaning: a correct answer phrased differently scores low, and a wrong answer that reuses your words scores high.
70% judge score plus 30% similarity. The judge decides what is good; the similarity term pulls answers toward your wording. You pay for the judge, and keep both blind spots, each at a smaller dose.
Takes the key figures in your reply (decimals, percentages, amounts; not list numbering) and matches each to the closest number in the answer: within 1% earns full credit, within 5% half, beyond that nothing. Units are aligned, so 1,705M and 1.705B count as the same figure. Replies with no numbers are skipped. It checks the figures only, not the explanation around them.
50% judge plus 50% numbers, so both the reasoning and the figures count. When your reply has no numbers, the judge score is used alone. Costs judge tokens like the judge alone.
The answer equals or contains the expected value, or a piece of code checks its format (valid JSON, a known label). Scores are usually all-or-nothing, which gives GRPO a very clean signal, but it only works when there is one checkable answer. Hosted providers offer these as built-in graders, for example a string check, next to text similarity, a scoring model and combinations of graders.
How similarity is measured: ROUGE-L
ROUGE was introduced by Chin-Yew Lin in 2004 to score machine-written summaries against human ones. The L variant looks for the longest common subsequence (LCS): the longest run of words that appear in both texts in the same order, not necessarily next to each other. Precision is LCS ÷ words in the answer, recall is LCS ÷ words in the reference, and F1 combines the two (their harmonic mean).
- Reference, 9 words: “Open the report, click Share and pick the client.”
- Answer A: “Click Share on the report and pick the client.” Longest common subsequence: “click share … and pick the client”, 6 words. Precision 6/9, recall 6/9, F1 ≈ 0.67.
- Answer B, correct but reworded: “Use the Share button in your report to send it to them.” (12 words). Longest common subsequence: just “the report”, 2 words. Precision 2/12, recall 2/9, F1 ≈ 0.19.
- Answer C, wrong recipient: “Open the report, click Share and pick the admin.” 8 words in common, F1 ≈ 0.89, the highest of the three.
- Implementations differ slightly in how they split words and whether they reduce words to their stems, but the lesson holds: similarity measures wording, which is why it is usually blended with a judge.
How the numbers grader scores
- Your reply: “WACC ≈ 9.91%; cost of equity 10.34%; after-tax debt 4.15%; EV ≈ $1,705M.” Four key figures.
- The answer contains 9.95%, 10.3%, 4.2% and 1,700M.
- 9.95 vs 9.91 is 0.4% off, 10.3 vs 10.34 is 0.4% off, 1,700 vs 1,705 is 0.3% off: full credit for each.
- 4.2 vs 4.15 is 1.2% off: within 5%, so half credit.
- Score = (1 + 1 + 1 + 0.5) ÷ 4 = 0.875. An answer saying only “WACC is 1.33%” scores 0.
The numbers grader checks figures, not the reasoning around them: a right number reached the wrong way still scores, and a sound method with one slip loses credit. An answer that lists many figures also has more chances of landing near yours. Judge + numbers exists to cover both gaps.
Why the judge sees all answers at once
In Scalata the judge sees all N answers to a prompt together. Seeing them side by side, it scores them consistently relative to each other, and consistent relative scores are exactly what GRPO uses. The judge's per-criterion marks and a short reason for every answer are shown on the run page next to each group.
Choosing a grader
| Your task | Start with | Why |
|---|---|---|
| Open-ended support or advisory chat | Judge model | Only a model can judge whether free text is right and helpful |
| Answers must also sound like your team | Judge + similarity | The judge guards meaning; similarity nudges toward your wording |
| Calculations with exact results | Judge + numbers, or numbers alone | Figures are checkable; the judge catches nonsense around them |
| One right label or value | Exact match / checks | All-or-nothing scores give the cleanest signal |
| A fixed output format | A format check, possibly blended with a judge | Validity is checkable; content may still need judgement |
Whichever you choose, score five or ten answers yourself and compare with the grader before a long run. If you disagree with its ranking, the model will learn the grader's opinion, not yours.
Common mistakes
- Using similarity alone where meaning matters. Correct paraphrases are punished and wrong answers in your words are rewarded.
- Picking a weak judge for a long run. Noisy scores become noisy advantages, and GRPO scales small differences up.
- Using the same model as grader and as evaluation judge. Its blind spots then look like improvements twice.
- Using the numbers grader on data whose replies have few figures; those rows are skipped and the run learns from a fraction of your data.
- Forgetting that anything the grader doesn't check, such as length, tone or invented details, is free for the model to change.
References and further reading
External sources open in a new tab.
- 01ROUGE: A Package for Automatic Evaluation of SummariesLin, 2004 · ACL Anthology · aclanthology.orgThe original definition of ROUGE-L and its longest-common-subsequence F-measure.
- 02GradersOpenAI API docs · developers.openai.comHosted grader types: string check, text similarity, score model, Python and multi-graders.
- 03Judging LLM-as-a-Judge with MT-Bench and Chatbot ArenaZheng et al., 2023 · arXiv · arxiv.orgHow well LLM judges agree with people, and their known biases: position, verbosity, self-preference.
- 04DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement LearningDeepSeek-AI, 2025 · arXiv · arxiv.orgWhy rule-based rewards were preferred over learned reward models for reasoning RL.