Resources/Model training/Evaluation

Best practice

Annotation queues: checking the judge by hand

People review model answers one at a time — scoring them, picking the better one or writing a correction — to measure how far the LLM judge can be trusted, and to turn their fixes into training data.

Chapter 86 min read5 references

What it is

An LLM judge is fast and cheap, but it is still a model. It can prefer the longer answer, miss a wrong number, or reward a confident tone. Strong judges agree with people about as often as people agree with each other — over 80% in Zheng et al. (2023) — but that is an average over public benchmarks, not a guarantee for your product's questions. Annotation queues put answers in front of a person, so you know how far to trust the judge on your data.

An annotation is a person's label on a model output: a score per criterion, a choice between two answers, or a corrected answer. An annotation queue is a to-do list of such items, with the rules for reviewing them.

How a queue works

  • Items are what is reviewed. From an evaluation, each judged question: the question, both answers and the judge's verdict. From a dataset, each trainable reply in its conversation. Items are copies, so archiving or deleting the source doesn't change the queue.
  • The rubric is the criteria scored 1–5 per answer. Using the judge's own criteria — correctness, helpfulness, tone, format — makes human and judge scores comparable. It locks once anything is reviewed; otherwise earlier scores would mean something else.
  • Blind review shows “Answer A / Answer B” without model names, and hides the judge's verdict, until the review is saved.
  • Locking keeps an item with the person reviewing it while they work on it, so two people don't review the same item.
Item 7 of 40blind

userHow do I share a report with a client?

Answer AOpen the report → Share → pick the client. They get an email link.
Answer BYou can export the report as a PDF and email it.
AB
Correctness52
Helpfulness43
Tone44
Format44
ABTpick A, B or tie⌘/Ctrl + Entersave
After savingA = tuned model · B = base model · judge picked A
One item in a blind queue. The reviewer sees the question, the reference and two answers labelled only A and B, scores both on the rubric and picks one. Model names and the judge's verdict appear only after saving, so neither can anchor the review.

Why blind review matters

People are anchored by what they see first. Show a reviewer that answer A came from the newly tuned model and they lean towards it, because that is the result they hope for. Show them the judge's verdict and they tend to agree with it — which defeats the purpose of checking the judge. Hiding both until the review is saved keeps the human label independent. Saving then reveals who was who and what the judge said, which is the moment disagreements become visible.

What to fill a queue with

PickWhy
A random sample (20–50) of all questionsA fair spot check of the judge
Questions where the other answer wonIs the tuned model really worse there, or is the judge wrong?
TiesThe judge couldn't decide; a person usually can
Low judge scoresThe best candidates for corrections
A dataset's repliesCheck training data before training: bad replies teach bad habits

Only a random sample measures how often the judge is right in general. The other picks are deliberately skewed towards hard cases: great for finding problems and writing corrections, misleading as an estimate of overall agreement.

Reading the results

COIN FLIPdon't rely on the judge alonea fair guide0%60%80%100%19 of 25 agree
Agreement is the share of reviewed questions where the person and the judge picked the same answer. With two answers to choose from, 50% is what a coin would get. In this illustrative queue the reviewer agreed with the judge on 19 of 25 questions (76%): close to a fair guide, and the six disagreements are worth reading.
the two numbers on a queue page
agreement      = questions where you and the judge picked the same answer
                 ÷ questions you both decided

score gap      = your average − judge's average, per criterion,
                 on the same answers          (|gap| ≥ 1 point → look closer)
  • Agreement with the judge. Around 80% or more: the judge's win rates are a fair guide. Under about 60%: don't decide on the judge's numbers alone — review more, change the judge model, or adjust what it is asked to check. Remember that with two answers to choose from, a coin would agree half the time, so 60% is only a little better than chance.
  • Score gaps. Per criterion, your average against the judge's on the same answers. A gap of 1 point or more means the judge scores that criterion differently from people. Judges often go easy on the correctness of product facts and hard on format.
  • Disagreements are the interesting items. Read why the judge chose differently. Either it is a judge problem — fix the judge — or a data problem: the reference reply itself was wrong or out of date.

Turning reviews into training data

  • A correction becomes the reply to train on (supervised data). The answer you liked less is stored as its worse pair, so the same example also serves preference tuning (DPO).
  • Without a correction, the answer you picked is the reply and the other one its worse pair.
  • Ties without a correction, and scores on their own, aren't exported. Exporting again updates the same examples instead of adding copies.
one exported example: the reply to learn, and the rejected answer
{"messages": [
  {"role": "user", "content": "How do I share a report with a client?"},
  {"role": "assistant", "content": "Open the report → Share → pick the client. They get an email link.",
   "rejected": "You can export the report as a PDF and email it.", "rejectedSource": "review"}
]}
Worked example: a 30-item spot check (illustrative)
  1. A blind queue is filled with a random sample of 30 questions from an evaluation, using the judge's four criteria as the rubric.
  2. Agreement: the reviewer and the judge picked the same answer on 24 of 30 questions — 80%. The judge's win rate is a fair guide for this model.
  3. Score gaps: on correctness the reviewer averages 2.9 against the judge's 4.0, a 1.1-point gap — the judge misses wrong product facts. On format the judge is the stricter one.
  4. Of the six disagreements, four are judge problems: it rewarded an answer that invented a feature. Two are data problems: the reference reply described a menu that has since moved.
  5. On one question both answers invented a capability. Picking A or B would reward an invented answer either way, so the reviewer writes a correction instead.
  6. Export: 5 corrections and 19 picks become 24 training examples with preference pairs; the remaining ties are not exported.

Settings and numbers

RubricTypical correctness, helpfulness, tone, format#

The criteria reviewers score from 1 to 5 for each answer. Use the judge's criteria to compare human and judge scores directly. Locked once the first item is reviewed.

Blind reviewTypical on#

Hides model names and the judge's verdict until the review is saved, so neither can anchor the reviewer. Turn it off only when you are labelling data rather than checking the judge.

Agreement with the judgeTypical ≈ 80%+ fair guide · under ≈ 60% don't rely on the judge alone#

The share of reviewed questions where you and the judge picked the same answer. Most meaningful on a random sample.

Score gapTypical 1 point or more: look closer#

Your average minus the judge's on the same answers, per criterion. Shows where the judge's sense of quality differs from yours.

Export to datasetTypical after a review round, before the next training run#

Writes reviewed items as training examples: a correction or the picked answer as the reply, the other answer as its worse pair. Ties without a correction and scores alone are skipped; exporting again updates the same examples.

Common mistakes

  • Reviewing with model names or the judge's verdict visible, then “confirming” the judge.
  • Estimating overall agreement from a queue filled with ties and losses rather than a random sample.
  • Picking A or B when both answers are wrong, instead of writing a correction.
  • Changing what the criteria mean halfway through — the rubric locks for this reason.
  • Blaming the judge for every disagreement when the reference reply itself was out of date.

References and further reading

External sources open in a new tab.

  1. 01Judging LLM-as-a-Judge with MT-Bench and Chatbot ArenaZheng et al., 2023 · arXiv · arxiv.orgJudge–human agreement over 80%, about the human–human level, and the judge's biases.
  2. 02Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human PreferencesShankar et al., 2024 · arXiv · arxiv.orgChecking LLM graders against human grades, and how criteria drift as people review.
  3. 03Cohen's kappaWikipedia · en.wikipedia.orgAgreement corrected for chance — why 60% on a two-way choice is weak.
  4. 04Inter-rater reliabilityWikipedia · en.wikipedia.orgHow agreement between raters is measured and read.
  5. 05Anchoring effectWikipedia · en.wikipedia.orgThe bias blind review protects against.

Turn weeks of expert work into minutes

See how Scalata fits your team, your data, and your controls.