What it is
An LLM judge is fast and cheap, but it is still a model. It can prefer the longer answer, miss a wrong number, or reward a confident tone. Strong judges agree with people about as often as people agree with each other — over 80% in Zheng et al. (2023) — but that is an average over public benchmarks, not a guarantee for your product's questions. Annotation queues put answers in front of a person, so you know how far to trust the judge on your data.
An annotation is a person's label on a model output: a score per criterion, a choice between two answers, or a corrected answer. An annotation queue is a to-do list of such items, with the rules for reviewing them.
How a queue works
- Items are what is reviewed. From an evaluation, each judged question: the question, both answers and the judge's verdict. From a dataset, each trainable reply in its conversation. Items are copies, so archiving or deleting the source doesn't change the queue.
- The rubric is the criteria scored 1–5 per answer. Using the judge's own criteria — correctness, helpfulness, tone, format — makes human and judge scores comparable. It locks once anything is reviewed; otherwise earlier scores would mean something else.
- Blind review shows “Answer A / Answer B” without model names, and hides the judge's verdict, until the review is saved.
- Locking keeps an item with the person reviewing it while they work on it, so two people don't review the same item.
userHow do I share a report with a client?
Why blind review matters
People are anchored by what they see first. Show a reviewer that answer A came from the newly tuned model and they lean towards it, because that is the result they hope for. Show them the judge's verdict and they tend to agree with it — which defeats the purpose of checking the judge. Hiding both until the review is saved keeps the human label independent. Saving then reveals who was who and what the judge said, which is the moment disagreements become visible.
What to fill a queue with
| Pick | Why |
|---|---|
| A random sample (20–50) of all questions | A fair spot check of the judge |
| Questions where the other answer won | Is the tuned model really worse there, or is the judge wrong? |
| Ties | The judge couldn't decide; a person usually can |
| Low judge scores | The best candidates for corrections |
| A dataset's replies | Check training data before training: bad replies teach bad habits |
Only a random sample measures how often the judge is right in general. The other picks are deliberately skewed towards hard cases: great for finding problems and writing corrections, misleading as an estimate of overall agreement.
Reading the results
agreement = questions where you and the judge picked the same answer
÷ questions you both decided
score gap = your average − judge's average, per criterion,
on the same answers (|gap| ≥ 1 point → look closer)- Agreement with the judge. Around 80% or more: the judge's win rates are a fair guide. Under about 60%: don't decide on the judge's numbers alone — review more, change the judge model, or adjust what it is asked to check. Remember that with two answers to choose from, a coin would agree half the time, so 60% is only a little better than chance.
- Score gaps. Per criterion, your average against the judge's on the same answers. A gap of 1 point or more means the judge scores that criterion differently from people. Judges often go easy on the correctness of product facts and hard on format.
- Disagreements are the interesting items. Read why the judge chose differently. Either it is a judge problem — fix the judge — or a data problem: the reference reply itself was wrong or out of date.
Turning reviews into training data
- A correction becomes the reply to train on (supervised data). The answer you liked less is stored as its worse pair, so the same example also serves preference tuning (DPO).
- Without a correction, the answer you picked is the reply and the other one its worse pair.
- Ties without a correction, and scores on their own, aren't exported. Exporting again updates the same examples instead of adding copies.
{"messages": [
{"role": "user", "content": "How do I share a report with a client?"},
{"role": "assistant", "content": "Open the report → Share → pick the client. They get an email link.",
"rejected": "You can export the report as a PDF and email it.", "rejectedSource": "review"}
]}- A blind queue is filled with a random sample of 30 questions from an evaluation, using the judge's four criteria as the rubric.
- Agreement: the reviewer and the judge picked the same answer on 24 of 30 questions — 80%. The judge's win rate is a fair guide for this model.
- Score gaps: on correctness the reviewer averages 2.9 against the judge's 4.0, a 1.1-point gap — the judge misses wrong product facts. On format the judge is the stricter one.
- Of the six disagreements, four are judge problems: it rewarded an answer that invented a feature. Two are data problems: the reference reply described a menu that has since moved.
- On one question both answers invented a capability. Picking A or B would reward an invented answer either way, so the reviewer writes a correction instead.
- Export: 5 corrections and 19 picks become 24 training examples with preference pairs; the remaining ties are not exported.
Settings and numbers
The criteria reviewers score from 1 to 5 for each answer. Use the judge's criteria to compare human and judge scores directly. Locked once the first item is reviewed.
Hides model names and the judge's verdict until the review is saved, so neither can anchor the reviewer. Turn it off only when you are labelling data rather than checking the judge.
The share of reviewed questions where you and the judge picked the same answer. Most meaningful on a random sample.
Your average minus the judge's on the same answers, per criterion. Shows where the judge's sense of quality differs from yours.
Writes reviewed items as training examples: a correction or the picked answer as the reply, the other answer as its worse pair. Ties without a correction and scores alone are skipped; exporting again updates the same examples.
Common mistakes
- Reviewing with model names or the judge's verdict visible, then “confirming” the judge.
- Estimating overall agreement from a queue filled with ties and losses rather than a random sample.
- Picking A or B when both answers are wrong, instead of writing a correction.
- Changing what the criteria mean halfway through — the rubric locks for this reason.
- Blaming the judge for every disagreement when the reference reply itself was out of date.
References and further reading
External sources open in a new tab.
- 01Judging LLM-as-a-Judge with MT-Bench and Chatbot ArenaZheng et al., 2023 · arXiv · arxiv.orgJudge–human agreement over 80%, about the human–human level, and the judge's biases.
- 02Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human PreferencesShankar et al., 2024 · arXiv · arxiv.orgChecking LLM graders against human grades, and how criteria drift as people review.
- 03Cohen's kappaWikipedia · en.wikipedia.orgAgreement corrected for chance — why 60% on a two-way choice is weak.
- 04Inter-rater reliabilityWikipedia · en.wikipedia.orgHow agreement between raters is measured and read.
- 05Anchoring effectWikipedia · en.wikipedia.orgThe bias blind review protects against.