What it is
A single evaluation answers “did this training run help?”. Sooner or later you have a different question: which of these models should we use? Perhaps a distilled open-weight model, a reinforcement-tuned model on one provider and a preference-tuned model on another. Their separate evaluations can't answer it, because each used its own questions, maybe its own judge — and a 4.0 from one setup is not a 4.0 from another.
A comparison fixes the conditions so the numbers line up. It is the same idea as public leaderboards such as HELM and Chatbot Arena — a shared test, applied the same way to every contestant — scaled down to your own questions.
How it works
- Pick the questions once: a prep's held-out test questions, its validation questions, or a dataset.
- Run one evaluation per model on exactly those questions, each model still compared with its own base (or a checkpoint, or scored alone).
- Rank the models by their average judge score, and show every model's answer per question next to the reference.
answer score = mean(correctness, helpfulness, tone, format) 1–5
model score = mean of its answer scores over the shared questions
ranking = models judged the same way, highest model score firstThe ranking is the summary; the per-question view is where you learn. Seeing every model's answer to the same question next to your reference shows what the best model still gets wrong, and whether the gap between two models comes from a few questions or from all of them.
Fair questions
| Question source | Fair when |
|---|---|
| A prep's held-out test questions | Every model was trained on that prep. A model trained on a different prep may have seen some of them. |
| A prep's validation questions | As above, and none of the models learned from them — though the provider saw them during training. |
| A dataset | None of the compared models was trained on it. |
| A training sample | Never: it always favours the model it came from. |
Side by side vs alone: two scales
The same judge gives noticeably lower scores to an answer read on its own than to the same kind of answer read next to another one — in Scalata's measurements, about a full point on the 1–5 scale for the same model. So a comparison only ranks models that were judged the same way. A model scored alone is listed separately and marked not comparable. To keep a model whose base is gone in the ranking, compare it with one of its checkpoints instead.
Judge independence
Don't use one of the compared models — or its base — as the judge. Judges rate text that sounds like their own writing more highly, so the judge's own family would get a quiet head start. Pick a judge from a family none of the contestants belongs to, and keep it the same for the whole comparison.
Cost and time
A comparison costs the sum of its evaluations: each model's answers, its baseline's answers and the judge's calls, plus hosting where a model needs a temporary deployment.
- The questions: 30 held-out test questions from the prep all three models were trained on. The judge: a model from a fourth family.
- Model A, distilled from a larger model, is evaluated against its base. Model B, reinforcement-tuned, has a retired base, so it is compared with an earlier checkpoint. Model C, preference-tuned, is evaluated against its supervised parent.
- Ranking by average judge score: C 4.2, B 3.9, A 3.4. A fourth model, D, could only be scored alone (2.9); it is listed apart as not comparable.
- C leads B by 0.3 on 30 questions — a modest gap. The per-question view shows C ahead on most billing questions and tied elsewhere, so the lead is broad rather than from one or two questions.
- Opening the questions where C scored lowest shows what even the best model still gets wrong: two answers name a report setting that was renamed. That is the next round of data.
Settings
The shared question set every model answers. Choose one that is new to all of them: usually the held-out test questions of the prep they were trained on.
One judge model for the whole comparison. It should come from a family that none of the compared models, or their bases, belongs to.
Common mistakes
- Ranking models from separate evaluations that used different questions or judges.
- Using held-out questions from one prep for a model trained on another, which may have seen them.
- Ranking a model scored alone against models judged side by side.
- Letting one of the contestants, or its base, act as the judge.
- Picking a winner from a few tenths of a point without reading the per-question answers.
References and further reading
External sources open in a new tab.
- 01Holistic Evaluation of Language ModelsLiang et al., 2022 · arXiv · arxiv.orgComparing many models under the same scenarios and metrics, so results line up.
- 02Chatbot Arena: An Open Platform for Evaluating LLMs by Human PreferenceChiang et al., 2024 · arXiv · arxiv.orgRanking many models from pairwise judgements on shared prompts.
- 03LLM Evaluators Recognize and Favor Their Own GenerationsPanickssery et al., 2024 · arXiv · arxiv.orgWhy the judge must not be one of the contestants.
- 04Adding Error Bars to Evals: A Statistical Approach to Language Model EvaluationsMiller, 2024 · arXiv · arxiv.orgWhy a small gap between two models on few questions may be noise.