Coming soonModel Training is launching soon in Scalata Enterprise. Teach a model to answer the way your team does.See what's coming

Resources/Model training/Evaluation

Metric

Reading evaluation results

How to read win rate, score difference and the secondary signals of an evaluation — and how to tell a real improvement from noise when the number of questions is small.

Chapter 84 min read4 references

The numbers

definitions
win rate          = tuned wins ÷ questions judged
score difference  = mean(tuned criteria score) − mean(base criteria score)

decided share     = tuned wins ÷ (tuned wins + base wins)
  • Win rate. The share of judged questions the tuned model won. A tie is a win for neither side, so with many ties a 45% win rate can still be a clear improvement — check how often the base model won.
  • Score difference. The average 1–5 criteria score of the tuned answers minus that of the base answers, overall and per criterion. It shows how much better or worse, and where: correctness, helpfulness, tone or format.
  • Similarity to reference. How close the answers are to your original replies. Rising similarity means the model picked up your style; it does not by itself mean answers are better.
  • Length and latency. If the tuned side wins and its answers got much longer, suspect the judge's verbosity bias. Latency matters for real use: a smaller tuned model that ties a larger one may still be the better choice.

Signal or noise?

Every evaluation is a sample: a few dozen or hundred questions standing in for all the questions users will ever ask. Run the same comparison on a different sample and the win rate moves. The question is not “is it above 50%?” but “is it far enough above 50% that chance is an unlikely explanation?”

0%25%50%75%100%← BASE BETTERTUNED BETTER →10 questions30 questions100 questions300 questions
interval crosses 50%: inconclusiveinterval clears 50%: a real difference
The same 60% win rate at four evaluation sizes, with its 95% confidence interval. At 10 or 30 questions the interval reaches well below 50%, so the result is compatible with no improvement at all; only around 100 questions does it clear the line.

A 95% confidence interval is the range of win rates consistent with what you observed. Its width is set mainly by the number of questions, and roughly halves each time the number of questions quadruples. With 10 questions a 6-to-4 split is entirely consistent with two equal models; with 100 questions a 60/40 split just clears the line; with 300 it is unambiguous.

Ties need care. If the judge calls many ties, compare wins with losses among the decided questions only. That is the share that carries information about which model is better.

Worked example: 100 questions, 18 ties
  1. Result: tuned won 52, base won 30, 18 ties. Win rate 52% — it looks like a coin flip.
  2. Among the 82 decided questions the tuned model won 52, or 63%. The 95% interval for that is about 53–73%: entirely above 50%. This is a real improvement.
  3. Score difference: +0.6 on correctness, +0.4 on format, −0.1 on tone. The gain is in substance, with tone about the same.
  4. Tuned answers average 95 words against 160 for base, so length is not what is winning.
  5. Reading the 30 base wins: nine reasons mention an invented setting name. That is the next dataset fix, and nine pairs to train on.

Held-out vs training questions

Results on the held-out test set are the ones to trust. Validation questions are a step weaker: the model didn't learn from them, but the provider saw them during training. When a dataset was too small for either, the only questions available are ones the tuned model trained on. It may simply be recalling your reference reply, so it looks better than it will on new questions. Treat such a result as an upper bound: if the tuned model doesn't win even on familiar questions, it won't on new ones; if it does, you still don't know by how much.

Other kinds of result

  • Against an earlier checkpoint: the result says whether the later training steps helped. The last checkpoint is usually identical to the final model, so comparing with it gives mostly ties.
  • Scored alone: an absolute 1–5 score with no wins. The same judge gives noticeably lower scores to an answer read on its own than to one read next to another, so a model scored alone can't be ranked against models judged side by side; a model comparison lists it separately.
  • A close or surprising result: before acting on it, check the judge. People reviewing a sample blind in an annotation queue show how often they agree with it, and their corrections become training data.

Turning results into action

What you seeLikely meaningNext step
Clear win, interval above 50%, reasons about correctnessTraining workedKeep it; archive weaker variants
Narrow win on a few questionsA hint, not proofEvaluate on more questions before acting
Mostly tiesLittle changed — common for small models or light trainingMore or better supervised data
Tuned wins but answers got much longerPossibly verbosity biasRead the reasons; compare correctness scores
Tuned loses with reasons like “invents details”Learned the style without the knowledgeMore data, smaller steps, or pairs that penalise invention
The judge's reasons don't match your own readingThe judge may be unreliable on this taskReview a sample by hand and measure agreement with the judge

Common mistakes

  • Calling 6 wins out of 10 an improvement.
  • Reading a low win rate as a loss without checking how many questions were ties.
  • Treating a score on training questions as a forecast for new ones.
  • Celebrating a win that came with much longer answers, without reading why the judge preferred them.
  • Comparing win rates of two runs evaluated on different question sets or with different judges.

References and further reading

External sources open in a new tab.

  1. 01Adding Error Bars to Evals: A Statistical Approach to Language Model EvaluationsMiller, 2024 · arXiv · arxiv.orgConfidence intervals, paired comparisons and how many questions you need.
  2. 02Binomial proportion confidence intervalWikipedia · en.wikipedia.orgThe maths behind an interval on a win rate, including the Wilson interval used in the figure.
  3. 03Chatbot Arena: An Open Platform for Evaluating LLMs by Human PreferenceChiang et al., 2024 · arXiv · arxiv.orgPairwise win rates at very large scale, with their uncertainty.
  4. 04Length-Controlled AlpacaEval: A Simple Way to Debias Automatic EvaluatorsDubois et al., 2024 · arXiv · arxiv.orgWhy length belongs next to the win rate.
ConceptBase vs tuned, side by sideAn evaluation asks the starting model and the trained model the same held-out questions and compares their answers, so you can see whether training actually made the answers better.ConceptTest questions: why “held out” mattersA slice of your conversations that data prep sets aside before anything else, so an evaluation can ask the trained model questions it has never seen. It is the only clean way to tell real improvement from memorisation.MethodLLM-as-a-judgeA separate language model reads the question, the reference answer and two candidate answers, scores both and picks a winner. Set up blind, with shuffled order and a different model family, it agrees with human raters about as often as humans agree with each other.Best practiceAnnotation queues: checking the judge by handPeople review model answers one at a time — scoring them, picking the better one or writing a correction — to measure how far the LLM judge can be trusted, and to turn their fixes into training data.Best practiceComparing several modelsTo choose between models — different providers, methods or base sizes — they must answer the same questions and be scored by the same judge. A comparison runs one evaluation per model on a shared question set and ranks them by average judge score.MethodScoring a model on its ownInstead of comparing two answers, the judge reads one answer next to the conversation and your reference reply, and scores it 1–5 per criterion. The result is an absolute quality score: no wins, and no measure of improvement.

Turn weeks of expert work into minutes

See how Scalata fits your team, your data, and your controls.

Contact Sales