Reading the loss curves

What the curves show

While a job runs, the provider records the loss, the average cross-entropy on target tokens, over time. Two curves matter, and some providers add a third number.

Training loss is measured on the batches being trained on, usually at every step. It is noisy because every batch is different: a batch of short, formulaic replies scores lower than one of long, unusual replies.
Validation loss is measured on the held-out validation set every so often, every N steps or once per epoch, with no weight updates. It uses the same examples each time, so it is smoother, and it is the honest one: the model never learned from them.
Token accuracy, reported by some providers, is the share of target tokens the model's top guess got exactly right. It moves in the opposite direction to loss and is easier to read, but coarser.

Training loss keeps falling because the model can always memorise more. Validation loss is the honest one: when it turns upward, the model has started learning your examples rather than the pattern behind them.

What the numbers mean

Cross-entropy is measured with the natural logarithm. A loss of L per token means the model gave the correct next token a probability of about e to the power of −L, averaged in the geometric sense. That makes a few values easy to picture.

Loss per token	Probability of the right token	Roughly
2.3	About 0.10	Often unsure; many plausible continuations
1.0	About 0.37	Fairly confident on average
0.5	About 0.61	Predicts the style and wording well
0.1	About 0.90	Near word-for-word; on training data often a sign of memorising

The absolute level depends heavily on the data. Free-form, varied replies have a high floor, because many different answers are equally good; templated replies can go very low without anything being wrong. So compare curves within a run, and between runs on the same base model and data, rather than against a number from elsewhere. Different base models use different tokenizers, which makes their losses incomparable.

Patterns and what to do

What you see	Meaning	What to do
Both still falling at the end	Still learning	More epochs may help
Training falls, validation flattens	About right	Stop here
Training falls, validation rises	Overfitting: memorising your examples	Fewer epochs, a lower learning rate, more varied data
Neither falls much	Underfitting, or inconsistent data	Check data quality; raise epochs or learning rate slightly
Jagged training loss with spikes	Learning rate too high	Lower it; add warm-up
Training loss drops in steps at each epoch boundary	The model recognises examples from the previous epoch	An early sign of memorising; watch validation loss
Validation loss far below training loss from the start	The validation set is easier, or overlaps with training	Check the split for shared or near-duplicate conversations

Reading a three-epoch run (illustrative numbers)

Setup: 1,200 training conversations, 130 held out, batch size 8, 3 epochs: 150 steps per epoch, 450 in total.
Epoch 1: training loss falls from 1.6 to 0.9, validation from 1.5 to 0.95. Both falling: learning.
Epoch 2: training reaches 0.6; validation bottoms out at 0.85 around step 250.
Epoch 3: training keeps falling to 0.35, but validation climbs back to 1.0. The gap widens: the model is memorising.
Conclusion: the best model was near step 250, early in epoch 2. Rerun with 2 epochs, or use a checkpoint from that point if the provider keeps one, and evaluate it against the base model.

What loss can't tell you

Loss measures how likely the model finds your target replies, token by token. It doesn't measure whether an answer is correct, helpful or safe. A model can lower its loss by matching your phrasing and formatting while still getting facts wrong, and a model with a slightly higher validation loss may give better answers. Use the curves to choose epochs and learning rate and to spot problems; use an evaluation, comparing answers from the tuned and base models side by side, to decide whether the model is actually better.

Common mistakes

Picking the run with the lowest training loss. It is usually the most overfit.
Reacting to single noisy points in the training curve instead of the trend.
Comparing losses across different base models, tokenizers or datasets.
Treating a missing validation curve as fine. Without held-out data, overfitting is invisible.
Stopping at the curves. A good-looking curve still needs an evaluation of real answers.

References and further reading

External sources open in a new tab.