Epoch, batch and step

Three units of training

Training data is fed to the model neither one example at a time nor all at once. It is cut into batches. The model processes a batch, averages the error over all of its target tokens, and updates its weights once. That single update is a step, also called an iteration. When every training example has been used once, one epoch is complete, and the data is usually shuffled before the next one, so batches mix differently each time.

Term	What it counts	Typical in fine-tuning
Batch	The examples processed together before one update	A handful to a few dozen examples; often Auto
Step	One weight update	Tens to thousands per run
Epoch	One full pass over all training examples	1–3 for supervised fine-tuning

24 examples with a batch size of 8. Each batch gives one weight update (a step), so one epoch is 3 steps and two epochs are 6. The data is shuffled every epoch: the highlighted example is seen once per epoch, in a different batch each time.

The arithmetic

Batches, steps and epochs

effective batch  = per-device batch × gradient accumulation × number of GPUs
steps per epoch  = ceil(training examples ÷ effective batch)
total steps      = steps per epoch × epochs
tokens processed = tokens in the training file × epochs

800 examples, batch size 8, 3 epochs

Steps per epoch: 800 ÷ 8 = 100 steps.
Total steps: 100 × 3 epochs = 300 steps. The model's weights are changed 300 times.
Times each example is seen: 3, once per epoch, each time in a different batch because the data is reshuffled.
A 5% warm-up: 0.05 × 300 = 15 steps of ramping up before the full learning rate is reached.
The same data with batch size 32: 800 ÷ 32 = 25 steps per epoch, 75 in total. Fewer, smoother updates, usually paired with a somewhat higher learning rate.
Tokens processed: if the training file holds 400,000 tokens, three epochs process 1,200,000, which is what per-token pricing bills.

When the numbers don't divide evenly, the last batch of an epoch is smaller: 810 examples in batches of 8 give 101 full batches and one of 2. Some training setups keep that partial batch, others drop it. Either way the effect on a run is negligible.

Why batch size matters

Each step's gradient is an estimate of the true direction downhill, made from the examples in that batch. A small batch gives a noisy estimate: individual examples pull the weights around, but there are many updates per epoch. A large batch averages more examples, so each step points more reliably downhill, but there are fewer steps, and each needs more GPU memory. Research on large-batch training shows diminishing returns: past a point that depends on the task, doubling the batch no longer halves the number of steps needed, and the extra compute is mostly wasted.

Gradient accumulation separates the batch the GPU computes from the batch the optimizer updates on. The GPU processes, say, 4 examples at a time and adds up the gradients of 8 such micro-batches before updating once: an effective batch of 32 at the memory cost of 4. With several GPUs, each processes its own micro-batch, so the effective batch is also multiplied by the number of GPUs.

Gradient accumulation

per-device batch 4  ×  accumulation 8  ×  1 GPU  =  effective batch 32
memory: as for 4 examples        updates: one per 32 examples

Why the number of epochs matters

An epoch is the natural unit for how much the model has seen. One epoch over a small dataset is often not enough to change behaviour noticeably; many epochs over the same small set let the model memorise individual replies. Supervised fine-tuning on a few hundred to a few thousand examples typically uses 1–3 epochs, and preference tuning 1–2. Small datasets can tolerate a few more epochs; very large ones often need just one.

Because epochs count repetitions and steps count updates, keep an eye on both. Three epochs over 100 examples with batch size 8 is only 39 steps, which can be too few for a learning-rate schedule to do its job. One epoch over 50,000 examples is 6,250 steps: plenty of updates, and no repetition at all.

Common mistakes

Raising epochs to make up for too little data. More passes over 40 examples teach those 40 replies, not the task.
Changing the batch size without revisiting the learning rate. Fewer, larger steps usually want a larger rate, and the reverse.
Copying a warm-up given in steps from another run. If the new run has far fewer steps, a fixed warm-up can swallow most of the training.
Counting messages instead of training records. Depending on the provider's format, a conversation with several targets can become several records.

References and further reading

External sources open in a new tab.