Resources / Guide

Model training, explained

Everything behind fine-tuning a language model on your own data, one concept at a time. Start anywhere: each card opens a full page with the mechanism, a worked example, the settings, common mistakes and the primary sources.

9 chapters68 concepts
01

The big picture

Fine-tuning does not build a model from scratch. It nudges an existing model with many examples of the behaviour you want.

  1. 01Sessionspick chats
  2. 02Datasetcollect examples
  3. 03Providerwhere it trains
  4. 04Data prepclean + convert
  5. 05Trainrun the job
  6. 06Evaluateprove it improved
  7. 07Test chattry it yourself
  8. 08Useput it to work
The training pipeline. Each stage hands a frozen, inspectable artifact to the next, so any run can be traced back to the exact chats it learned from.
02

Tokens and context

Models read and write tokens, not characters or words. Almost every limit and every price in training is counted in them.

03

From chats to training examples

A chat becomes training data by deciding which of its replies the model should learn to write.

04

Datasets

A frozen, named collection of examples — from real chats, generated chats, public datasets or chat feedback — and the single biggest factor in how good the trained model turns out.

05

Data preparation

Data prep turns dataset examples into the exact training files a provider expects: cleaned, redacted, trimmed and split into training, validation and test sets.

Pipeline stepCleaningCleaning removes the replies a model should never learn from — errors, refusals, interface markup, duplicates and canned answers — and fixes the structure so every record is valid for training.Read in depth →Pipeline stepPersonal data and secrets (PII)Before training, emails, phone numbers, card numbers, IBANs, national IDs and API keys are detected and, by default, replaced with placeholders, because a fine-tuned model can repeat its training text word for word.Read in depth →SettingToken limits and trimmingEvery provider caps how many tokens one training example may contain. Long conversations over the cap are either trimmed — oldest exchanges removed first — or dropped.Read in depth →Pipeline stepValidation splitA share of conversations is held out of training and used to watch the loss on data the model does not learn from. It is split by whole conversation, so no part of a held-out chat leaks into training — and it is not the same as the test set kept back for evaluation.Read in depth →SettingSystem promptThe system prompt is a standing instruction placed before every conversation. Added at training time, the model learns its behaviour in the presence of that exact prompt, so it should stay the same when the model is used.Read in depth →ConceptJSONL training filesJSONL (JSON Lines) is a text file with one complete JSON object per line. It is the standard format for fine-tuning data: one line is one training record.Read in depth →
06

What happens during training

Training is a loop: predict, measure the error, work out how each weight should change, change it a little — and repeat thousands of times.

ConceptThe training loopFine-tuning is one short procedure repeated thousands of times: predict the target reply, measure how wrong the prediction was, work out which way each weight should move, and move it a little.Read in depth →ConceptEpoch, batch and stepA batch is the group of examples behind one weight update, a step is one update, and an epoch is one full pass over the training data. Together they decide how often the model changes and how many times it sees each example.Read in depth →MethodFull fine-tuningFull fine-tuning updates every weight in the model. It has the most room to change behaviour, but needs many times the model's own size in GPU memory and is the most likely to erode skills the base model had.Read in depth →MethodLoRA and adapters (PEFT)LoRA freezes the base model and trains two small matrices beside selected layers, whose product is added to the frozen weights. It cuts trainable weights and memory dramatically and, configured well, matches full fine-tuning on typical fine-tuning datasets.Read in depth →MethodQLoRA (4-bit LoRA)QLoRA trains an ordinary LoRA adapter on a base model stored in 4 bits instead of 16. Training memory drops to roughly a third, so the same machine can fine-tune a model about three times larger, at some cost in speed.Read in depth →ConceptGradient checkpointingGradient checkpointing saves memory by keeping only some of the intermediate results from the forward pass and recomputing the rest when the backward pass needs them. It uses much less memory and makes training about 20–30% slower.Read in depth →Best practiceWill the model fit? Memory for trainingTraining needs room for the model's weights, its gradients and the optimizer's state, plus working memory. A simple rule of thumb, about 2 GB per billion parameters for LoRA and 12 GB for full fine-tuning, tells you before you start whether a model will fit your machine.Read in depth →ConceptOpen-weight modelsAn open-weight model is one whose trained weights are published for anyone to download, so it can be fine-tuned and run on hardware you choose. Its licence sets what you may do with it, and which release you start from matters as much as which family.Read in depth →SettingHyperparametersHyperparameters are the training settings you choose, such as epochs, learning rate, batch size and adapter rank, as opposed to the weights the model learns. Auto picks them from your data and makes a good first run; this page explains each one so you can tune from there.Read in depth →SettingLearning rateThe learning rate sets how far the weights move on each update. It is the most sensitive training setting: too high overshoots and can erase skills, too low barely changes the model. A schedule usually warms it up and then decays it over the run.Read in depth →MetricReading the loss curvesTraining loss shows how well the model fits the examples it learns from; validation loss shows how well it predicts held-out examples it never trained on. The gap between the two curves is the most useful signal a training run gives you.Read in depth →MetricOverfitting and underfittingOverfitting is when a model memorises its training examples instead of learning the pattern behind them, so it does well on those examples and worse on new ones. Underfitting is the opposite: it hasn't learned the pattern at all.Read in depth →ConceptHow training is billedMost managed fine-tuning is billed per token processed: tokens in the training file times epochs times a price. Jobs on rented GPUs are billed by time instead. Using the model afterwards is a separate cost.Read in depth →
07

Training methods

The methods differ in what signal the model learns from — and therefore in what data you need and what each can and cannot teach.

The four methods differ in one thing: the signal the model learns from.

Preference tuning

MethodPreference tuning (DPO)Direct Preference Optimization trains a model on pairs of answers, one better and one worse, so it learns to prefer the better kind, without the reward model and reinforcement-learning loop that classic RLHF needs.Read in depth →SettingBeta (β) in DPOBeta sets how strongly preference tuning holds the model to its starting point: lower values let the pairs change it more, higher values keep it closer to the original.Read in depth →MetricReading the DPO lossThe DPO loss measures how clearly the model now prefers the better answer in each pair, relative to where it started. It begins at ln 2 ≈ 0.693 and falls toward 0 as the preference is learned.Read in depth →Best practiceWhere good preference pairs come fromA preference pair is one prompt with a better and a worse answer. How well DPO works depends mostly on whether the difference between the two answers is the thing you actually want to change.Read in depth →MetricPreference accuracyPreference accuracy is the share of pairs for which the model now gives the preferred answer a higher implicit reward than the non-preferred one, relative to where it started. It begins near 50% and rises as the preference is learned.Read in depth →Best practiceThe standard recipe: SFT, then DPOPreference tuning sharpens answers a model can already give, so the usual recipe has two steps: supervised fine-tuning to teach the task, then DPO continued from that model and evaluated against it.Read in depth →

Reinforcement fine-tuning

MethodReinforcement fine-tuning (RFT)Reinforcement fine-tuning trains a model on scores instead of example answers: it answers each prompt several times, a grader scores every attempt, and the model leans toward what scored better.Read in depth →MethodGRPO, step by stepGroup Relative Policy Optimization scores several answers to the same prompt and reinforces those above the group's average. It replaces PPO's separate value model with that simple comparison.Read in depth →SettingGraders: how answers are scoredA grader turns an answer into a number between 0 and 1. In reinforcement fine-tuning it is the only definition of “good” the model ever sees, so the choice of grader decides what the model learns.Read in depth →SettingRFT settingsThe six settings of a reinforcement fine-tuning run: what scores the answers, how many answers the model writes and how long they may be, and how big and how anchored each update is.Read in depth →MetricRFT metrics to watchReward, clipped ratio, reward spread and completion length: the four charts that tell you whether a reinforcement run is learning, starved of signal, or gaming its grader.Read in depth →MetricHeld-out checks in reinforcement fine-tuningWhen a provider runs reinforcement fine-tuning on a hosted reasoning model, it re-asks a few held-out validation questions every few steps and grades the answers with the same grader. Each question becomes a timeline showing whether training carries over to questions the model never trained on.Read in depth →Best practiceReward hackingReward hacking is when a model raises its score without getting better at the task, by exploiting what the grader fails to check. It is the main way reinforcement fine-tuning goes wrong.Read in depth →
08

Evaluation

Loss curves show how well the model fits. Evaluation shows whether its answers are actually better.

ConceptBase vs tuned, side by sideAn evaluation asks the starting model and the trained model the same held-out questions and compares their answers, so you can see whether training actually made the answers better.Read in depth →ConceptTest questions: why “held out” mattersA slice of your conversations that data prep sets aside before anything else, so an evaluation can ask the trained model questions it has never seen. It is the only clean way to tell real improvement from memorisation.Read in depth →MethodLLM-as-a-judgeA separate language model reads the question, the reference answer and two candidate answers, scores both and picks a winner. Set up blind, with shuffled order and a different model family, it agrees with human raters about as often as humans agree with each other.Read in depth →MetricReading evaluation resultsHow to read win rate, score difference and the secondary signals of an evaluation — and how to tell a real improvement from noise when the number of questions is small.Read in depth →ConceptCheckpoints: when the base model is goneCheckpoints are copies of the model the provider saves partway through training. When the base model can no longer be deployed, comparing the final model with an earlier checkpoint shows whether the later training steps helped.Read in depth →MethodScoring a model on its ownInstead of comparing two answers, the judge reads one answer next to the conversation and your reference reply, and scores it 1–5 per criterion. The result is an absolute quality score: no wins, and no measure of improvement.Read in depth →Best practiceComparing several modelsTo choose between models — different providers, methods or base sizes — they must answer the same questions and be scored by the same judge. A comparison runs one evaluation per model on a shared question set and ranks them by average judge score.Read in depth →Best practiceAnnotation queues: checking the judge by handPeople review model answers one at a time — scoring them, picking the better one or writing a correction — to measure how far the LLM judge can be trusted, and to turn their fixes into training data.Read in depth →
09

Trying and serving a model

A trained model has to be deployed before it can answer — and a dedicated deployment costs money for every hour it exists.

10

Glossary

Every concept, A to Z. Each opens its full page.

Turn weeks of expert work into minutes

See how Scalata fits your team, your data, and your controls.