Coming soonModel Training is launching soon in Scalata Enterprise. Teach a model to answer the way your team does.See what's coming

Resources/Model training/The big picture

Concept

Pre-training and the base model

How a language model learns language in the first place, what the base model you start from already knows, and why fine-tuning adjusts it rather than rebuilding it.

Chapter 15 min read5 references

What pre-training is

A large language model starts as billions of numbers — its weights, or parameters — set at random. Pre-training adjusts those numbers by showing the model enormous amounts of text (web pages, books, code, reference works) and asking it, over and over, one question: given the text so far, what comes next? Each wrong guess nudges the weights a little. After trillions of guesses the model has absorbed grammar, facts, styles, reasoning patterns and a great deal of world knowledge, because all of those help predict the next word.

This objective is called next-token prediction, or causal language modelling. Nobody has to label the data: the text is its own answer key, which is why pre-training can use so much of it.

The scale involved

Pre-training is by far the most expensive stage in a model's life. GPT-3, described in 2020, had 175 billion parameters and was trained on about 300 billion tokens. DeepMind's Chinchilla study (2022) then showed that, for a fixed compute budget, models should see much more text per parameter: their 70-billion-parameter Chinchilla, trained on 1.4 trillion tokens, outperformed considerably larger models. Current open models are trained on many trillions of tokens, on thousands of GPUs, for weeks.

That is why nobody fine-tuning a model for their own use repeats this stage. It is also why the starting model matters so much: its knowledge, its languages and its reasoning ability set the ceiling for what any later training can achieve.

Pre-trainingtrillions of tokens of public textmodel maker · weeks on thousands of GPUs
→raw base checkpointInstruction tuningexample conversations and feedbackmodel maker · learns to follow requests
→the base model you start fromYour fine-tuninghundreds to thousands of your examplesyou · minutes to hours
Three stages, very different in scale. Almost all of a model's knowledge comes from the first; your fine-tuning adjusts the result of the second.

From raw model to the model you start from

A model fresh out of pre-training is good at continuing text but not at following instructions. Ask it a question and it may reply with three more questions, because that is what a lot of text on the web looks like. Model makers therefore train it further — instruction tuning (supervised fine-tuning on example conversations), usually followed by preference tuning on human or AI feedback — to turn it into an assistant. OpenAI's InstructGPT work showed how much this matters: people preferred the answers of a 1.3-billion-parameter instruction-tuned model to those of the 175-billion-parameter GPT-3, despite it having over 100 times fewer parameters.

Open model families are often published in both forms: a raw “base” checkpoint and an “instruct” or “chat” version. In Scalata, and in this guide, base model simply means the model a training run starts from. In practice that is almost always an instruction-tuned model, so your examples only need to teach what is specific to you.

What fine-tuning can and cannot change

Fine-tuning uses the same machinery as pre-training — predict the next token, measure the error, adjust the weights — but on hundreds or thousands of your examples instead of trillions of tokens. At that scale it is very effective at reshaping behaviour, and much less effective at adding knowledge. Research suggests that models learn genuinely new facts from fine-tuning slowly, and that pushing them to can make them more inclined to invent answers.

  • Changes well: tone and style, output format, length, domain phrasing, consistent handling of recurring requests, getting a small model to do one job like a larger one.
  • Changes poorly: large bodies of new facts, information that changes often, and abilities the base model fundamentally lacks — a small model will not learn advanced reasoning from a few hundred chats.
  • Does not change: the model's architecture, its tokenizer and its context window.
What a base model already brings
  1. You want an assistant that answers questions about your invoicing product in your support team's tone.
  2. The base model already knows English, how invoices work in general, how to be polite and how to write numbered steps — from pre-training and instruction tuning.
  3. It does not know your menu names, your policies, or your team's habit of ending with a link to the help centre.
  4. A few hundred good support chats can teach the tone and the format. The menu names and policies are better supplied at request time, through retrieval or the system prompt, so they stay correct when they change.

Choosing a base model

Everything you train sits on top of the base model, so choose it deliberately:

  • Capability on your task. Try the base model on real questions first. Fine-tuning sharpens what is there; it rarely creates ability from nothing.
  • Size against cost. Larger models know more but cost more to train and to run. A common goal is the smallest model that does the job well once fine-tuned.
  • Languages. If your users write in German or Japanese, check that the model handles those languages well; pre-training data is usually dominated by English.
  • Limits. The context window and the per-example training limit come with the model and do not change with training.

Common mistakes

  • Expecting fine-tuning to teach a large, changing body of facts such as a full product catalogue or price list.
  • Starting from a model that cannot do the task at all, and expecting a few hundred examples to fix it.
  • Picking a raw pre-trained checkpoint when an instruction-tuned version of the same open model was intended.
  • Not testing the base model first, so there is no baseline to compare the trained model with.

References and further reading

External sources open in a new tab.

  1. 01Language Models are Few-Shot LearnersBrown et al., 2020 · arXiv · arxiv.orgThe GPT-3 paper: what pre-training at scale alone can do.
  2. 02Training Compute-Optimal Large Language ModelsHoffmann et al., 2022 · arXiv · arxiv.orgThe Chinchilla study on how much text a model of a given size should be trained on.
  3. 03Training language models to follow instructions with human feedbackOuyang et al., 2022 · arXiv · arxiv.orgInstructGPT: how a pre-trained model becomes an instruction-following assistant.
  4. 04How do Transformers work?Hugging Face LLM Course · huggingface.coA gentle introduction to pre-training, transfer learning and fine-tuning.
  5. 05Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?Gekhman et al., 2024 · arXiv · arxiv.orgEvidence that new facts are learned slowly in fine-tuning and can increase hallucination.
ConceptPrompting, retrieval or fine-tuningThree ways to make a model fit your use: what each one changes, what it costs, and how to choose between them or combine them.ConceptTokensThe chunks of text a model actually reads and writes — how they are made, why counts differ between models, and why every limit and price in training is measured in them.MethodSupervised fine-tuning (SFT)Supervised fine-tuning teaches a model by example: it is shown a conversation and the reply you want, and its weights are nudged so that reply becomes more likely, token by token.MethodContinuing from a trained modelA training run can start from the weights of one of your earlier runs instead of the base model, so the model keeps what it learned and adds what the new data teaches. This is how multi-step recipes such as supervised then preference tuning are built.ConceptBase vs tuned, side by sideAn evaluation asks the starting model and the trained model the same held-out questions and compares their answers, so you can see whether training actually made the answers better.MethodLoRA and adapters (PEFT)LoRA freezes the base model and trains two small matrices beside selected layers, whose product is added to the frozen weights. It cuts trainable weights and memory dramatically and, configured well, matches full fine-tuning on typical fine-tuning datasets.

Turn weeks of expert work into minutes

See how Scalata fits your team, your data, and your controls.

Contact Sales