Coming soonModel Training is launching soon in Scalata Enterprise. Teach a model to answer the way your team does.See what's coming

Resources/Model training/What happens during training

Concept

Open-weight models

An open-weight model is one whose trained weights are published for anyone to download, so it can be fine-tuned and run on hardware you choose. Its licence sets what you may do with it, and which release you start from matters as much as which family.

Chapter 64 min read4 references

What it is

A closed model, such as Gemini or GPT, lives only on its provider's servers. You can fine-tune it through the provider, but you can never download it, so it can't be trained on your own machine. An open-weight model is published as weight files anyone can download: Qwen, Llama, Mistral, Phi and Gemma, Google's open family, among others. Open weights are not the same as open source: the training data and code are often not published, and the licence sets what you may do with the model.

The trade is control against convenience. With open weights you choose the hardware, keep the trained model, and can run it wherever you like, including fully offline. With a closed model the provider handles the hardware, but the tuned model stays on their servers and can only be used through their service.

What to check before you pick one

Example 7B instruct modelopen-weight · text in, text out✓Open weightsweight files can be downloaded✓Formatsafetensors✓PrecisionBF16, not FP8 or MXFP4✓Size7B · about 17 GB to train with LoRA✓Releaseinstruct, ready for chat dataiLicenceread it; it sets what you may doiGatedaccept once on Hugging Face
What to check on a model's page before fine-tuning it, for an illustrative 7B model. The first five decide whether it is a sensible starting point at all; licence and gating are terms you have to read and accept yourself.
  • Size, which sets the memory a machine needs to train it.
  • Licence. Many models use permissive licences such as Apache 2.0; others have their own terms, such as Llama's community licence or Gemma's terms of use. Read them before using a model commercially.
  • Gating, whether Hugging Face requires you to accept terms before downloading.
  • Format. Prefer weights in safetensors, a file format that holds only numbers, so loading it can't run hidden code. Older pickle-based checkpoints can execute code when they are loaded.
  • Architecture and tooling. A brand-new architecture can take a while to be supported by common training libraries; a well-established family is the safer start.

Reading a release

A family usually publishes several releases of each size. A base model is trained only to continue text; an instruct (chat) model is tuned further to follow instructions; and there are often pre-quantised copies, in formats such as FP8, MXFP4, GGUF or AWQ, made to run cheaply. The number in the name, such as 7B, counts parameters in billions and sets how much memory training needs.

Full precision only

Some models are published already compressed, in 8-bit (FP8) or 4-bit (MXFP4) formats, to make them cheaper to run. Training needs the full-precision version, because the small changes training makes to each weight get lost in such coarse numbers. Look for the version in BF16, for example Ministral 3's versions ending in -BF16. QLoRA compresses a full-precision model itself, so it starts from the BF16 version too.

Picking a release to fine-tune (illustrative)
  1. A 7B instruct model in BF16 safetensors, Apache 2.0: a good starting point. LoRA needs about 2 × 7 + 3 ≈ 17 GB to train it.
  2. The same model's FP8 release: cheaper to run, but not a starting point for training. Use the BF16 version, or let QLoRA quantise it to 4 bits, about 8 GB to train.
  3. A gated 8B model under a community licence: usable once you accept its terms on Hugging Face, but read the licence for limits on commercial use before you build on it.

Gated models

Some publishers, including Meta for Llama and Google for Gemma 3, gate their models: you accept the licence once on the model's Hugging Face page before you can download it. Access belongs to your Hugging Face account, so any tool that downloads the model for you needs to act on behalf of an account that has accepted the terms.

Image-capable models

Some models, such as Gemma 3 and 4, Qwen3.5 and Ministral 3, can also read images. Fine-tuning them on text-only data trains the language side; their image abilities usually stay much as they were, though heavy training on text alone can weaken them a little. If images matter for your task, include image examples, or check image behaviour after training.

Choosing a base model

  • Fit first. Pick from the models your hardware can train: the memory page's rule of thumb gives a quick answer.
  • Instruct versions for chat. Most families publish a raw base version and an instruct (chat) version. For training on conversations, the instruct version already follows the chat format and needs fewer examples to behave well.
  • Smaller is often enough. A small model fine-tuned on a narrow task can match a much larger general one on that task, and it trains and answers faster. Try a small model first and move up only if the evaluation shows it falls short.
  • Languages and domain. Families differ in how well they handle languages other than English, code or long documents. A quick test chat with the base model shows where it starts from.

Common mistakes

  • Looking for Gemini or GPT weights to download. They are closed; Gemma is Google's open-weight family.
  • Starting from a model's FP8 or MXFP4 release instead of its BF16 version.
  • Forgetting to accept a gated model's licence on Hugging Face before the first run.
  • Assuming open weights means any use is allowed. The licence decides.
  • Picking a large model before checking whether it will fit in memory to train.

References and further reading

External sources open in a new tab.

  1. 01Gated modelsHugging Face Hub docs · huggingface.coHow gating works and how access is requested and granted.
  2. 02LicensesHugging Face Hub docs · huggingface.co
  3. 03SafetensorsHugging Face docs · huggingface.coThe weight format that stores tensors safely, without executable code.
  4. 04Downloading modelsHugging Face Hub docs · huggingface.co
ConceptPre-training and the base modelHow a language model learns language in the first place, what the base model you start from already knows, and why fine-tuning adjusts it rather than rebuilding it.Best practiceChoosing a training methodThe right training method depends mostly on the data you already have and the change you want. Chats teach answers, pairs teach preferences, scores teach checkable skills, documents teach knowledge, and a big model can teach a smaller one.Best practiceWill the model fit? Memory for trainingTraining needs room for the model's weights, its gradients and the optimizer's state, plus working memory. A simple rule of thumb, about 2 GB per billion parameters for LoRA and 12 GB for full fine-tuning, tells you before you start whether a model will fit your machine.MethodQLoRA (4-bit LoRA)QLoRA trains an ordinary LoRA adapter on a base model stored in 4 bits instead of 16. Training memory drops to roughly a third, so the same machine can fine-tune a model about three times larger, at some cost in speed.MethodLoRA and adapters (PEFT)LoRA freezes the base model and trains two small matrices beside selected layers, whose product is added to the frozen weights. It cuts trainable weights and memory dramatically and, configured well, matches full fine-tuning on typical fine-tuning datasets.ConceptPublic datasets from Hugging FaceWhat a public Hugging Face dataset holds, the common formats such datasets come in, and why some datasets become chats, some become documents and some can't be used for training at all.

Turn weeks of expert work into minutes

See how Scalata fits your team, your data, and your controls.

Contact Sales