Coming soonModel Training is launching soon in Scalata Enterprise. Teach a model to answer the way your team does.See what's coming

Resources/Model training/Trying and serving a model

Concept

Inference and deployment

Inference is using a model to generate answers. A trained model must first be deployed — its weights loaded onto hardware behind an endpoint — which can take minutes, and then costs money per token or per hour.

Chapter 93 min read4 references

What it is

Inference is everything that happens after training: a prompt goes in, the model produces an answer. Nothing is learned; the weights stay fixed. Every test chat, every evaluation answer, every judge verdict and every reply in production is inference.

How it works

A model generates text one token at a time. First it reads the whole prompt in one pass (called prefill), which is fast because all the tokens can be processed in parallel. Then it decodes: predicts the next token, appends it, and repeats until it produces an end token or reaches a length limit. Each new token needs another pass through the model, which is why long answers take longer than long questions.

  • Time to first token is mostly prefill: longer prompts, longer wait before anything appears.
  • Tokens per second is decode speed: it sets how fast the answer streams.
  • Sampling settings such as temperature decide how the next token is picked from the model's probabilities. Low values make answers more predictable; high values more varied.
  • Batching: a server answers many requests at once to keep the GPU busy. Serving systems such as vLLM manage memory so well that they raised throughput 2–4× over earlier systems at the same latency.

Deployment

Training produces a set of weights stored at the provider. To answer, those weights must be deployed: loaded into the memory of one or more GPUs, behind an endpoint that accepts requests. The memory needed is roughly the number of parameters times the bytes used per parameter:

why big models need big hardware
memory for weights ≈ parameters × bytes per parameter

7B model  × 2 bytes (16-bit)  ≈ 14 GB
70B model × 2 bytes (16-bit)  ≈ 140 GB  → several GPUs

Allocating that hardware, downloading the weights and loading them into GPU memory is the cold start. For managed models it is usually quick or invisible; for open models on dedicated GPUs it often takes 10 to 40 minutes. Once loaded, the deployment stays warm and answers in seconds — for as long as it exists.

  1. 01Choose durationhourly cost shown up front
  2. 02Deployingload weights · cold start
  3. 03Livechat · the meter runs
  4. 04Deletingbilling stops
A dedicated deployment is billed for every hour it exists, used or not. The lifecycle ends by deleting it.
Worked example: testing a tuned open model
  1. A supervised run on an 8-billion-parameter open model has finished. In 16-bit precision its weights take about 16 GB, so it needs a GPU with more memory than that, plus room for the requests being processed.
  2. You start a test chat for two hours. The hourly price is shown before anything is created.
  3. Deploying: the GPU is allocated and the weights load. This takes about 15 minutes; billing for the deployment has started.
  4. Live: each question is answered in a second or two. You try ten real questions from last week.
  5. You finish early and end the session. Deleting removes the deployment and billing stops. Cost: the time the deployment existed × the hourly price, however few questions you asked.

Common mistakes

  • Leaving a dedicated deployment running overnight “in case” — it bills by the hour, used or not.
  • Judging a model on its first answer after a cold start, or on response time while it is still loading.
  • Testing with different sampling settings or a different system prompt from the ones used in evaluation.
  • Expecting a fine-tuned model to know things that happened after training: inference doesn't learn.

References and further reading

External sources open in a new tab.

  1. 01Efficient Memory Management for Large Language Model Serving with PagedAttentionKwon et al., 2023 · arXiv · arxiv.orgThe vLLM paper: how serving systems batch requests and manage GPU memory.
  2. 02Text Generation InferenceHugging Face docs · huggingface.coA production serving stack for open models, with its features explained.
  3. 03Inference EndpointsHugging Face docs · huggingface.coWhat deploying a model behind a managed endpoint involves.
  4. 04Cold start (computing)Wikipedia · en.wikipedia.orgThe general idea of a cold start.
ConceptServerless vs dedicatedServerless serving bills per token on shared capacity with no idle cost; a dedicated deployment reserves GPUs for your model and bills for every hour it exists. Which is cheaper depends on how much you use it.Best practiceWill the model fit? Memory for trainingTraining needs room for the model's weights, its gradients and the optimizer's state, plus working memory. A simple rule of thumb, about 2 GB per billion parameters for LoRA and 12 GB for full fine-tuning, tells you before you start whether a model will fit your machine.ConceptWhy tokens drive costHow token counts turn into the price of training and of using a model, with worked numbers, and which levers actually lower the bill.SettingSystem promptThe system prompt is a standing instruction placed before every conversation. Added at training time, the model learns its behaviour in the presence of that exact prompt, so it should stay the same when the model is used.ConceptBase vs tuned, side by sideAn evaluation asks the starting model and the trained model the same held-out questions and compares their answers, so you can see whether training actually made the answers better.ConceptHow training is billedMost managed fine-tuning is billed per token processed: tokens in the training file times epochs times a price. Jobs on rented GPUs are billed by time instead. Using the model afterwards is a separate cost.

Turn weeks of expert work into minutes

See how Scalata fits your team, your data, and your controls.

Contact Sales