Resources/Model training/Tokens and context

Concept

Why tokens drive cost

How token counts turn into the price of training and of using a model, with worked numbers, and which levers actually lower the bill.

Chapter 23 min read4 references

What you pay for

Providers meter language-model work in tokens because the computation a model performs grows with the number of tokens it processes. Training reads every token of every record, once per epoch. Using a model reads every input token and generates every output token, one at a time.

Training · per-token billing
tokens in the training filecontext + targets, every record×epochs×price per training token=training cost
Using · per request
input tokens × input pricesystem prompt, history, documents, question+output tokens × output priceusually the higher price=cost of one request
Training is paid once per run and scales with epochs. Using a model is paid on every request, and the input is sent again each time.

Training cost

With per-token training — how most managed fine-tuning services bill — the formula is simple:

Per-token training
training cost = tokens in the training file × epochs × price per training token

Every token in the file counts, not only the target replies. The system prompt and the context of each record are read during training too, even though the model is graded only on the targets. A long system prompt repeated in every record, or long conversation histories, make training more expensive.

Training on rented GPUs is billed by time instead: runtime × the GPU's hourly price. Tokens still matter, because more tokens take longer, but model size, training method and hardware matter as much.

Estimating a training run
  1. A prepared dataset has 500 records averaging 800 tokens: 400,000 tokens.
  2. Three epochs means the model processes 1.2 million tokens.
  3. At an illustrative price of $3 per million training tokens, that is about $3.60.
  4. Halving a 200-token system prompt removes 50,000 tokens from the file — 150,000 fewer tokens processed over three epochs.

Usage cost

A trained model is served either per token on shared capacity, or on a dedicated deployment billed per hour whether it is used or not. Per-token pricing separates input tokens (everything sent: system prompt, history, retrieved documents, the question) from output tokens (the answer). Output tokens are usually several times more expensive, because they are generated one at a time.

Per-token use
cost per request = input tokens × input price  +  output tokens × output price

Because the input is sent again on every call, a long prompt is a recurring cost. This is one of the practical arguments for fine-tuning: behaviour learned into the weights no longer has to be explained in every request.

What a long prompt costs over a month
  1. An assistant handles 100,000 requests a month. Its instructions are 1,500 tokens; the question and history add another 500.
  2. That is 2,000 input tokens per request, or 200 million input tokens a month.
  3. After fine-tuning, the instructions shrink to 150 tokens: 650 input tokens per request, or 65 million a month — about a third.
  4. Whether that saves money depends on the fine-tuned model's price, which on some providers is higher than the base model's. Compare both before deciding.

What lowers the bill

  • Fewer, better examples. A few hundred well-chosen records often beat thousands of noisy ones, at a fraction of the training cost.
  • Fewer epochs. Cost grows in step with epochs, and more epochs are not always better.
  • Shorter records. Trimming old history and keeping the system prompt short cuts both training and usage tokens.
  • A smaller model. Prices per token fall steeply with model size.
  • Per-token serving where possible. A dedicated deployment costs money every hour it exists.

Common mistakes

  • Counting only the target replies when estimating training tokens.
  • Forgetting to multiply by the number of epochs.
  • Comparing models on input price alone when output tokens usually cost more.
  • Leaving a dedicated test deployment running after you have finished with it.
  • Treating an estimate as a quote.

References and further reading

External sources open in a new tab.

  1. 01PricingClaude Platform Docs · platform.claude.comAn example of separate input and output token prices.
  2. 02Amazon Bedrock PricingAmazon Web Services · aws.amazon.comPer-token inference, customisation and hourly hosting prices on one page.
  3. 03Token countingClaude Platform Docs · platform.claude.comCounting tokens before sending a request.
  4. 04Understand and count tokensGemini API docs · ai.google.devHow another provider counts and bills tokens.

Turn weeks of expert work into minutes

See how Scalata fits your team, your data, and your controls.