Resources/Model training/Data preparation

Setting

System prompt

The system prompt is a standing instruction placed before every conversation. Added at training time, the model learns its behaviour in the presence of that exact prompt, so it should stay the same when the model is used.

Chapter 53 min read3 references

What it is

A chat has three roles: user, assistant and system. The system message comes first and is not said by anyone in the conversation; it is a standing instruction — who the assistant is, what it should and shouldn't do, how long and in what format it answers. In everyday use it is how you steer a model without retraining it.

In data preparation, the system prompt is an optional setting that is inserted at the top of every training record. It turns a pile of individual chats into examples of one consistent assistant.

How it works during training

systemYou are Scalata's support assistant.read · masked
userHow do I reset my password?read · masked
assistantOpen Settings → Security and choose Reset password.scored · loss counted
The model reads every token, but its error is only measured on the target reply. The context shapes the prediction; it is never itself graded.

Like the user's message, the system prompt is context: the model reads it when predicting the target reply, but the error is measured only on the target tokens. So training does not teach the model to write the prompt; it teaches the model what replies look like when that prompt is present.

That conditioning is the important part. Before it reaches the model, a chat is flattened into one token sequence using the model's chat template, with special tokens marking where the system, user and assistant parts begin. The model's learned behaviour is associated with that exact sequence. If you later use a different system prompt — or none at all — the input no longer looks like what the model was trained on, and the learned behaviour transfers only partly. The same reason is why the Hugging Face documentation stresses formatting training data with the same chat template the model will see in use.

one training record with a system prompt
{"messages": [
  {"role": "system",    "content": "You are the support assistant for Acme Billing. Answer in two to four short sentences. Name the exact menu path. If you are not sure, say so and offer to connect a human."},
  {"role": "user",      "content": "How do I download last month's invoice?"},
  {"role": "assistant", "content": "Go to Billing → Invoices, pick last month and choose Download PDF. Invoices appear on the 2nd of each month."}
]}

What to put in it

  • Identity and scope: whose assistant it is and what it helps with.
  • Answer style: length, structure, whether to use lists, what to do when unsure.
  • Hard rules the examples might not cover, such as never quoting prices or always offering a human.
  • Not knowledge. Facts belong in the examples (or in retrieval at run time); a prompt full of product details is long, costly and goes stale.

Make sure the prompt and the target replies agree. If the prompt says “answer in two sentences” but half the targets are long walkthroughs, the model learns that the instruction doesn't mean much — and may ignore it in other places too.

Worked example: what changing the prompt does
  1. A model is trained on 1,500 support chats with the roughly 50-token prompt above. Its replies are short, name the menu path and offer a human when unsure.
  2. In use, someone replaces the prompt with a 400-token one listing every product feature. Replies get longer and the “offer a human” habit appears less often: the input no longer matches training.
  3. Restoring the original prompt brings the trained behaviour back. If the new instructions are really needed, add them to the training prompt and train again.
  4. Cost: 50 tokens × 1,500 records × 3 epochs = 225,000 training tokens for the prompt alone. The 400-token version would be 1.8 million.
System promptTypical 1–5 sentences#

Optional text added before every conversation. A few sentences on identity, scope and answer style is usually enough; leave it empty only if the model will always be used without one.

Common mistakes

  • Training with one system prompt and deploying with another, then concluding that fine-tuning “didn't work”.
  • Packing facts into the prompt that belong in the examples or in retrieval.
  • A prompt that contradicts the target replies, teaching the model that its instructions are optional.
  • Forgetting the token cost: a long prompt is paid for in every record, every epoch and every later request.

References and further reading

External sources open in a new tab.

  1. 01Chat templatesHugging Face Transformers docs · huggingface.coHow system, user and assistant messages become one token sequence, and why training and use should match.
  2. 02Supervised fine-tuningOpenAI API docs · developers.openai.comTraining examples in the chat format, including the system message.
  3. 03Prepare data for fine-tuning your modelsAmazon Bedrock User Guide · docs.aws.amazon.comA format with a separate system field per record.

Turn weeks of expert work into minutes

See how Scalata fits your team, your data, and your controls.