Reinforcement fine-tuning (RFT)

What it is

In supervised fine-tuning the model is shown the answer and graded on every token of it. Reinforcement fine-tuning (RFT) never shows the model an answer. It writes its own, a grader scores the result, and training makes high-scoring kinds of answers more likely. The only thing you must supply is a way to tell a better answer from a worse one.

That makes RFT the method of choice when a good answer is easy to recognise but hard to write down in advance: a calculation whose result can be checked, a classification with a known label, a JSON object that must validate, or an open question where a capable judge model can tell a careful answer from a sloppy one.

The vocabulary of reinforcement learning

RFT borrows its words from reinforcement learning (RL), the field that trains game-playing agents and robots. Mapped onto a language model they mean:

Two things make RL on language models different from supervised training. The reward arrives only at the end: the grader scores the finished answer, so training has to work out which of hundreds of tokens deserve the credit. And the data is generated during training: every step, the current model writes fresh answers, so what it learns from changes as it improves. This is called learning on-policy.

How a score becomes an update

The update rule at the heart of RFT is called a policy gradient. In plain words: for each answer, take every token the model wrote and push its probability up if the answer's advantage is positive, or down if it is negative, by an amount proportional to the advantage. A token that appeared in a well-scored answer becomes a little more likely in that context; a token from a poorly scored answer a little less.

Raw scores make a poor signal on their own. If every answer to an easy prompt scores 0.9, raising all of them says nothing about what made them good. So RL methods compare each score with a baseline, an estimate of what an ordinary answer to that prompt scores, and only the difference counts. Older methods such as PPO learn the baseline with a second neural network. GRPO, the method most RFT services now use, simply takes the average score of several answers to the same prompt; see GRPO, step by step.

for each step:
    prompts = a few rows from your data
    answers = model.sample(prompts, n = 4)          # 4 tries per prompt
    scores  = grader(prompt, answers, reference)    # 0–1 each; the model never sees the reference
    adv     = (scores − group mean) / group std     # per prompt
    update the model: tokens of answers with adv > 0 become more likely,
                      tokens of answers with adv < 0 less likely

Your own reply appears only in the grader's input, as a reference. The model being trained never sees it, which is the main practical difference from supervised training: the model can't copy its way to a high score, it has to find one.

{"prompt": [{"role": "user", "content": "How do I share a report with a client?"}],
 "reference": "Open the report → Share → pick the client. They get an email link."}

• One row per reply, ending at that reply. A chat with three of your replies gives three rows. Each row's reference is only its last reply; earlier replies stay in the prompt as context.
• System instructions as a `developer` message. Hosted reasoning models (the o-series) don't accept a system message in reinforcement examples. The same instructions go in a developer message, which these models treat the way other models treat system.

What RL can and can't do

Because it learns only from its own attempts, RFT can only reinforce behaviour the model already produces some of the time. If none of the N answers to a prompt is ever good, every answer scores about the same, the comparison is empty, and the prompt teaches nothing. Research backs this up: Yue et al. (2025) found that RL-trained reasoning models beat their base models when allowed one attempt per problem, but that with many attempts the base models solved more. RL made the model find its good answers more reliably; it did not give it answers it could never reach.

That is why the standard recipe is supervised first, then RL. Supervised training on good replies teaches the task, so the model's attempts start landing in the right area; RL then sharpens which of those attempts it prefers. On a very strong base, RL alone can go far: DeepSeek-R1-Zero, trained with rule-based rewards only, raised its AIME 2024 pass@1 from 15.6% to 77.9%. Most fine-tuning starts from much smaller models, where a supervised step first makes the difference.

When to use it

• The answer can be checked: a figure, a label, a format, a test. Rule-based graders are cheap, exact and hard to game.
• The starting model already gets a fair share right, for example after supervised training on your data, or because it is large.
• You can judge quality but can't write ideal answers at scale: a strong judge model can score free-text answers against your reply.
• Not to teach new facts or a new product: those have to come from supervised data or retrieval.

Common mistakes

• Running RFT on a model that almost never answers well. With every score near the floor there is nothing to compare; train it supervised first.
• Treating a rising reward as proof of improvement. The reward is the grader's opinion; confirm with an independent evaluation.
• Choosing a grader that measures the wrong thing, such as word overlap on a task where meaning matters. The model optimises exactly what is measured.
• Expecting RFT to add knowledge. It reorders what the model already does.
• Setting the answer limit below a good answer's length, so complete answers are cut off and scored as bad.

References and further reading

External sources open in a new tab.

1. 01DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning↗DeepSeek-AI, 2025 · arXiv · arxiv.orgLarge-scale RL with rule-based rewards, and why a supervised cold start still helped.
2. 02Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?↗Yue et al., 2025 · arXiv · arxiv.orgEvidence that RL sharpens what a base model can already do rather than adding new reasoning.
3. 03Reinforcement fine-tuning↗OpenAI API docs · developers.openai.comA hosted RFT service: graders, data, and when RFT is a good fit.
4. 04Training language models to follow instructions with human feedback↗Ouyang et al., 2022 · arXiv · arxiv.orgThe paper that made RL from feedback the standard final step for chat models.

Related concepts