Key takeaways

  • RLHF trains a language model to produce answers people prefer, using a reward model learned from human rankings.
  • OpenAI's InstructGPT paper reported that labelers preferred a 1.3B parameter RLHF model over the 175B GPT-3.
  • DPO, published in 2023, trains on the same preference data without a reward model or a reinforcement learning loop.

RLHF, short for reinforcement learning from human feedback, is a way to train a language model using people's judgments of its answers. Instead of only copying text from the internet, the model is rewarded for the kind of answers people say they prefer. It is the technique that turned raw language models into assistants that follow instructions, and it is the step OpenAI's 2022 InstructGPT paper described in detail.

Reinforcement learning is training by reward: the model tries something, gets a score, and shifts toward what scored well. The hard part is where the score comes from. For "write a helpful reply" there is no formula. So RLHF learns the score from people.

Why models need it

A model trained only to predict the next word learns to continue text, not to do what you asked. Ask it a question and it may answer with more questions, because that is what a list of questions looks like on the web.

The InstructGPT authors at OpenAI made this point directly: making a model bigger does not by itself make it better at following what the user wants. Their fix was to fine-tune GPT-3 with human feedback. The result is the paper's headline. Labelers preferred outputs from a 1.3 billion parameter InstructGPT model over outputs from the 175 billion parameter GPT-3, a model about 100 times larger. The paper also reports gains in truthfulness and fewer toxic outputs, with small regressions on public NLP benchmarks.

How RLHF works, step by step

The InstructGPT paper lays out three steps. Most RLHF since follows the same shape.

  1. Supervised fine-tuning. Human labelers write example answers to prompts. The model is trained on those demonstrations. The paper says this used about 13,000 training prompts.
  2. Train a reward model. The model produces several answers to the same prompt, and labelers rank them. A second model, the reward model, learns to predict which answer a person would prefer. The paper used about 33,000 prompts for this step and 6 billion parameter reward models. The authors say training a 175 billion parameter one could be unstable.
  3. Optimize against the reward model. The language model generates answers, the reward model scores them, and a reinforcement learning algorithm called PPO nudges the model toward higher scores. About 31,000 prompts, all from the API, were used here, with no human labels. The paper also describes a variant, PPO-ptx, that mixes in the original pretraining objective to reduce regressions on benchmarks such as SQuAD, DROP and HellaSwag.

The reward model is a stand-in for the labelers. That lets the model practice on thousands of prompts that no person had to rate one by one.

Helpful and harmless: Anthropic's version

In April 2022 Anthropic published Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. The paper applies the same recipe with two goals at once: helpfulness and harmlessness, which can pull against each other.

Anthropic reports that this alignment training improves results on almost all NLP evaluations, and that it is compatible with specialized skills such as Python coding and summarization. It also describes an online mode, where the preference model and the policy are refreshed every week with new human feedback. The authors report a roughly linear relationship between the RL reward and the square root of the KL divergence from the starting policy. KL divergence is a measure of how far the tuned model has drifted from where it began.

The paper released its preference data, which is public as hh-rlhf on GitHub.

What RLHF gets wrong

RLHF teaches a model to please its raters, and that is not the same as being right.

  • Whose preferences. InstructGPT is aligned mainly to about 40 contractor labelers, most of them English speakers in the US or Southeast Asia, plus the researchers who wrote the instructions and the API customers whose prompts were used. The authors say these groups are not representative of all users, and that they do not claim these are the right people to decide.
  • Still makes mistakes. The authors note InstructGPT still makes simple errors.
  • Complexity. Training three models in sequence, with a reinforcement learning loop on the end, is complex, which is the objection DPO later raised.

DPO: RLHF without the reinforcement learning

In May 2023, Rafailov and colleagues published Direct Preference Optimization, subtitled "Your Language Model is Secretly a Reward Model." They describe standard RLHF as complex and often unstable.

Their method, DPO, reworks the math so the model can be trained directly on pairs of preferred and rejected answers, with a simple classification loss. There is no separate reward model and no reinforcement learning loop, and no sampling from the model during fine-tuning. The authors report that DPO beats PPO-based RLHF at controlling the sentiment of generations, and matches or beats existing methods on summarization and single-turn dialogue.

So the label "RLHF" now covers two things: the classic reward model plus PPO pipeline, and cheaper methods like DPO that learn from the same kind of human preference data. The human feedback stays. The machinery around it is what changed.

How it relates to reasoning models

Reasoning models are a related line of work. Our explainer on what a reasoning model is covers how they are trained and used.

Start with the abstract of the InstructGPT paper, then the Anthropic paper for the helpfulness and harmlessness tradeoff, then DPO for the simpler alternative. All three are free on arXiv.

Frequently asked questions

What does RLHF stand for?

Reinforcement learning from human feedback. A model is trained with a reward signal that comes from people's preferences between its answers.

What is the difference between RLHF and DPO?

Classic RLHF trains a reward model and then runs a reinforcement learning loop against it. DPO trains directly on preferred and rejected answers with a classification loss, with no reward model and no RL loop.

Why did InstructGPT beat a much larger GPT-3?

Its authors say bigger models do not automatically follow user intent. Fine-tuning with human feedback made a 1.3B parameter model preferred by labelers over the 175B GPT-3.

Where can I get RLHF training data?

Anthropic released the preference data from its helpful and harmless assistant paper as hh-rlhf on GitHub.

Sources

  1. Training language models to follow instructions with human feedbackarXiv (Ouyang et al., OpenAI)
  2. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human FeedbackarXiv (Bai et al., Anthropic)
  3. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelarXiv (Rafailov et al.)
  4. hh-rlhfAnthropic on GitHub

Add ClaudeGPT Si as a preferred source on Google