Reinforcement Learning from Human Feedback

A language program like ChatGPT first learns only a single thing: to predict the next word in a text. To do this, it reads huge amounts of text from the internet. Afterward, it can formulate fluently, but it doesn’t know what answer a human actually wants to hear. This is exactly where Reinforcement Learning from Human Feedback comes in, usually abbreviated as RLHF. Humans rate different answers from the program, and the program adjusts its behavior to these ratings. In this way, a text continuer becomes an assistant that answers questions, follows instructions, and refuses sensitive requests.

Why chatbots would be useless without RLHF

A pure prediction model tends to answer the question “How do I bake bread?” with further questions. This is because on the internet, such sentences often appear in lists of similar questions. Technically, this is a good prediction, but practically it is useless. RLHF closes exactly this gap between statistically probable and actually helpful.

The second reason is safety. The training material also contains insults, misinformation, and dangerous instructions. Without post-processing, a model would willingly reproduce all of this. Through human ratings, it learns to avoid such answers and instead refuse or warn.

RLHF was therefore less of a technical breakthrough than a product breakthrough. The underlying models had partly existed for years before. Only the alignment with human expectations made them usable for millions of people. The technical term for this is alignment, meaning the adjustment of system behavior to human intentions.

Reward model and training loop

The process has three stages. First, paid testers write sample answers to typical queries, and the model learns to imitate this style. Then the model generates several different answers to a question. Humans rank these answers by quality, from best to worst.

From these rankings a second, smaller program emerges: the reward model. Its sole task is to assign a score to each answer. It has learned to predict how humans would likely have judged it. This allows millions of answers to be evaluated without anyone having to read them. Human evaluation would be far too slow and too expensive for these quantities.

In the final stage, the language model practices against this reward model. It tries out different formulations and reinforces those that receive high scores. This is the same basic idea as training a dog with treats. The catch: the model can learn to maximize the score instead of actually giving good answers. Experts call this reward hacking. Typical symptoms are excessively long, friendly-sounding answers without real content.

RLHF in products and in the debate

Every major chatbot has been fine-tuned with RLHF or a successor method. When a chatbot offers you two answers to choose from and asks which is better, the provider is collecting exactly this kind of comparison data. The thumbs up or thumbs down under an answer also belongs to this. Users thus themselves supply part of the training material.

In the news, RLHF mostly appears in connection with criticism. One accusation concerns working conditions: part of the rating work is done in countries with low wages, sometimes involving distressing content. Another accusation concerns bias, since the opinions of a few thousand raters shape the behavior of a globally used system.

In the meantime, further developments have emerged. With Direct Preference Optimization, or DPO for short, the separate reward model is no longer needed, which makes training simpler. With Constitutional AI, an AI evaluates the answers based on written rules, instead of humans checking every answer. The basic principle, however, remains the same everywhere: a model learns from comparisons between better and worse answers.