RLHF

RLHF stands for “Reinforcement Learning from Human Feedback,” meaning reinforcement learning from human feedback. It refers to a training method for language programs like ChatGPT. Such programs first only learn how texts on the internet typically continue. Afterward, they can form sentences, but they don’t know which answer is actually useful to a human. That’s why, in RLHF, humans rate many of the program’s answers and say which one is better. The program is then adjusted so that it more often delivers the preferred answers.

Why a language model without feedback seems useless

A program that only continues text often behaves strangely. If you ask it something, it might respond with ten more questions. This is, after all, a common pattern on the internet, for example in forums or quiz lists. Technically that’s correct, but useless for the user. RLHF closes exactly this gap between “linguistically plausible” and “helpful.”

The second reason is safety. The training material from the internet also contains insults, conspiracy narratives, and instructions for dangerous things. Without correction, a model would readily reproduce all of that. Human raters rate such answers poorly. After training, the model independently refuses many of these requests.

RLHF is therefore considered one of the reasons for ChatGPT’s success starting in late 2022. The underlying technology already existed beforehand. Only the fine-tuning by humans turned it into a product that millions of people could use without instructions.

From thumbs up to reward model

The process runs in three steps. First, paid specialists write sample answers to typical questions. The model is trained on these examples and thereby learns the desired response style. This step is called supervised fine-tuning.

In the second step, the model generates several answers to a question. Humans rank these from best to worst. From thousands of such comparisons, a second program emerges, the reward model. Its sole task is to rate every answer with a score. It thus imitates the taste of the human raters.

In the third step, the language model practices against this reward model. It generates answers, receives points, and adapts in order to get more points. This is similar to how a player learns a video game: through trial and error and score, not through memorization. The trick is that the points are now awarded automatically. Without a reward model, a human would have to rate every single practice round, which would be far too slow.

Limits, criticism, and successors

RLHF is now built into almost every well-known chatbot, for example at OpenAI, Google, or Meta. When you tap thumbs up or thumbs down in a chat app, you yourself are providing raw material for such training. Even the typical response “I can’t help with that” is a direct result of this process.

However, the method has clear weaknesses. The model learns what raters like, not what is true. Friendly-sounding but false answers sometimes get good marks as a result. Experts call this ingratiating behavior sycophancy. Additionally, the result depends on the values and background of the raters, which regularly leads to debates about political bias.

That’s why you also encounter alternatives in the news. With Constitutional AI, a second AI rates the answers based on written rules, which saves human labor. Methods like DPO manage entirely without a separate reward model and are cheaper. The basic idea remains the same in all cases: human preferences are meant to shape the machine’s behavior.