RLAIF

A chat program like ChatGPT is not just fed text from the internet. After this initial learning phase comes a second one, in which it is taught which responses are desirable and which are not. For a long time, paid testers compared two responses each and marked the better one. RLAIF replaces these humans with a second computer program that takes over the evaluation. The abbreviation stands for “Reinforcement Learning from AI Feedback” — in other words: learning through reward, where the reward comes from a machine. The result is the same principle as before, just without the bottleneck of human working hours.

The bottleneck of paid evaluators

Human evaluation is slow and expensive. A single large language model needs hundreds of thousands of comparisons during this phase. Each one costs money and takes minutes, because a person has to read and weigh up two often lengthy texts. For companies, this is one of the biggest cost items in the final phase of development.

On top of that, there’s a quality problem. Humans get tired, are inattentive, and evaluate according to different standards. Two people can rate the same answer in opposite ways. An evaluating program, by contrast, is always equally strict and keeps working around the clock.

It’s important to distinguish this from the older method RLHF, where the H stands for “Human,” meaning human feedback. RLAIF is not a counter-approach but a further development. In practice, providers mix both: humans set the rules, the machine applies them millions of times over.

When one model grades the other

First, the model being trained writes two different answers to a question. A second model, often called the “judge,” is shown both and is supposed to pick the better one. So that it doesn’t decide arbitrarily, it is given a written set of rules. This states, for example: prefer the answer that is more helpful, contains no insults, and gives no instructions for dangerous actions.

From many such judgments, a reward model emerges. This is a third program that assigns a score to every answer. The actual model is then adjusted repeatedly until it achieves the highest possible scores. You can think of it like vocabulary drills with instant feedback: whatever earns points happens more often, whatever costs points happens less often.

A well-known pitfall is self-confirmation. If the judge and the student are the same base model, the system rewards itself for its own preferences. Particularly long or politely worded answers then tend to become popular, without the content actually getting better. Experts call this reward hacking: the model optimizes for the score instead of for actual quality.

RLAIF in current chatbots and news coverage

The method is most visible at Anthropic, the company behind the chatbot Claude. There, the underlying set of rules is called the “Constitution,” which is why people also speak of Constitutional AI. These rules are publicly viewable. They explain why Claude declines certain requests and does so by giving reasons rather than simply blocking them.

In news about AI, RLAIF usually comes up in connection with cost and speed. When a provider unveils a new model version within just a few months, automated evaluation is often behind it. Freely available models from smaller teams also use the technique, because these groups cannot afford large teams of evaluators.

For investors and observers, one point is crucial: human oversight doesn’t disappear, it shifts. Whoever writes the rulebook indirectly determines the behavior of millions of conversations. This is exactly what is currently being debated, including with regard to regulation such as the EU AI Act.