Constitutional AI

Programs like ChatGPT initially only learn to continue text. After that, they have to be taught to behave decently: not to insult, not to give instructions for building bombs, to stay honest. Usually this is done by humans who compare many pairs of answers and mark the better one each time. Constitutional AI replaces a large part of this work. The desired behavioral rules are written down in plain language, similar to a short constitution. The program then checks its own answers against these rules itself and rewrites them. The method was developed and named in 2022 by the AI company Anthropic.

Rules on paper instead of the gut feeling of click workers

The most important benefit is traceability. In the classic method, the understanding of values is embedded in millions of individual clicks by paid testers. Nobody can later say why the model refuses a particular question. With Constitutional AI, the principle is written in a readable document. It can be quoted, criticized, and changed.

There is also a practical point: cost and speed. Paying people to review thousands of responses for violence, hate, or misinformation is expensive and slow. The process also burdens the reviewers, since they constantly have to read disturbing content. A model that corrects itself produces as many training examples in hours as a team would in months.

However, there is a clear limit. The list of rules is still written by humans, namely the company’s employees. So whoever decides what “harmful” means still holds a lot of power. Constitutional AI merely shifts the human value judgment to a different point. It does not eliminate it, but it does make it visible.

Critique and revision in two rounds

The first step is called self-critique. The model is given a sensitive question and first answers it unfiltered. It is then asked to measure its own answer against one of the principles. Finally, it is supposed to write an improved version. These revised answers form a training dataset that is used to retrain the model.

In the second step, the model becomes the judge. For each question it generates two answers and decides itself which one better fits the constitution. These comparisons train a second, smaller program that scores answers with points. The main model then practices until it achieves the highest possible scores. Experts call this reward-based learning reinforcement learning.

The crucial difference from the older method lies exactly here. In the usual Reinforcement Learning from Human Feedback, or RLHF for short, the comparisons come from humans. In Constitutional AI, they come from the model, guided by the list of rules. That is why it is also called RLAIF, meaning feedback from artificial intelligence instead of from humans.

Anthropic’s Claude and the published list of rules

The method is most directly encountered in Anthropic’s chatbot Claude. It is the best-known competitor to ChatGPT and is tuned according to this principle. Anthropic has published the associated constitution publicly online. It contains sentences from the Universal Declaration of Human Rights, but also rules against condescension and against unnecessary lecturing.

In business news, the term usually comes up in connection with AI safety and regulation. Companies increasingly have to demonstrate how they limit the risks of their models. A written set of rules is a good argument to present to regulators in this regard. Other providers are now also working with model specifications that follow the same idea.

A common misconception is that the program actively adheres to a legal text while chatting. It does not. The rules only take effect during training and shape behavior there. In a later conversation, the model does not consult a constitution, but follows habits it has trained into itself. That is why violations remain possible, and safety tests remain necessary.