Policy

Some computer programs are not meant merely to answer a question, but to act. A robotic arm reaches for a cup, a program plays chess, a control system regulates the cooling of a data center. Such programs must decide in every situation what to do next. The rule that makes this decision is called a policy. One can picture it as a rule of thumb: state in, action out. The crucial point is that this rule is usually not written by humans, but learned through trial and error.

Why acting is harder than answering

In a normal prediction task there is a correct answer, and the program learns from examples. Acting is different. No one can write down the best move for each of the billions of possible chess positions. Often there isn’t even a single clearly correct move, only ones that are more or less promising.

On top of that, decisions have consequences. A move that looks good right now can lead to a loss twenty moves later. A policy must therefore not maximize immediate advantage, but success at the end. It is precisely this delay between action and outcome that makes learning demanding.

The term is also important because modern language models are now being fine-tuned according to the same principle. When a chatbot learns to give helpful rather than offensive answers, in technical jargon its policy is being adjusted. Every word it outputs then counts as an action.

Reward as the only teacher

The program receives no sample solution, only a number: the reward. In chess, for instance, there is one point for a win and zero for a loss at the end. The program plays a great many games and observes which actions more frequently led to high rewards. It makes such actions more likely in the future, and the others less likely. This procedure is called reinforcement learning.

Technically, a policy is usually backed by a neural network, that is, a system made up of many adjustable numbers that learns patterns from data. It takes the current situation as input and outputs a probability for each possible action. A policy is therefore rarely rigid. In a sense, it rolls the dice with weighted odds, so that the program also tries out new things.

A common mistake is to confuse a policy with a fixed set of rules. If a human writes “stop at a red light,” that is a line of code, not a learned policy. The difference lies not in the outcome, but in the origin of the rule.

From AlphaGo to the warehouse robot

The principle became best known in 2016, when the program AlphaGo defeated the world’s best Go player. Its policy had emerged from millions of games played against itself. In news about robotics, the term regularly comes up today, for instance when companies work on policies that transfer a gripping system to unfamiliar objects.

In everyday life one encounters policies mostly invisibly. They control which video a platform suggests next, or how a warehouse robot navigates between shelves. The fine-tuning of well-known chatbots also falls under this. One more warning about a double meaning is worth noting: in reports about regulation, “policy” often simply means a guideline or a governing rule. Whether the technical or the organizational meaning is intended is usually revealed by context.