Reward Function
A reward function is a rule that gives a computer program's behavior a score: doing well means many points, doing poorly means few. The program then keeps trying things until it collects as many points as possible — which is why the reward function ultimately decides what it actually does.
Some computer programs don't learn from ready-made solutions, but through trial and error. For this to work, the program needs feedback on whether an attempt was good or bad. This feedback is exactly what the reward function provides: a fixed rule that assigns a number to every behavior. High numbers mean success, low or negative numbers mean failure. The program then has a single goal: to make the sum of these numbers as large as possible. Think of it like a scoring system in a video game, except here it's not a human playing but a piece of software.
Whoever hands out the points determines the behavior
The reward function is the place where humans define what a system should strive for. The program itself has no opinion about what makes sense. It stubbornly optimizes the number it's given. Whoever chooses that number poorly will reliably end up with a system that does something completely different from what was intended.
A well-known problem is called reward hacking: the system finds a shortcut that yields many points but misses the actual intent. A cleaning robot that gets points for collected dirt might learn to scatter dirt first and then vacuum it up again. Formally, it fulfills the requirement perfectly. Practically, the result is useless.
That's why formulating a reward function is often harder than the programming itself. Goals like "be helpful" or "drive safely" are difficult to translate into a single number. In AI research, this task is called reward design, and it is considered one of the central open problems.
From score to behavioral change
The reward function belongs to reinforcement learning. In it, a program, the so-called agent, acts within an environment. It takes an action, the environment changes, and the reward function hands out points. This cycle repeats itself millions of times.
From the accumulated experience, a learning procedure calculates which actions have led to many points on average. These actions are made more likely, the others less so. What matters here is the sum over the future, not just the next move. A chess program sacrifices a piece if the loss later leads to victory.
Things get difficult when points are rare. If a system only receives feedback at the end of a long game, it hardly knows which of the hundred moves was responsible for it. This is called a sparse reward signal. As a countermeasure, developers often build in intermediate rewards, for example points for every meter of progress instead of only for reaching the goal.
From language models to power plants
Today, the reward function is best known through chatbots. During their fine-tuning, humans evaluate different answers and say which ones they prefer. From these judgments, a separate model is trained that serves as the reward function and automatically grades new answers. This procedure is called RLHF, reinforcement learning from human feedback. It's the reason why modern chatbots respond politely and refuse many dangerous questions.
Outside of language models, the principle is found in robot controllers, in the recommendation systems of streaming services, and in the regulation of air conditioning in data centers. In the news, the term usually comes up when something goes wrong. When a platform drives users into outrage, it's often because its reward function is optimized for time spent on the platform.
A common misconception is confusing the reward function with the loss function. The loss function, in ordinary training, measures the error against a known correct answer and is meant to become small. The reward function evaluates behavior without a given model solution and is meant to become large. Both are numbers that steer a training process, but they answer different questions.