Deep Deterministic Policy Gradient

Deep Deterministic Policy Gradient, DDPG for short, is a method by which a program learns a control task through trial and error. It is not presented with a solution, but only a reward: a number that says how well it is currently doing. From many attempts and their associated rewards, it derives which actions are worthwhile. What makes DDPG special is that it can output finely graded actions. A robotic arm is not controlled with “left” or “right”, but with numerical values such as 0.37 Newton-meters of torque. It was precisely for such continuous quantities that the method was developed at Google DeepMind in 2015.

Why continuous control requires its own method

Older learning methods go through every possible action and pick the best one. In a video game with six buttons, this is easy: you compare six options. For a joint that can take on any angle between minus 90 and plus 90 degrees, there are infinitely many. Trying them all out is simply impossible.

DDPG solves this by not searching for the best action but calculating it directly. A network receives the current situation and immediately outputs the appropriate numerical values. This completely eliminates the need to compare all possibilities. That is why DDPG was for a long time the standard method for robotics, vehicle control, and simulations of movement.

An important distinction: DDPG is not a language model and does not write texts. It belongs to reinforcement learning, the area in which programs learn behavior through reward and punishment. The same basic ideas were later used to train game-playing programs and control systems for data centers.

Actor and critic working together

DDPG consists of two learning networks. The first is called the Actor. It observes the situation and decides what to do. The second is called the Critic. It evaluates how much reward a particular action is likely to yield in that situation.

You can picture this like a learner driver with a driving instructor. The student steers, the instructor says whether that was a good idea. The critic learns from the rewards actually received to improve its evaluations. The actor learns to choose actions that the critic rates highly. Both are trained simultaneously and pull each other upward.

Two tricks keep the training stable. First, a memory of past experiences from which examples are drawn at random. This way the system learns not only from the last few seconds but from its entire history. Second, target networks: frozen copies that only slowly adapt to the original. Without them, the system would end up chasing itself and the values could spiral out of control. In addition, some randomness is added to every action so that the program actually tries out new things.

DDPG in robotics, research, and successor methods

In practice, one encounters DDPG mainly in simulations. Standard tasks include a two-legged runner, a grasping robotic arm, or a virtual car on a racetrack. The principle also appears in industry, for example in the regulation of heating systems, cooling circuits, or power grids. In all these cases, it is about finely adjustable control variables rather than simple yes-no decisions.

In articles and news today, DDPG is often presented as the starting point of a family. Direct successors are called TD3 and SAC. They address a known weakness: the critic in DDPG tends to rate actions too optimistically. The program then overestimates itself and learns poor behavior. TD3 uses two critics and trusts whichever of the two is more pessimistic.

A common misconception is that DDPG is a finished product. It is an algorithm, i.e., a computational procedure that is implemented in libraries such as PyTorch. Anyone tackling a new control problem today usually turns to the successor methods. Nevertheless, DDPG remains important because it established the basic structure of actor, critic, and experience replay memory.