Computer-Use Agent

A computer-use agent is a program with artificial intelligence that operates a computer by itself. It is shown images of the screen, just as a human would see them. It then decides where the mouse pointer should click and which keys should be pressed. This way it can open a browser, fill out a form, or save a file in a spreadsheet. The difference from a normal chat program is important: a chat program replies with text, whereas this system acts on its own. You give it a goal in plain language, such as “Book the cheapest train to Hamburg on Friday.” It then works out the path to that goal itself, step by step.

Why clicking instead of a programming interface

Software is normally controlled via so-called interfaces, that is, fixed, predefined commands for programs. Such interfaces are reliable, but not every program has one. Many companies work with old industry software for which nothing of the kind exists. An agent that simply clicks and types, by contrast, works anywhere a human could work.

For this reason, this technology is seen as a possible major step in automating office work. What is meant are dull, repetitive processes: transferring data from emails into a system, checking invoices, tracking orders. These are exactly the kinds of tasks that cost many work hours today. Providers such as OpenAI, Anthropic, and Google have introduced their own versions since 2024, which also makes the topic interesting for investors.

At the same time, new risks are emerging. An agent with a mouse and keyboard can delete files, transfer money, or send out data. A text model that makes a mistake merely writes nonsense. An acting agent that makes a mistake causes real damage.

The loop of seeing, thinking, and clicking

At its core, the same loop always runs. First, a picture of the screen is taken, a so-called screenshot. This image is passed to a model that can process images and text at the same time. The model describes what it sees and proposes the next action. This action is carried out, and then everything starts over from the beginning.

The most difficult sub-task is called grounding: the model must specify at which point in the image the intended button is located. So it provides coordinates, for example pixel 840 from the left and 312 from the top. Small deviations lead to misclicks. Some systems therefore additionally use a webpage’s source code to locate buttons more reliably.

To keep errors from escalating, such agents mostly run in an isolated environment, a virtual machine. This is a simulated computer within the computer, whose damage remains contained. For sensitive steps such as payments, the agent asks for confirmation. A well-known problem is hidden instructions on websites meant to redirect the agent. Experts call this prompt injection, that is, smuggled-in commands.

Where these agents already work today

So far, consumers encounter this technology mainly in browsers. There are features that independently compare products, fill shopping carts, or transfer appointments from a website into a calendar. These offerings are often tied to a paid subscription and work noticeably slower than a practiced human.

In companies, it is less about purchases and more about routine work in the background. What is being tested, for instance, is transferring customer data between systems or automatically clicking through software during quality testing. Progress is measured using test suites such as OSWorld, which give the agent hundreds of real computer tasks. Success rates there long remained clearly below those of humans, but are rising quickly.

A common misconception is that a computer-use agent is the same as classic office automation. That older technology follows rigid, manually recorded click sequences and breaks down as soon as a window shifts position. The agent, by contrast, makes a new decision at every step and can adapt. In exchange, it is less predictable and more expensive to run.