Computer Use

Most AI programs that write text do exactly that: they return text. Computer Use goes a step further. Here, the program gets to see an image of the screen and is allowed to control the mouse and keyboard itself. It can therefore open a window, type into a search field, scroll, and click on buttons. The human only states what the goal is, for example: “Search the travel portal for a train to Hamburg on Friday.” The program figures out the individual steps to get there on its own.

Why the mouse matters more than an interface

Normally, programs are connected via an interface, a kind of agreed-upon socket for data. Experts call this an API. That works well, but only if the provider offers such a socket at all. Very many programs don’t have one. Old company software, internal administrative systems, websites without data access: there, the only way in is through the user interface.

This is exactly where the appeal of Computer Use lies. An interface for humans is always present, otherwise nobody could use the program. An AI that can click therefore needs no cooperation from the manufacturer. In theory, it can operate anything that runs on a screen.

Economically, this is the reason for the great interest. Many office jobs consist to a good extent of click work: transferring data from an email into a form, matching orders, copying together reports. If software takes this over, it changes the division of labor in offices. That’s why Computer Use features show up in tech companies' quarterly reports as a promise for the future.

See, decide, click: the loop

Technically, this runs in a loop that repeats itself many times. First, a picture of the screen is taken, a so-called screenshot. This image goes to the AI model, together with the task and the history so far. The model doesn’t respond with text, but with an instruction: “Click on position x, y” or “Type this word”.

A small helper program then actually carries out this instruction, like an invisible hand on the mouse and keyboard. After that, a new screenshot is taken, and everything starts over from the beginning. This way, the model sees whether the click had an effect. A single task can require dozens of such rounds.

For this to work, the model must be able to understand images. Such models are called multimodal, because they process several kinds of input: text and image. The hardest sub-task is surprisingly mundane. The model has to recognize exactly where a button sits in the image, down to the pixel. Small errors of just a few millimeters lead to clicks into empty space.

That’s why Computer Use remains unreliable to this day. In tests, current systems often solve longer office tasks completely only in a minority of attempts. On top of that, the method is slow and expensive, because a whole image has to be freshly analyzed for every single click.

Where click agents are already showing up

Anthropic released a Computer Use feature for its model Claude in 2024, and OpenAI followed shortly after with similar tools for the browser. Google offers something comparable. In products, this is often not called Computer Use but Agent or Assistant. It usually runs inside a shielded virtual computer, so that the AI can’t do damage in the real system.

It’s important to distinguish this from an older method called RPA, short for robotic process automation. There, software also clicks, but following a rigidly recorded script. If the webpage changes its layout, the script breaks. A Computer Use agent looks at what’s there and, ideally, finds the button even if it has moved.

A common misconception is that such an agent works unsupervised. Realistically, today there is human oversight, especially for payments and for deleting data. A real risk is called prompt injection: a webpage contains text that sounds like an instruction, and the agent obeys it instead of its user. That’s why such systems are usually given only tightly limited permissions.