WebArena

WebArena is a testing procedure for programs that are supposed to operate websites on their own. For this purpose, complete replicas of real internet sites were set up: an online shop, a discussion forum, a map, a software platform. These sites run on the researchers' own servers and are cut off from the open internet. A program is given a task there in plain language, for example: Find the cheapest camera under one hundred euros and put it in the shopping cart. It then has to click, type, and scroll like a human. In the end, WebArena automatically checks whether the result is actually correct. The environment was developed in 2023 at Carnegie Mellon University in the USA.

Why clicking is harder than answering

Most well-known tests for AI systems consist of questions and answers. The model writes a text, and someone compares it to the model solution. WebArena demands something different: the program has to act, and in doing so, change something. A wrong answer is annoying, a wrong order costs money.

That is precisely why the test is meaningful. It does not measure whether a system sounds plausible, but whether the correct item ends up in the shopping cart at the end. Experts call this outcome-based evaluation. A program cannot talk its way out of it with clever phrasing.

The results were sobering at first. The best systems solved only about 14 percent of the tasks, whereas humans solved around 78 percent. This gap has become the most important argument against overly high expectations for so-called AI agents. These are programs meant not just to talk, but to carry tasks through to completion on their own.

A replica of the internet in the lab

The central trick of WebArena is its self-containedness. Real websites change daily, prices fluctuate, offers disappear. A test on such sites would no longer be repeatable the next day. The replicated sites, by contrast, look the same on every run. This allows two systems to be compared fairly.

The process is always the same. The program sees the current state of the page, usually as a text description of the available buttons, fields, and links. It then chooses an action, for example a click on a specific element. The page responds, a new state arises, and the cycle begins again. Some tasks require thirty steps or more.

The final evaluation runs without humans. For each of the more than 800 tasks, it is specified how success can be recognized. Sometimes it is a text that must appear in the answer. Sometimes the database is checked directly to see whether the entry was actually created. Typical sources of error are long chains: a single wrong click at step three renders all subsequent steps worthless.

Where the numbers show up in the news

When a company presents a new AI system meant to work in the browser, the press release almost always includes a percentage figure. WebArena is one of the numbers cited, alongside related tests such as WebVoyager or OSWorld. If the figure rises from 15 to 40 percent, the industry considers that significant progress.

For practical purposes, putting the figure in context matters more than the number itself. 40 percent means that more than every second task fails. Such a system cannot yet be trusted to work unsupervised with a credit card. Yet this is exactly what products aiming to handle shopping or booking a trip are targeting.

A common mistake is to apply results from WebArena directly to the real internet. Out there, there are advertising banners, cookie notices, login forms, and checks designed to keep machines out. On real sites, success rates therefore usually drop even further. The benchmark shows a lower bound of difficulty, not the full reality.