OSWorld

OSWorld is a standardized test for computer programs that are meant to operate a computer independently. Such programs see the screen, move the mouse pointer, and type on the keyboard — just as a human would. OSWorld presents them with around 370 tasks drawn from everyday office life: sorting a spreadsheet, renaming a file, changing a setting in the browser. The tasks don’t run in a simulation but in a real operating system, mostly Linux, sometimes also Windows. At the end, a script checks whether the result is actually correct. The test was published in 2024 by researchers from the University of Hong Kong and other institutions and is freely available.

The stress test for computer-operating AI

Language models are good at writing text. Whether they can also act is an entirely different question. That is precisely the gap OSWorld measures. A model that can talk fluently about spreadsheets may still fail to click the right cell in an actual spreadsheet.

The numbers show this clearly. Humans solve over 70 percent of OSWorld tasks. The first AI systems tested in 2024 achieved around 12 percent. By 2025, the best scores had risen to around 60 percent. This increase is one of the reasons why many companies are now betting on so-called agents — AI systems that don’t just respond but carry out tasks.

For investors and observers, OSWorld is therefore a kind of fever thermometer. When OpenAI, Anthropic, or Google present a new model capable of operating a computer, the OSWorld score is almost always mentioned in the press release. It has become one of the standard figures used to gauge progress in this field.

Virtual machine, screenshot, click

For each task, the test launches a virtual machine. This is a complete computer running as a program on another computer. This way, every attempt starts from the same clean initial state, and nothing can damage real data.

Then a loop runs. The AI receives an image of the screen along with the task description. It decides what to do and outputs an action: click on this coordinate, type this text, press Ctrl+S. The action is executed, a new screenshot is taken, and the loop starts over. After a fixed number of steps, it ends.

The evaluation is the most sophisticated part. Each task has its own verification script that examines the final state of the system. It might, for instance, open the saved file and check whether the column has really been sorted. So it is the result that is evaluated, not the path taken to get there. After it emerged in 2025 that some verification scripts were flawed, a revised version called OSWorld-Verified was created.

From lab figure to product on the screen

You rarely encounter OSWorld directly — it’s a tool for research and development. Indirectly, however, you very much do. Features like Anthropic’s “Computer Use,” OpenAI’s Operator, or Google’s Project Mariner are exactly the kind of system that OSWorld measures. When such tools show up in office programs or browsers, this test area is behind them.

In news reports, the name usually appears as a percentage, such as: “achieves 62 percent on OSWorld.” Such figures should be read with caution. The results depend heavily on how many steps were allowed and which version of the test was used. Two numbers from different press releases are not automatically comparable.

A common misconception is also that a high score means a reliable digital assistant. Even 60 percent means: in four out of ten cases, something goes wrong. In the case of a bank transfer or a deleted file, that would be hard to justify. That’s why such systems currently mostly operate under human supervision.