τ³-Bench

τ³-Bench is a standardized test for computer programs that complete tasks independently. Such programs are called agents: they are given a goal and choose the steps to reach it themselves. In τ³-Bench, such an agent plays a customer service representative, for example at an airline. A second program plays the customer and makes requests, often several at once. The agent must resolve the concerns, query databases in the process, and adhere to the company’s rules. In the end, what counts is whether the final state is correct: that is, whether the right thing was actually booked, cancelled, or refunded. The name alludes to older test series whose tasks were considerably simpler.

Why companies are watching this particular test

Language models have long been very strong on classic knowledge tests. They often answer questions from medicine, law, or mathematics better than humans. But that says little about whether they can be entrusted with real work processes. This is exactly where τ³-Bench comes in: it measures action rather than knowledge.

The difference is economically enormous. A company doesn’t replace a support department because a model solves exam questions. It only replaces it once the model reliably closes tickets without making false promises to customers. Mistakes here cost real money, such as an unjustified refund.

The results are sobering, and therefore interesting. Even leading models fail on a large share of tasks as soon as several concerns arrive at the same time. This is a useful counterweight to marketing promises about autonomous employees supposedly just around the corner.

The structure of the simulated customer conversations

The test consists of three parts. First, there is a simulated world: a database of bookings, customers, and prices. Second, there are tools, i.e., commands with which the agent can modify this database. Third, there is a rulebook that describes what is allowed and what is not.

The customer is played by another language model. It does not follow a fixed script but reacts freely to responses. This produces conversations that unfold somewhat differently each time. Evaluation, however, remains strict and automatic: a checking program compares the database at the end with the desired target state.

What is new compared to earlier versions are interlocking tasks. For example, the customer wants to rebook a flight and simultaneously get a refund for the old one. Both depend on each other, and both follow their own rules. A typical model error: they handle the first point and forget the second, or they grant a refund that the rulebook forbids.

Where the numbers show up in news and product announcements

You rarely encounter τ³-Bench directly, since it is a research tool. Indirectly, though, it comes up constantly. When a provider introduces a new model, it usually shows a table with benchmark values, i.e., test results compared to the competition. Agent tests like this one now rank high on such tables.

For investors and journalists, this is a clue as to how far the technology really is. A jump from 40 to 60 percent of tasks solved is solid news. Conversely: anyone selling automation in customer service must deliver such numbers.

One caveat applies. Benchmarks wear out once their tasks end up in training data. Then the scores rise without the models actually acting any better. That’s why new test series keep emerging, and old best scores lose their significance.