QwenChineseBench

QwenChineseBench is a collection of test tasks for computer programs that process language. Such programs answer questions, summarize texts, or write texts themselves. The test consists of many tasks in Chinese, for which the correct solution has been determined in advance. The program works through them, and afterward the number of correct answers is counted. The result is a score that allows different programs to be compared directly. The name refers to Qwen, a well-known program family from the Chinese corporation Alibaba.

Why Chinese needs its own test

Most well-known tests for AI systems are written in English. A system can score very well on them and still be weak in Chinese. The reason lies in the training material: whoever has mainly read English texts is more familiar with English idioms. A good English score therefore says little about performance in Chinese.

On top of that, Chinese works differently on a technical level. There are no spaces between words, and a single character can mean something different depending on the neighboring characters. The topics also differ: questions about Chinese history, proverbs, or the local legal system rarely appear in English-language tests. It is exactly this gap that a test like QwenChineseBench is meant to close.

For companies, this has a very concrete reason. Anyone wanting to sell an AI product in China must be able to prove that it works well there. A score from a Chinese test is an argument for this that customers and investors can understand.

From task sheet to score

The process resembles an exam with an answer key. First, a fixed set of tasks is compiled, often several thousand. Each task has a stored model solution. The program being tested is presented with the tasks one after another and answers them. An evaluation program then compares the answers with the model solutions.

For multiple-choice questions this is straightforward: the letter is either right or wrong. It becomes more difficult with freely formulated answers, such as a summary. Here similarity measures are used, or a second, very strong AI system is used to grade the answer. This procedure is called LLM-as-a-judge, meaning AI as examiner. It saves work, but has its own weaknesses, because the examiner can also be wrong.

A well-known problem is called contamination. If the test tasks were freely available on the internet at some point, they could have ended up in the program’s training material. In that case the system remembers the solution instead of actually reasoning. The score looks good but says nothing about genuine ability. For this reason, providers keep parts of such tests confidential or renew them regularly.

Where such scores show up

You most often encounter such figures in reports about new AI models. There you’ll typically find a table with test names in the left column and percentage figures next to them. Chinese providers such as Alibaba, DeepSeek, or Moonshot use such figures to compare themselves with American competitors. Financial news outlets also pick them up, since they are seen as an indicator of the technological gap between countries.

A degree of skepticism is worthwhile when reading such figures. The company selling a model chooses itself which tests to show. Tests with poor results are then simply left out. A lead of one point is also usually meaningless, since the measurement fluctuates. Clear gaps or results that have been verified by independent parties are more meaningful.

Such tests are especially useful in practice for developers. Anyone building an application for Chinese users can use them to pre-select several models before trying them out personally. However, the test cannot replace one’s own hands-on trial. A model with a good score can still fail on a company’s specific texts.