ERQA

ERQA stands for Entity-Relationship Question Answering. “Entity” here means a named thing in the world — a person, a company, a country, a product. “Relationship” is the connection between two such things, for example: Who founded which company? Which country borders which other one? ERQA is a benchmark, meaning a standardized test that measures how well an AI model answers exactly these kinds of questions. It checks not only whether the model knows a fact, but whether it establishes the correct connection between two things. That’s harder than it sounds.

Why simple factual knowledge isn’t enough

Many early tests for language models checked isolated facts: What is the capital of France? That’s useful, but too simple. In practice, people much more often ask questions that connect multiple pieces of knowledge: Which company was led by the same founder who previously worked at Tesla? To answer that, a model must keep track of two entities and their relationship at the same time.

This is exactly where ERQA comes in. It exposes an error that would otherwise remain hidden: a model can perform well on simple factual questions and still fail as soon as multiple entities come into play. For developers, this is important feedback, because real applications — search engines, automated research tools, knowledge bases — need exactly this kind of relational linking.

Structure of an ERQA test

An ERQA benchmark consists of many question-answer pairs. Each question is constructed to query a clearly defined relationship between two or more entities. The answer is usually short and unambiguous — a person, a date, a place name. The model receives the question, provides an answer, and this is compared automatically against the correct answer.

The composition of the questions is crucial here. Good ERQA datasets ensure that the correct answers cannot be reached through simple pattern recognition. The model is supposed to genuinely reason, not just repeat the most common word combination from its training. Some variants of the test therefore deliberately include rare or nested relationships that occur less often in normal language use.

A related concept is so-called knowledge-graph QA: here, the question is not posed in free-form language but directed straight at a structured database. ERQA, by contrast, works with ordinary text and is therefore a tougher test for language models intended for use in the open world.

Where ERQA shows up in practice

ERQA is mainly encountered in research papers that introduce or compare new language models. When an AI lab releases a new model, it typically lists a table of benchmark results. ERQA is one of several benchmarks specifically targeting relational reasoning — alongside other tests for logic, mathematics, or text comprehension.

ERQA doesn’t appear directly in products — no user ever sees the benchmark. But its results indirectly influence how well a language assistant answers questions like “Who is the CEO of the company that makes the iPhone?” Search engines that incorporate AI-generated answers, such as Google's AI Overviews or Microsoft's Copilot in Bing, benefit directly from models that perform well on such tests.

For companies that use AI models for internal knowledge bases, ERQA is especially relevant. There, the task is often precisely about establishing connections between people, projects, and documents — exactly what ERQA measures.