Case Vignette

A case vignette is a short, invented description of a situation. It is usually only a few sentences to half a page long. An example: “A 54-year-old patient has complained of chest pain when climbing stairs for three days.” A question then follows: What would you do? Who should receive the medication? Is this person guilty? The same text is presented to many people, and one compares how they answer. This exact procedure is used today to test computer programs that generate human-like texts.

Why researchers prefer invented cases to real ones

Anyone wanting to study the judgment of humans or machines has a problem: real cases are never identical. Two patients differ in a hundred details. If two doctors decide differently, one cannot know whether this was due to the doctors or the cases. The case vignette solves this. Everyone receives exactly the same text, so differences in the answers can be attributed to the respondents.

The second advantage is control over individual details. Researchers write two versions of a vignette that differ in exactly one word. In one, the patient is named Michael, in the other Aischa. Everything else remains identical. If the recommendations then diverge, this points to a bias. This method is called the paired-vignette design and is one of the most important tools in AI fairness research.

Added to this is a practical point: experimenting with real patient data or real court records is legally sensitive and often prohibited. Invented cases contain no personal data. They may be published and shared, and other research groups can repeat the test exactly.

From text to analyzable number

A usable vignette is not simply written down on the spot. Experts first determine which characteristics are to be varied, such as age, gender, or severity of symptoms. This produces a construction kit: from five characteristics with two variants each, 32 variants of the same base case can be built. Independent experts then check whether each variant sounds medically or legally plausible.

Then comes the run. With a language model, that is, a program that continues texts, each variant is sent as a query hundreds or thousands of times. This is necessary because such models do not always give the same answer. Only from many runs does a stable average emerge. With humans, so many repetitions would be unaffordably expensive; with machines, they cost only a few cents per query.

Finally, the free-text answers are translated into categories, such as “emergency room recommended” or “watchful waiting.” Statistical methods then show whether the groups really differ or whether the deviation could be due to chance. A well-known mistake is inferring a pattern from a single striking answer. Without many repetitions, a single outlier is meaningless.

Case vignettes in studies and headlines

When a news report states that an AI system treats female patients differently from male patients, this is almost always based on a vignette study. Well-known investigations of large chatbots have shown that recommendations can shift depending on the assumed origin or insurance status of the described person. Application vignettes are also common: identical résumé, different name.

Outside of AI, the method is old and has been standard in medicine, psychology, social work, and legal studies for decades. Exam questions in medical school are practically always case vignettes. What is new is only that the entity being tested today is often software.

The limitation of the method is important. A vignette is a text, not reality. How someone reacts to a described case does not necessarily correspond to what they would do in an actual hospital. Experts call this the gap between stated and actual decision. Vignette studies therefore provide strong indications of problems, but no proof of real-world behavior.