VLOSE

Some computer programs can process images and language together. You show them a photo, ask a question about it, and they respond in full sentences. Such programs are trained beforehand on millions of example images with matching descriptions. The interesting question is: What happens when something appears in the image that never occurred in these examples? This is exactly what VLOSE targets. The abbreviation stands for Vision-Language Open-Set Evaluation, meaning roughly: testing image-and-language systems against an open, not predetermined set of things. The name does not describe a single product, but rather a way of systematically testing such systems.

The opposite is called closed-set testing. There, the system is given a fixed list of possible answers in advance, say a hundred animal names. It only has to select the correct one from that list. An open-set test removes this list. The system must formulate on its own what it sees — or admit that it does not know.

Why the list of known things is never complete

The world contains more objects than any training dataset can cover. A self-driving system will eventually encounter a toppled-over construction tent, a mattress on the highway, or a horse on a country road. If the system is only allowed to choose between previously learned categories, it will inevitably misclassify the unknown. It might say “trash can” and keep driving. That is more dangerous than an honest “unknown obstacle”.

Behind this lies a fundamental problem: models are trained to always give an answer. They often even report high confidence scores when they are completely wrong. Experts call this overconfidence. VLOSE-style tests therefore measure not only how often a system is correct. They also measure whether it notices when it reaches its limits.

For companies, this is a very concrete issue. Anyone deploying an image system in quality control wants to know how it reacts to a type of defect that has never occurred before. A test result of 95 percent on known cases says nothing about that. Only open-set tests provide a reliable figure here.

How an open-set test is structured

The basic idea is simple. The available image categories are split into two groups. The model is trained with the first group, while the second group is withheld entirely. During testing, both groups are mixed together again. The model does not know which images belong to which group.

Evaluation then proceeds in several steps. First: Does the model continue to reliably recognize the known things? Second: Does it flag the withheld things as unknown instead of assigning them to a wrong category? Third: How well does it describe the unknown in its own words? A system that answers “a metallic apparatus with hoses” for a device it has never seen is more useful than one that simply guesses “coffee maker”.

Evaluating freely formulated answers is the difficult part. There is no single correct solution to compare against. In practice, similarity measures between the answer and several acceptable descriptions are used. Sometimes humans also review samples manually. Both approaches are more elaborate than simply checking off an item on a selection list.

VLOSE in model announcements and product datasheets

When a company introduces a new vision-language model, a table of test results almost always follows. Alongside familiar tests for image captioning and question answering, open-set tests are increasingly appearing there. These scores are typically noticeably lower than the others. This is not a sign of a poor model, but of a harder test.

In everyday life, you encounter this more indirectly. When a phone app is supposed to identify a plant and responds “probably an umbellifer, no reliable identification possible,” exactly this principle is at work behind it. Inspection systems in factories and in medical technology are also increasingly being judged by this standard.

One common misconception should be cleared up. A good result in an open-set test does not mean that a model can correctly name unknown things. Above all, it means that it more reliably assesses its own ignorance. For safety-critical applications, that is precisely the decisive point.