Black-Box Access

A computer program that has learned from example data is called a model. Anyone who wants to test such a model basically has two options. Either they are allowed to look inside and see all the internal numbers and intermediate steps. Or they are only allowed to use it: they type something in, get a response, and see nothing more. The second case is called black-box access, literally access to a black box. The name means: the inside remains opaque, only what goes in at the top and comes out at the bottom is visible. The counter-term is white-box access, meaning the open view of everything inside.

Why examiners usually sit in front of a sealed box

The large AI models from companies like OpenAI, Google, or Anthropic are trade secrets. Their internal numbers, the so-called weights, are not published. Anyone wanting to examine these systems almost always has access only through a programming interface. They can send requests and read responses. This makes black-box access not the exception but the norm.

This has consequences for the oversight of AI. Authorities, research groups, and journalists are supposed to check whether a system discriminates or provides dangerous information. They can only base this on the outputs. Whether an error is a rare slip-up or a deep structural problem is barely distinguishable from the outside. The European AI Act therefore requires additional disclosures to regulators for particularly powerful models.

Conversely, restricted access also has an advantage. A test from the outside examines exactly the system that users actually encounter. This includes filters, safety rules, and downstream review steps. Looking only at the bare model would overlook these protective layers.

Testing with inputs and outputs

Anyone examining a black box works like a scientist with experiments. They ask many systematically varied questions and compare the answers. One example: the same application text is sent twice, once with a German name and once with a Turkish name. If the evaluation differs, that is strong evidence of a bias in the system. Why this happens, however, remains unclear.

A second method is called red teaming. Here, a team deliberately tries to make the system misbehave. It rephrases requests, hides instructions in long texts, or switches languages. If it finds a loophole, that constitutes proof. If it finds none, that proves nothing — the loophole may still exist.

The decisive drawback is the effort involved. Every request costs money and time, and the number of possible inputs is practically infinite. Providers also often limit how many requests are allowed per minute. A common misconception is that black-box tests can guarantee safety. They can demonstrate problems, but never their absence.

From credit lending to security research

The term crops up wherever software makes decisions about people. When a loan is denied or an application is rejected, there is often a model behind it that no one is allowed to inspect from the outside. Affected individuals and consumer advocates then have only black-box access. This is precisely where the political demand for more transparency and for explanations of individual decisions arises.

In IT security, the term has long been common. A black-box test of a website means that the tester does not know the program code and proceeds like a real attacker. In AI models, the term also appears in research papers. There it might say, for instance, that an attack works even with pure black-box access — this is considered particularly concerning, because it means any paying user could carry it out.

It is important to distinguish this from a similarly sounding problem. Even someone who sees all the internal numbers of a large model does not automatically understand its behavior. Billions of numbers do not add up to readable meaning for humans. Black-box access describes a matter of permission, not of comprehensibility. An open model can therefore still remain puzzling.