LAB Benchmark
LAB is a method developed by IBM and the MIT-IBM Watson AI Lab that fine-tunes language models using artificially generated training data. The associated benchmark measures how well a model trained this way performs compared to models improved with expensive, human-written data.
A language model is a computer program that has learned from vast amounts of text to continue sentences. After this initial learning phase, it can generate text, but still follows instructions poorly. That’s why it undergoes a second round of training with examples of good questions and answers. Such examples used to be written by hand by humans, which is slow and very expensive. LAB is a method that instead has another model generate these training examples. The LAB Benchmark is the associated measurement: it checks whether a model trained with artificial examples becomes just as good as one trained with human ones. The name stands for “Large-scale Alignment for chatBots.”
What cheap training data changes for the model world
The second round of training is the point at which a text continuer becomes a usable assistant. Large companies employ thousands of people for this purpose, writing and evaluating example answers. These datasets are a key part of their competitive edge and are usually not published. Smaller labs and universities can hardly keep up with these costs.
LAB targets exactly this point. If a model can produce its own training exercises, costs drop drastically. IBM released models called Labradorite and Merlinite for this purpose, built on open base models. In measurements, they achieved scores close to models painstakingly fine-tuned by hand.
Proper classification matters here. LAB is not an exam-style test like MMLU, where models answer knowledge questions. It is first and foremost a training method, and the benchmark measures whether this method works. Anyone reading about “LAB” in the news should check whether the method or the measurement result is being referred to.
Tree diagram instead of random questions
The core of LAB is a so-called taxonomy. This is a tree-structured list of skills and areas of knowledge the model is meant to master. One branch might be called “programming,” a sub-branch “Python,” and beneath that “debugging.” Humans maintain this tree and place a few example tasks on each leaf.
After that, a teacher model takes over. It receives the examples from a branch and generates hundreds of similar tasks with answers from them. A second step filters out poor or incorrect results. This way, very large training sets are created from just a few human templates.
During the actual fine-tuning, LAB proceeds in stages. First, the model learns factual knowledge, then skills such as summarizing or reasoning. This order is meant to prevent newly learned material from overwriting old knowledge. This forgetting is known in research as “catastrophic forgetting,” and it is a well-known problem in fine-tuning.
LAB in open models and corporate projects
LAB is most visible in the InstructLab project, which IBM and Red Hat operate as open software. There, anyone can add a new skill to the skills tree, similar to a Wikipedia entry. The system generates training data from this and updates the model. This allows a model to be extended with a laptop and without a data center.
For businesses, this is particularly interesting when it comes to internal knowledge. An insurance company can add its own tariff rules to the tree without handing the data over to an external provider. IBM markets its Granite models with exactly this pitch. In stock market and tech news, LAB therefore usually comes up in connection with the competition between open and closed models.
A common misconception is the assumption that artificial training data is always equally good. If a model only learns from itself, its errors reinforce themselves over multiple rounds. LAB attempts to cushion this through human-maintained structure and automatic quality control. Whether this is sufficient in the long run remains an open question in research.