Bits per Byte

A byte is the usual storage unit for a character in a text, for example for an “A” or a comma. A bit is the smallest unit of information there is: a zero or a one. Eight bits fit into one byte. Bits per Byte now measures how many bits a computer program actually needs on average to capture such a character of a text. Programs that can predict language well get by with significantly fewer than eight bits, because they can already guess much of it. This is precisely why this number is used to compare how well different programs have understood language.

Why a compression number reveals something about language understanding

Behind this metric lies a surprising idea: prediction and compression are the same problem. Whoever knows which character comes next no longer needs to write it out in full. After “Good morn”, the next character is almost certainly an “i”. This “i” then costs only a fraction of a bit of information. A model that understands language poorly, on the other hand, must allocate a lot of space for every character.

This leads to a very practical property. Bits per Byte can be calculated on any text whatsoever, without anyone having to formulate tasks with correct answers beforehand. No exam questions, no evaluation by humans, no extra work are needed. The text itself is the test. This makes the number cheap, objective, and repeatable as often as desired.

In research, this matters because many other tests have become unreliable. Well-known question sets eventually turn up in the training data, and from that point on the test only measures memorization. With Bits per Byte, one can simply use a brand-new text that did not yet exist during training. The so-called scaling laws too – the rules of thumb about how much better a model gets through more data and more computing time – are based on such measurements.

From probability to a number

A language model reads a text and outputs probabilities for the next unit of text. If it considers the correct continuation highly probable, the prediction was good. From this probability, one calculates the information content in bits. A prediction with a 50 percent chance of being correct costs exactly one bit. One with 25 percent costs two bits, one with almost a hundred percent costs almost zero bits.

These bits are summed over the entire text and divided by the number of bytes in the original text. This last step is crucial. Modern models do not read text character by character, but in chunks, so-called tokens. How large these chunks are differs from model to model. Dividing by the bytes makes this difference disappear from the calculation again.

This is precisely where the advantage over the related metric perplexity, which one encounters far more often in technical articles, lies. Perplexity is calculated per token and thus depends on how the chunks are divided up. Two models with different tokenization can therefore hardly be compared fairly. Bits per Byte is largely immune to this effect. For reference: good language models today score below one bit per byte on English text, while classic compression programs like ZIP need roughly two to three times as much.

Where the number turns up in technical papers and products

Bits per Byte is most often encountered in scientific publications about new models. There it appears in tables alongside test datasets with names like “The Pile”, a large collection of mixed texts. Companies such as OpenAI, Meta, or DeepMind also use the number internally to check during training whether a model is making any progress at all.

In everyday life, however, one never sees the value. No chatbot displays how many bits per byte it is currently working with. The metric is a tool for developers, not a selling point. Yet when news reports speak of a “better” model, such a measurement is often lurking in the background.

A common misconception deserves mention in closing. A low value does not automatically mean that a model answers helpfully, politely, or factually correctly. Only the predictive quality at the text level is measured. A model can continue language superbly and still assert nonsense. That is why the number always appears in reports alongside other tests, never alone.