Bidirectionality

Programs that process language do not read a sentence like a human does, from front to back. Some look at each word together with everything that comes before AND after it. That is exactly what bidirectionality means: the processing runs in both directions. The benefit becomes clear with a simple example. In the sentence “She sits at the bank and waits for her money”, only the later word “money” makes clear that no riverside or financial institution confusion is meant here — wait, more precisely: it clarifies which meaning of “bank” is intended. A program that only reads forward does not yet have this information at the word “bank”.

What looking backward brings to text comprehension

Language is full of words whose meaning only becomes clear later in the sentence. This applies to ambiguous words like “lock”, “ball”, or “runner”. It also applies to references: who is meant by “he”? A model that surveys the entire sentence at once resolves such questions considerably more reliably.

That is why bidirectional models were a major leap starting in 2018. The best-known one is called BERT, developed at Google. At the time, it noticeably improved results on tasks like text classification or question answering in one stroke. Classification here simply means: a text is assigned a category, such as “complaint” or “praise”.

It is important to distinguish this from chatbots like ChatGPT. These models deliberately do not work bidirectionally. They predict the next word one word at a time and are therefore never allowed to look into the future. Otherwise the task would be trivial: the sought word would already be in view. Bidirectionality is therefore not a mark of quality, but a deliberate choice made for a specific purpose.

Masked words as a training trick

If a model is allowed to see the whole sentence, it cannot be trained with the task “guess the next word”. So a different exercise is used instead. Individual words are randomly removed from the text and replaced with a placeholder marker. The model is supposed to guess what is missing. Experts call this masked language modeling.

This works like a fill-in-the-blank exercise in language class. With “The ___ barked loudly and pulled at the leash”, the part after the blank helps you at least as much as the part before it. The model trains this exact ability millions of times over on large amounts of text. Typically, about 15 percent of the words are masked.

Technically, this is made possible by what is known as the attention mechanism. Put simply, every word in the sentence is allowed to look at every other word and weigh how important it is for its own meaning. In models that generate text, the view to the right is artificially blocked out. In bidirectional models, it is left open. The difference thus lies not in the basic architecture, but in this one restriction.

Bidirectional models in search engines and enterprise software

You most often encounter bidirectionality without noticing it: in web search. Since 2019, Google has been using BERT-like models to better interpret search queries. Especially with longer questions containing small words like “for”, “without”, or “after”, context changes the meaning completely. This is exactly where two-way reading shows its strength.

In companies, such models run in the background of many unspectacular tasks. They sort incoming emails, recognize company names in contracts, or assess sentiment in product reviews. The advantage is practical in nature: these models are often small enough to run on ordinary hardware. A chatbot model with hundreds of billions of parameters would be needlessly expensive for such routine tasks.

In the news, the term appears less often today than a few years ago. This is due to the dominance of text-generating models. A common misconception is therefore that bidirectional models are outdated. They are, rather, specialized: those who want to understand and categorize texts often still reach for them.