Bidirectional Transformer
A bidirectional transformer is a language model that looks at every word of a sentence in context with all other words at once – including the ones that come after it. This makes it strong at understanding text, but unsuitable for writing new sentences.
Computers can’t read language directly. They first have to turn every word into a string of numbers that captures its meaning within the sentence. This is exactly what a bidirectional transformer does. For every word, it looks at the entire sentence – the words before it and the words after it. The word “bank” thus gets a different string of numbers in “I sit on the bank in the park” than in “The bank froze my account.” The term “bidirectional” means exactly that: the model looks in both directions, not just from left to right.
Why reading in both directions makes the difference
Models like ChatGPT work differently. They read a text from front to back and always predict the next word. When they reach the word “bank,” they only know what came before it. For writing text, that’s exactly right, since the future hasn’t been decided yet. For understanding a finished text, it’s an unnecessary constraint.
A bidirectional transformer doesn’t have this constraint. It receives the complete text all at once and is allowed to skim it like a human would. That’s why it delivers more precise meaning representations than equally sized models with only one reading direction. On tasks like “Is this customer review positive or negative?”, a small bidirectional model often outperforms a much larger text-generation model.
There’s also a cost advantage. Well-known bidirectional models have a few hundred million adjustable numerical values, called parameters. Large chat models run into the hundreds of billions. As a result, a bidirectional model can run on a regular server or even on a laptop.
Fill-in-the-blank training
A model like this can’t be trained to guess the next word, since it already sees that word. Instead, a fill-in-the-blank procedure is used, known in English as “Masked Language Modeling.” About 15 percent of the words are removed from a sentence, and the model has to reconstruct them from context.
So “The goalkeeper firmly caught the ___” should become “ball.” To do this reliably, the model has to build up grammar, word meanings, and world knowledge. This training runs without any human labeling, using only text from the internet and from books. That’s the reason huge amounts of data can be used at all.
Technically, this relies on the transformer’s attention mechanism. Put simply: every word calculates how much attention it should pay to every other word in the sentence. In text generators, the view to the right is artificially blocked. In a bidirectional transformer, this block is simply left out – that’s the entire technical difference.
Search engines, spam filters, and their role in news
The best-known model of this type is called BERT and comes from Google, dating from 2018. Google has used it in web search since 2019 to capture the intent behind search queries. Someone who types “medication without prescription pharmacy” means something different than someone who just types the individual words. A bidirectional model reliably picks up on such nuances.
In companies, these models mostly run invisibly in the background. They sort support requests by topic, filter spam, recognize company names in news texts, or assess sentiment in stock market announcements. The search function in many AI systems also uses bidirectional models to find matching passages of text in databases.
A common misconception is that bidirectional models have been made obsolete by chatbots. They can’t write fluent responses, true, but that’s not what they’re meant to do. In technical articles, they’re therefore often referred to as “encoder models,” as opposed to the text-generating “decoder models.” Where the task is classifying and retrieving rather than formulating, they remain the most efficient choice.