FastText

Computers cannot do math with letters, but they are very good at it with numbers. That’s why there are programs that translate every word of a language into a long sequence of numbers. Words with similar meanings receive similar number sequences. FastText is one such program, released in 2016 by Facebook's research division, today’s Meta. Its trick: it looks not only at whole words, but also at their fragments of three to six letters. FastText can also sort texts, for example by language or by subject area.

Why word fragments make such a difference

Older methods treated every word as an independent symbol. To them, “house”, “houses”, and “housetop” were three completely unrelated things. They had to learn each form separately, and when faced with an unknown word, they simply gave up. For German this is a real problem, because the language compounds and inflects endlessly.

FastText solves this through fragments. Because “housetop” contains the letter sequence “house”, it automatically ends up close to “house”. Even a completely made-up word like “housetoptileworks” gets a usable number sequence. The model guesses from the parts what the whole might mean.

The second reason for its popularity is speed. FastText trains on an ordinary laptop in minutes, whereas larger language models need data centers and days. Meta released ready-made models for 157 languages free of charge. This made FastText the standard tool for years for anyone without expensive hardware.

From letter sequences to number sequences

Training follows a simple basic idea. The program slides a window over huge amounts of text, for example the entirety of Wikipedia. For each word, it looks at which words appear directly next to it. It then adjusts the number sequences until words with similar neighborhoods have similar numbers. Whatever keeps showing up next to “barks” and “leash” ends up in the neighborhood of “dog”.

The difference from its predecessors lies in the intermediate step. FastText first cuts each word into overlapping letter pieces, so-called character n-grams. From “dog”, for example, come “do”, “og”, “dog”. Each piece gets its own number sequence. The number sequence of the whole word is then the sum of its parts.

There is one important limitation worth knowing. FastText always assigns the same number sequence to a word, regardless of the sentence. The “bank” of a river and the “bank” where you keep your money share a single entry. Modern language models like the one behind ChatGPT, by contrast, recalculate for each occurrence and take the whole sentence into account. But that makes them thousands of times more computationally expensive.

FastText in use for search and spam filters

Most commonly, one encounters FastText as a language detector. When a browser asks whether it should translate a page, someone first has to determine the language. The ready-made FastText model for this is only a few megabytes in size and decides in milliseconds. Large text collections used to train modern AI are also pre-sorted by language using it.

In companies, FastText often works behind the scenes in unremarkable text tasks. Support inquiries are automatically routed to the right department. Product descriptions are sorted into shop categories. Spam filters and search in online shops use it so that “sneaker” and “athletic shoes” are treated as related.

In the news, the name appears less often today because large language models are grabbing all the attention. But FastText hasn’t disappeared. For simple sorting tasks it remains fast, resource-efficient, and accurate enough. One common misconception, by the way, is that FastText is a chatbot. It doesn’t write texts, it only classifies them.