Stop Words

In every English text, certain words appear extremely often: “the,” “and,” “is,” “in,” “to,” “of.” They are necessary for grammar, but on their own they say little about the topic. When a computer program is supposed to figure out what a text is about, such words tend to get in the way rather than help. That’s why, early on, people started simply deleting them before analysis. These removed words are called stop words. Which words belong on that list is defined in a fixed list that someone compiled beforehand.

What removing filler words achieves

The practical benefit is, first of all, efficiency. In a typical English text, roughly 30 to 40 percent of all words consist of a handful of such everyday words. Removing them means significantly less data to store and search through. For classic search engines, which had to sift through millions of documents in fractions of a second, this was a concrete advantage.

The second benefit concerns the quality of the results. Many older methods evaluate a document based on which words appear especially often in it. Without cleanup, the answer would always be the same: “the,” “and,” “is.” Only once the filler words are removed do the meaningful terms stand out. A word cloud generated from a newspaper article only becomes readable this way.

It is important to note, however, that stop words are not a law of nature but a decision. There is no official, universally valid list. Different software libraries use different lists — sometimes 100 words, sometimes more than 600. Anyone wanting to reproduce an analysis therefore needs to know which list was used.

From word list to cleaned text

The process is remarkably simple. First, the text is broken down into individual words, a step experts call tokenization. Then the program goes through word by word and compares each one against the stop word list. If the word is on the list, it gets removed. So “The company’s stock rose sharply in May” becomes something like “company stock rose sharply May.”

This is often combined with further cleanup steps. Everything is converted to lowercase, punctuation is removed, and words are reduced to their base form. “Rose” then becomes “rise.” The goal is to prevent related forms of the same word from being counted as different terms.

This is exactly where the best-known pitfall lurks. Meaning sometimes resides precisely in the small words. “The price is not rising” and “The price is rising” mean the opposite of one another — once “not” is removed, both sentences become identical. Search queries like “To be or not to be” also consist almost entirely of stop words. This is why modern language models, such as those behind chatbots, remove nothing. They read every piece of text and learn on their own how much weight to give a word in a given context.

Where stop word lists are still used today

This method is encountered wherever texts need to be sorted quickly and cheaply. Spam filters, keyword extraction in editorial systems, and topic analysis of large document collections frequently rely on it. The word clouds seen in presentations or on news sites are also almost always the result of stop word removal.

In news coverage about AI, the term usually comes up as a point of contrast. When describing how today’s systems differ from the text processing of the 2000s, doing away with stop words is a typical example. In the past, the computer was taught what was unimportant. Today, that judgment is left to the system itself.

Anyone who does their own programming will quickly come across this: toolkits like NLTK or spaCy come with ready-made English stop word lists that can be applied with a single line of code. For school projects or simple text statistics, this remains a sensible first step. One should just check whether the specific question at hand requires a word that the list happens to discard.