Natural Language Processing

Computers naturally work with numbers, not sentences. Human language is at first just a string of characters without meaning to them. Natural Language Processing, roughly “machine language processing” in German, is the field of research that closes this gap. It encompasses all the methods by which a program takes in spoken or written language, analyzes it, and generates it itself. This includes translating a text, recognizing names in a newspaper article, or answering a question. The English abbreviation NLP has also become established in German.

Why language is so hard for machines

Language is ambiguous, and almost everywhere. The word “bank” can mean a financial institution or the edge of a river. Only context reveals which meaning applies. For humans this happens automatically; for a program it is a genuine computational task.

On top of that, we leave out a great deal. We use irony, allusions, and half-finished sentences, and yet the other person usually understands us anyway. A program does not have this world knowledge built in. It has to be derived from enormous amounts of text, and it remains full of gaps.

NLP matters because the largest part of human knowledge exists in the form of language. Emails, contracts, court rulings, medical reports, news articles: all text. Anyone who wants to make sense of this volume won’t get far without machines. No company can read millions of customer reviews by hand.

From grammar rules to language models

The first systems in the 1960s worked with hand-written rules. Experts entered grammar and dictionaries, sentence by sentence. This worked in narrow domains but failed at real language. Every exception required a new rule, and there are infinitely many exceptions.

Today, systems learn from examples instead. A program is given very large amounts of text and calculates from them which words typically appear together. In the process, each word is converted into a series of numbers that describes its meaning in relation to other words. Words with similar meanings receive similar number series. This turns language into something a computer can actually process.

The big leap came starting in 2017 with an architecture called the Transformer. It can weigh, for each word, which other words in the sentence are currently important. This made it possible to manage context across long stretches of text. Today’s large language models, like the ones behind ChatGPT, are based on this technology. However, language models are only one part of NLP, albeit the best-known one.

NLP in products and headlines

In everyday life, one encounters NLP constantly without noticing it. The spam filter in your inbox decides based on word patterns. Autocorrect on your phone suggests the next word. Search engines now understand entire questions instead of just individual search terms. Voice assistants first convert sound into text and then analyze that text.

In business, the most common application is analyzing opinions, often called sentiment analysis. Software reads reviews or posts on social networks and sorts them into positive, negative, or neutral. Banks and law firms also use NLP to search long contracts for specific clauses. This does not replace experts, but it considerably shortens the preliminary work.

In the news, the term often comes up where companies explain their text-based features. A common misconception here is equating NLP with artificial intelligence. NLP is only a subfield and describes the application area of language. A simple spam filter belongs to it just as much as a modern language model with billions of parameters.