Content Moderation

On large internet platforms, users upload huge amounts of text, images, and videos every minute. Some of it is prohibited or unwanted: violent videos, scams, hatred against groups of people, stolen films. Content moderation is the work of finding and removing exactly these kinds of content. Responsible for this are, on the one hand, programs that automatically check posts, and on the other hand, paid reviewers who look at disputed cases. The basis for this is two sets of rules: the laws of the respective country and the platform’s own self-imposed rules, often called community guidelines. Whoever violates them will have their post deleted, shown to fewer people, or given a warning label.

Between over-deletion and a garbage dump

Without moderation, a platform becomes unusable within a short time. Advertising spam, fraudulent ads, and harassment crowd out normal posts because they can be produced cheaply in bulk. No advertiser wants their ad to appear next to a beheading video. Moderation is therefore not idealism for operators, but a precondition for the business to function at all.

At the same time, private companies are deciding here what millions of people get to see. If they delete too much, legitimate opinions, satire, or reports from war zones disappear. If they delete too little, hate speech and disinformation remain online. Both types of errors cannot be reduced to zero at the same time: the stricter the filtering, the more real violations are caught, but also the more innocent people. This trade-off is at the core of almost every public debate on the topic.

Added to this is the toll on the people who do this work. Moderators spend hours sifting through material that others should never have to see. Reports of psychological strain and lawsuits against service providers have repeatedly brought the topic into the news in recent years.

Filter, queue, appeal

As a rule, moderation works in several stages. First, software checks every new post. For known material, such as an already banned video, a digital fingerprint helps: a short identifying code is calculated from the file and compared against a list of known violations. For new content, a trained AI model estimates how likely a violation is. It has previously learned from many examples rated by humans what, say, an insult typically looks like.

The system removes clear-cut cases immediately. Uncertain cases end up in a queue for humans, as does everything users report themselves. A person looks at the post and decides according to a guidebook that is often hundreds of pages long. There is usually an appeals process for decisions, in which the case is reviewed again.

A common misconception is that the software understands the meaning of a post. It recognizes patterns, not context. A quote that criticizes hate speech contains the same words as the hate speech itself. Precisely for this reason, humans remain in the loop, even though automated systems catch the bulk of the volume.

From TikTok to the Digital Services Act

Every reported comment, every vanished post, and every warning label under a video is content moderation in action. Spam filters in email inboxes and the rules in online games are also part of it. Anyone who approves posts on a platform themselves, for instance in a school group or a forum, is doing the same thing on a small scale.

In the EU, the Digital Services Act, or DSA for short, has regulated since 2024 how large platforms must handle this. They must offer reporting channels, justify deletions, allow for appeals, and regularly publish reports on their decisions. Violations can result in fines of up to six percent of worldwide annual revenue.

In business news, the term usually comes up for one of two reasons: either a company is cutting its moderation teams and replacing them with AI, or a regulator is opening proceedings over overly lax controls. Both move stock prices, because they are directly tied to advertising revenue and potential fines.