Source Poisoning

Programs like ChatGPT did not invent their knowledge themselves. They have read vast amounts of text from the internet: websites, forums, news articles, encyclopedias. Some of these programs additionally search the web live during a conversation. Source poisoning exploits exactly this. Someone deliberately publishes false content in places that are read by such programs. The goal is for the misinformation to later come back out as a seemingly reliable answer. The name literally means “poisoning of the source.”

Why poisoned sources are hard to detect

If you open a dubious website yourself, you can often tell from its appearance alone that something is off. An AI answer loses this clue. The text arrives in a calm, matter-of-fact tone, without advertising banners and without spelling mistakes. The same claim suddenly appears credible as a result. It is precisely this packaging that makes source poisoning effective.

On top of that comes reach. A manipulated page might reach a few hundred people. But if it is picked up by a widely used AI assistant, it can appear in millions of answers. The effort for the attacker stays the same, while the impact multiplies. Experts call this a leverage effect.

Particularly troublesome is the persistence. Once a piece of misinformation has made its way into a model’s training material, it remains lodged in its “memory” afterward. You can delete the original page, but the model still knows it. A simple correction afterward is barely possible, because no one can say exactly which sentence came from which source.

The path from the false website to the answer

There are two main routes. In the first, the attacker targets training. They put thousands of texts containing the same claim online before a new model is built. When the data is collected, these texts get swept up along with it. Because the claim appears so often, the model treats it as established knowledge.

The second route targets the search that happens during a conversation. Many assistants pull fresh results from the internet for current questions and summarize them. This process is called retrieval. Anyone who manages to get their page ranked among the top results hands the AI the ingredients for its answer directly. Here you don’t need a thousand pages — sometimes one well-placed page is enough.

A well-known trick involves networks of hundreds of automatically generated news sites. They look like real local newspapers and copy from one another. To a human, this is transparent, but to a crawling program it looks like broad confirmation. This should be distinguished from prompt injection: there, someone hides commands in a text to steer the AI’s behavior. Source poisoning does not change the instructions, but the knowledge.

Poisoned sources in politics, medicine, and corporate knowledge

The most widely discussed case is disinformation. Investigations by fact-checking organizations have repeatedly shown that chatbots repeat political falsehoods originating from such fake networks. Health questions are similarly vulnerable, where fabricated studies get cited. Companies are affected too: competitors or extortionists can deliberately plant reputational damage so that the AI parrots it.

A second area is program code. AI assistants suggest ready-made building blocks when coding, which they know from public repositories. Attackers publish blocks there with malicious functions and matching names. If such a suggestion is adopted, the malicious code ends up in real software. Security firms now include this attack vector in their risk lists.

As a countermeasure, providers check the origin of their data and favor sources with a good reputation. Some systems provide links to their sources. These links are the most important starting point for you. If an answer contains a surprising claim, it is worth checking who actually runs the site being cited.