Deduplication

Anyone who collects very large amounts of data inevitably ends up collecting duplicates as well. The same newspaper article appears on twenty websites. The same photo sits in three folders. Deduplication is the process by which such copies are detected and removed down to a single instance. The term comes from Latin: "duplex" means double, and the prefix "de" means away with it. The word is used in two areas: when cleaning up data that a computer program is meant to learn from, and when storing files on hard drives.

Why models choke on duplicates

Modern language models learn from huge amounts of text, most of which comes from the internet. Such collections contain a great deal of redundancy. News agencies deliver the same text to hundreds of portals. Legal texts, imprints, and cookie notices repeat themselves millions of times over. Studies of well-known datasets have shown that a noticeable share of texts occurs more than once.

For training, this is harmful. Whatever a model sees especially often gets especially deeply imprinted. With duplicates, that means the model memorizes individual passages of text instead of grasping the general patterns of language. It can then reproduce entire paragraphs word for word. That is legally delicate and dangerous from a data-protection standpoint if the text contains personal information.

On top of that, there is a plain cost argument. Compute time on specialized graphics chips is expensive. If ten percent of the training data are copies, you burn ten percent of the budget on content the model already knows. Another point concerns evaluation: if a test example accidentally also turns up in the training material, the model appears better on the test than it actually is. Experts call this data leakage.

From exact copies to near-identical texts

The simplest case is exact copies. For these, a kind of fingerprint is calculated for each file, a short sequence of numbers derived from the content. Two identical pieces of content yield the same fingerprint. So instead of comparing file against file, one only needs to look up short numbers in a list. This still works quickly even with billions of documents.

Near-duplicates are trickier. Two articles are 95 percent identical but differ in the headline or in an ad banner at the end. Their fingerprints then turn out to be completely different. Here, methods are used that break a text down into many overlapping word groups and estimate the overlap between two texts. The best-known of these is called MinHash. If the estimated overlap exceeds a defined threshold, the text counts as a duplicate.

This threshold is a trade-off. Set it too strict, and you discard texts that merely happen to sound similar, such as two weather reports. Set it too loose, and redundancy remains. In storage technology, the principle works in a similar way, just at the block level: a hard drive breaks files down into small chunks and stores each chunk only once. If it occurs again, the system merely stores a reference to it.

Dedup in the data center and on the phone

Deduplication is most visible in a smartphone's photo management. It recognizes burst shots and identical images and suggests deleting them. Cloud storage also uses the same principle: if a thousand users upload the same piece of music, the provider often stores it on disk only a single time. Corporate backup systems regularly save large portions of their storage needs this way, since usually only little changes between two backups.

In AI news, you'll mostly encounter the term in connection with datasets. When a lab describes the provenance of its training data, deduplication is almost always mentioned as a processing step, alongside language filtering and the removal of spam. Often a figure is given for what percentage of the original data volume remained.

A common misconception is that deduplication is merely cosmetic. In fact, experiments show that deduplicated datasets lead to better models at the same data volume. The term should be distinguished from compression: compression shrinks a single file by encoding patterns within it more cleverly. Deduplication, by contrast, looks across many files and removes repetitions between them. In practice, the two are combined.