Sharding

Some amounts of data are too large for a single computer. Sharding is the solution to this: you break the dataset into pieces and place each piece on a different machine. Such a piece is called a shard. Each machine then only knows its own portion, not the whole. Together, the pieces make up the complete dataset again. The comparison with a library fits well: instead of cramming all the books into one room, you distribute them across several buildings, and a directory tells you where each book is located.

Why a single machine eventually isn’t enough

A computer has fixed limits. Its working memory is finite, so is its hard drive, and it can only answer a certain number of requests per second. For a long time, these limits were pushed back by buying more expensive machines. This is called vertical scaling. Eventually, though, there is no bigger machine left to buy.

Sharding takes the other path: you use many ordinary machines instead of one giant one. This is usually cheaper and has hardly any upper limit. When the dataset grows, you simply add more machines. Large services such as social networks or online shops could not exist without this principle.

A second advantage is fault tolerance. If one machine fails, only the data on that particular shard is affected. The rest of the system keeps running. This does not fully prevent data loss, however, which is why sharding is almost always combined with copies of the individual shards.

How the data is distributed across the shards

The crucial question is which rule is used to split the data. This rule is called the shard key. For example, you could distribute user accounts by the first letter of the name: A to F on machine one, G to M on machine two, and so on. More often, a kind of checksum is calculated from the user number, and the target machine is derived from that. This distributes the load more evenly, because there are far more Smiths than Xaviers.

A poorly chosen shard key quickly backfires. If the most active users all end up on the same machine, that machine becomes overloaded while the others do almost nothing. Experts call this a hotspot. It also becomes tricky when a single request needs data from multiple shards at once. Such queries are noticeably slower, because the system has to query several machines and assemble the answers.

In artificial intelligence, the principle works similarly, but it concerns not data but the model itself. Large language models have hundreds of billions of adjustable numerical values, so-called parameters. These do not fit on a single graphics chip. So they are distributed across many chips, which constantly exchange intermediate results during computation. This is referred to as model sharding.

Sharding in apps, data centers, and cryptocurrencies

In everyday use, you don’t notice sharding at all, and that is exactly the goal. Anyone scrolling through Instagram or placing an order is, in truth, talking to one machine among thousands. Online games also use this principle. There, the shards are often simply called servers or worlds, and players on different worlds never meet.

In tech news, the term comes up especially in connection with cryptocurrencies. Blockchains are normally slow because every participant verifies every transaction. Sharding is meant to change this by having groups of machines process only a portion of the transactions. Ethereum has been pursuing this approach for years, though in a heavily modified form.

A common misconception is that sharding is the same as a backup. That’s not true: a backup is a copy of the same data, whereas a shard is a different subset. Sharding increases capacity, copies increase safety. Serious systems need both.