S3 Bucket

Large internet companies don’t store their files on a hard drive in their own office. They rent storage space in someone else’s data centers, meaning halls full of computers that run around the clock. Amazon rents out such storage space under the name S3, short for Simple Storage Service. A bucket is a single named container there into which one throws files. Every company sets up as many of these containers as it likes and decides for itself who is allowed to look inside. Payment is based on the amount of data stored and the number of accesses.

Why half the internet sits in buckets

S3 launched in 2006 and was one of the first services with which Amazon rented out computing power. Today it is one of the most widely used storage services in the world. Anyone who starts a video on a streaming service very often fetches the data from such a bucket. Photos in online shops, company backups, and training data for AI models are also stored there.

The appeal lies in the carefreeness. A company doesn’t need to buy hard drives, plan for outages, or pay staff for maintenance. Amazon promises that data is practically never lost, because every file is stored multiple times in different locations. If an app grows from a thousand to ten million users, nobody has to rebuild anything.

The downside regularly shows up in the news. An incorrectly configured bucket is readable by anyone on the internet. Millions of customer records have already been made publicly accessible because a single switch was set to open. Such data breaches are among the most common security incidents of recent years.

Objects instead of folders

On a normal computer, files sit in folders that sit inside further folders. S3 works differently. Every file is a so-called object and gets a long name, the key. The key can contain slashes, for example photos/2024/vacation.jpg. This looks like folders, but it’s just text. In reality, everything lies flat side by side in a single container.

Every object is reachable via a web address. Programs query this address, similar to how a browser loads a webpage. In doing so, they send along a digital ID so that Amazon can check whether the access is permitted. Which accesses are permitted is defined by a rule list, the bucket policy. This is exactly where most mistakes happen.

A bucket is also tied to a region, for example Frankfurt or Northern Virginia. The data then physically resides in that area. This matters for data protection laws and for speed. Anyone who rarely needs very old data can move it to cheaper storage classes, which respond more slowly in return.

S3 in news, apps, and AI projects

As a user, you normally never see a bucket. Nevertheless, a good part of your daily life runs through one as soon as you use an app. In reports about data leaks, on the other hand, the term appears openly, usually in the phrase unprotected S3 bucket. It’s also present during outages: when S3 was disrupted for several hours in 2017, thousands of websites worldwide had problems.

In the AI industry, buckets are the usual place for massive data collections. A language model is trained with text amounting to many terabytes, and this text has to sit somewhere. The computers doing the training then fetch it piece by piece from storage. Finished models are also distributed this way.

S3 is not the only provider, but it is the most well-known. Google and Microsoft offer very similar services under different names. Because S3 is so widespread, its technical language has become an unofficial standard. Many competitors therefore deliberately speak the same language so that companies can switch without having to rebuild anything.