Cloud GPU

Cloud GPU

A cloud GPU is a graphics processor that you rent over the internet instead of owning it physically. AI companies and researchers use them to train computation-intensive models without having to buy their own hardware.

A graphics processor — GPU for short — is a chip that was originally developed for video games. It can perform thousands of simple computing operations simultaneously, which makes it especially useful for AI training. Those who don’t own a GPU themselves can rent one over the internet: that’s a cloud GPU. You pay for the time you use it — often by the hour — and return it afterward. The hardware sits in a data center somewhere in the world.

Why cloud GPUs are democratizing AI development

A powerful GPU costs several thousand euros to buy. The chips that train large AI models cost even more — over 30,000 euros apiece — and you often need thousands of them at once. Only a few companies can afford that. Cloud GPUs change this: instead of a million-euro investment, you book computing capacity only for the hours you actually need it.

For startups and universities, this is crucial. They can start AI experiments without first raising capital for hardware. If an experiment fails, only rental costs for a few hours are incurred — not a permanent loss. This difference has contributed to the fact that, in recent years, many small teams have been able to release serious AI models.

What’s technically behind the rented GPU

Data centers house racks full of servers with GPUs built in. The provider connects these chips via very fast internal networks, so that many GPUs can work together on a single model. The user accesses them via a software interface — they don’t see any hardware, but instead reserve a certain number of chips for a certain duration.

Billing works similarly to a taxi meter: the clock runs as long as the chip is working. Some providers also sell so-called spot instances — computing time that nobody else currently needs — at a cheaper price. The downside: the provider can reclaim this capacity at any time if a customer willing to pay more shows up. Running calculations are interrupted in the process.

A common misconception is that cloud GPU and cloud computing are the same thing. Normal cloud servers use general-purpose processors — so-called CPUs — which are built for many different tasks. A cloud GPU, on the other hand, is specialized hardware that must be explicitly requested and costs significantly more than a standard server.

Cloud GPUs in products and headlines

The largest providers of cloud GPUs are Amazon Web Services, Microsoft Azure, and Google Cloud. In addition, there are specialized providers such as CoreWeave or Lambda Labs, which sell exclusively GPU computing time. When news articles report that an AI company is “investing millions in computing capacity,” they almost always mean rental time in such data centers.

ChatGPT, Google’s Gemini, and similar systems run on GPUs in the cloud for every single user request. The model was trained once, but so-called inference — that is, computing an answer to a specific question — happens in real time on rented or owned GPU hardware. The operating costs of such services therefore depend directly on how much computing time is consumed per answer.

Recently, cloud GPUs have also become politically relevant. The United States has introduced export restrictions on certain chips so that countries like China cannot buy high-performance GPUs. In response, regulators are examining whether access via cloud services could have the same effect — that is, whether someone is indirectly using banned hardware by simply renting it remotely.

Subscribe free. Unsubscribe the second it sucks.

High-signal news across AI, business, UX, and tech. Every morning.