Why AI Runs on GPUs (and Why One Chip Costs as Much as a Car)

In the overview, we kept saying one word: GPUs. Every cost in AI traces back to these chips somehow, so before we can price a training run or a token, we need to answer the real question: why this hardware? Your laptop has a perfectly good processor. Why does AI demand something different — and why does the different thing cost as much as a car?

The shape of AI's math

Recall what's inside a model: billions of parameters, and generating a token means pushing numbers through essentially all of them. Strip away the mystique and the actual work is astonishingly mundane — multiply two big grids of numbers together, add the results, repeat. No individual step is clever. A ten-year-old could do any one of them.

The problem is the quantity: billions of multiplications per token, hundreds of tokens per reply, millions of users. And here's the crucial property — the multiplications don't depend on each other. Row 1 times column 1 doesn't need to wait for row 2 times column 2. In principle, you could do them all at the same time.

That single property decides the hardware.

CPUs: one brilliant chef

A CPU — the processor in your laptop — is engineered for sequential work: run this instruction, then that one, where each step may depend on the last. It's a brilliant chef: give it a complicated recipe full of "if this, then that" and it flies through it, one step at a time, at billions of steps per second.

A CPU has a handful of very powerful cores. For almost everything computers do — running your browser, your spreadsheet, this website's server — that's exactly right.

But hand the chef ten billion identical multiplications and the brilliance is wasted. It doesn't matter how fast you do them one at a time; you needed ten thousand mediocre cooks doing them all at once.

GPUs: ten thousand line cooks

A GPU was invented for a different job entirely: video game graphics. Drawing a frame means computing the colour of millions of pixels — and each pixel's math is simple and independent of its neighbours'. So GPU designers made the opposite trade to CPU designers: instead of a few powerful cores, thousands of small, simple ones, all executing the same operation on different pieces of data simultaneously.

Sometime in the late 2000s, researchers noticed that "millions of simple, independent calculations" describes neural networks just as well as it describes pixels. The fit was almost eerie:

Modern data-centre GPUs barely resemble graphics cards any more. They have tensor cores — circuitry purpose-built for exactly the grid-multiplication at the heart of Transformers — and they're sold to data centres by the tens of thousands, not to gamers one at a time.

The bottleneck nobody expects: memory

Here's the counterintuitive part. On modern AI chips, the math itself is often not the limiting factor. The limit is getting the numbers to the math — because to produce one token, the chip must stream the model's entire set of parameters out of memory and through its cores. Billions of numbers, fetched per token, every token of every reply.

A useful mental picture: the tensor cores are a monstrous industrial kitchen, and memory bandwidth is the loading dock. Most of the engineering (and cost) goes into making the loading dock keep up with the kitchen.

So why does one chip cost as much as a car?

Stack up the reasons:

• The most advanced manufacturing on Earth. These chips are fabricated at the cutting edge of what's physically possible, in fabs that cost tens of billions of dollars to build — and nearly all of the world's leading-edge capacity sits with a single manufacturer, TSMC. Scarcity starts at the factory.
• Exotic memory and packaging. HBM stacks and the advanced packaging that bonds them to the chip are themselves supply-constrained and expensive.
• One dominant seller. NVIDIA designs the overwhelming majority of AI training chips, which means pricing power — its reported margins on data-centre hardware are the envy of the industry.
• The software moat. Fifteen-plus years of AI code has been written for CUDA, NVIDIA's programming platform. Even when a rival chip looks competitive on paper, switching means rewriting and revalidating the software stack — so buyers mostly don't. The moat is code, not silicon.
• Demand from everyone at once. Every lab, cloud, enterprise, and government wants the same chips in the same years. Companies don't just budget for GPUs; they queue for them.

And you never buy just one

The final cost multiplier: frontier models are too big for any single GPU. The parameters alone can exceed one chip's memory, and training at frontier scale needs thousands of chips working for months.

So GPUs are deployed in clusters — thousands to tens of thousands of chips, connected by specialised networking (NVLink within a server, InfiniBand-class fabrics between them) fast enough that the cluster behaves like one enormous computer. That networking is its own major line item, and it's why the real unit of AI capacity isn't "a GPU" — it's a cluster, priced like infrastructure, not electronics.

That last box — megawatts — is a preview of where this guide is headed.

Recap

• AI's math is simple but astronomically plentiful, and the operations are independent — a perfectly parallel problem.
• CPUs are a few brilliant sequential cores; GPUs are thousands of simple parallel ones. Neural networks want the GPU, and modern AI chips add tensor cores built for exactly this math.
• The real bottleneck is often memory bandwidth — streaming billions of parameters through the chip per token — which is why AI GPUs carry expensive HBM memory.
• Chips cost tens of thousands each because of cutting-edge fabrication, exotic memory, a dominant seller with a CUDA software moat, and everyone demanding them at once.
• Frontier AI never runs on one chip: it runs on clusters of thousands, joined by specialised networking — infrastructure, not electronics.

Now you know what the machines are and why they're priced like exotic cars. Next question: what happens when you rent thousands of them for six months? The cost of training a frontier model.