Why AI Is So Expensive: GPUs, Training Costs & Inference Economics

Every AI reply you've ever received cost someone real money — actual electricity, on actual hardware, in an actual building. This guide is about that money, because it quietly explains half the AI industry's behaviour: the rate limits, the subscription tiers, the per-token pricing, the trillion-dollar data-centre buildout. Underneath it all sits one mechanical fact: a model generates its answer token by token, re-reading everything in its context window as it goes. So the question that starts everything: what does each of those tokens actually cost?

The answer is genuinely surprising in both directions. Training a frontier model costs more than a skyscraper — and yet the price of using one has been falling faster than almost any technology in history. Let's follow the money.

It all starts with a very unusual chip

Recall what's inside a model: billions of numbers — parameters — and producing a single token means pushing your input through essentially all of them. That's billions of multiplications and additions, per token, and a chatty reply is hundreds of tokens.

An ordinary computer processor (a CPU) is the wrong tool for this. A CPU is like a brilliant chef: it does one complicated thing at a time, very fast. But the model's math isn't complicated — it's simple multiplication repeated billions of times, and what you want for that is ten thousand mediocre cooks working in parallel. That's a GPU.

GPUs (graphics processing units) were built to colour millions of screen pixels simultaneously, which turns out to be the same shape of problem as neural-network math: huge grids of numbers, all processed at once. The AI boom runs almost entirely on them — and they are neither cheap nor easy to get:

• A single top-end AI GPU costs tens of thousands of dollars — the price of a car, for one chip.
• A serious training cluster contains tens of thousands of them, wired together with specialised networking so they behave like one giant computer.
• Each one draws roughly as much power as a household microwave running non-stop — and removing that heat adds a hefty cooling bill on top.
• Demand has outstripped supply for years. Companies don't just budget for GPUs; they queue for them.

So the raw ingredient is expensive. Now look at the two very different bills it generates.

Bill #1 — Training: the mortgage

You've seen how a model is trained: show it a huge slice of the internet, let it predict the next token, nudge billions of parameters after every mistake, repeat astronomically many times. Now price it.

Training a frontier model means running thousands of GPUs, around the clock, for weeks to months. Reported and estimated figures for recent frontier models put the compute bill alone in the hundreds of millions of dollars, and the trend line points upward. And compute is only part of it:

• Data — licensing content, cleaning it, and paying humans to write examples and rate outputs for the feedback phase of training.
• People — elite research and engineering teams, among the most expensive talent in the world.
• Failures — training runs crash, ideas don't pan out, and smaller experimental runs burn money long before the big one starts.
• Electricity — months of megawatts, plus the cooling to match.

The key economic fact about training: it's a one-time, upfront cost, like building a factory. Spend nine figures, and what you get is a single file of parameters — the finished model. That file cost a fortune to create and costs almost nothing to copy.

Which sounds like great news — build once, sell forever! Except for the second bill.

Bill #2 — Inference: the utility bill that never stops

Inference is the industry's word for using the model — everything that happens when you hit send. And here's the part most people miss: as the inference chapter of our LLM guide covers in depth, answering you means running the whole model, once per token of the reply.

A 500-token answer isn't one computation — it's five hundred, each one a pass through billions of parameters on hardware that costs more per hour than a lawyer. Multiply by millions of users sending messages all day, and inference becomes the cost that dominates everything:

• Training is huge but finite. Inference is smaller per event but never ends — it scales with success. The more popular the product, the bigger the bill.
• Long conversations cost more, because the model re-reads the entire context window while generating. This is why providers charge for input tokens too, not just the reply.
• This is also exactly why per-token pricing exists. It's not a marketing choice — it mirrors the physics. The model does work token by token, so the meter runs token by token.

So why do prices keep falling?

Here's the twist. Given everything above, you'd expect using AI to get more expensive. Instead, the price of a given level of capability has collapsed — what cost dollars per million tokens a couple of years ago now costs cents. Several forces stack up:

• Better hardware. Each GPU generation does several times more AI math per watt and per dollar than the last.
• Smaller models got good. Techniques like distillation — training a compact model to imitate a giant one — mean yesterday's frontier quality now fits in a model a tenth the size, which costs a tenth as much to run.
• Quantization. Storing parameters with less numeric precision (think rounding to fewer decimal places) shrinks the model and speeds it up with barely any quality loss.
• Batching. GPUs can serve many users' requests simultaneously in one pass, the way a bus is cheaper per passenger than a taxi.
• Caching. If the start of your conversation hasn't changed, providers can reuse the computation from last time instead of re-reading everything from scratch — which is why "cached input" is billed at a steep discount.

But there's a counter-force, and it's why the industry's total spending keeps climbing even as tokens get cheap: we immediately spend the savings on more tokens. Reasoning models that think for thousands of tokens before answering. Agents that work for minutes at a time. Contexts stuffed with entire codebases. Cheaper tokens don't shrink the bill — they expand what we ask for. Economists call this the Jevons paradox: make a resource more efficient, and consumption rises.

Recap

• Generative AI runs on GPUs — chips built for massive parallel math, costing tens of thousands of dollars each, power-hungry, and scarce. "Compute" is the industry's limiting resource.
• Training is the mortgage: a one-time, enormous cost — thousands of GPUs for months, plus data, people, and power — reported in the hundreds of millions for frontier models.
• Inference is the utility bill: every token of every reply is a full pass through the model, so the cost never stops and scales with popularity. That's why pricing is per token and free tiers have limits.
• Prices per token have fallen dramatically anyway — better chips, distillation, quantization, batching, and caching keep cutting the cost of a given quality level.
• Total spend still rises, because cheaper tokens make us use far more of them — longer contexts, reasoning, agents. Efficiency wins per token; ambition wins overall.

That's the whole economic machine in one pass. The rest of this guide takes each piece apart properly — and it starts with the hardware everything else is built on: why AI runs on GPUs, and why one chip costs as much as a car.