Why You Pay Per Token: The Economics of AI Inference

The training bill was the mortgage — enormous, but paid once per model. This chapter is about the other bill: inference, the cost of actually answering people. It's smaller per event and infinitely recurring, it dominates the economics of every AI product you use, and it explains the strangest-looking thing in AI: the price sheet. Why per token? Why does output cost more than input? Why is "cached input" nearly free? By the end of this chapter, every line of that price sheet will look inevitable.

The meter matches the machine

Start from the mechanics you already know: a model generates its reply one token at a time, and producing each token means streaming all of the model's parameters through a GPU once more. A 500-token answer is five hundred full passes through the network.

So the cost of serving you is literally proportional to tokens processed. Per-token pricing isn't a business-school invention — it's the physics of the machine, forwarded to your invoice. The meter runs exactly as fast as the machine does.

But look closely at any AI price sheet and you'll see the interesting part: input tokens and output tokens have different prices — output typically several times more expensive. That difference is the key to everything, and it comes from the two phases of inference.

Prefill and decode: reading vs writing

When your message arrives, the model does two very different kinds of work:

• Prefill (reading). The model processes your entire prompt — every token of it — in one parallel sweep. This is exactly the kind of do-everything-at-once work GPUs were built for, so it's fast and comparatively cheap. That's your input tokens.
• Decode (writing). Generating the reply can't be parallelised, because each token depends on the one before it. The model must run fully, wait for a token, append it, and run again — hundreds of sequential round trips, each one bottlenecked on streaming billions of parameters through memory. That's your output tokens.

Reading is parallel; writing is one-word-at-a-time. That's why output tokens cost more than input tokens — not markup, mechanics.

The two tricks that make it affordable

If every user got a dedicated GPU, AI products couldn't exist at these prices. Two pieces of serving engineering close the gap:

• Batching: the bus, not the taxi. A GPU streaming the model's parameters for your token can apply that same stream to dozens of other users' tokens in the same pass, almost free. Providers therefore group concurrent requests into batches — a bus, where the enormous per-trip cost is split across every passenger, instead of a taxi carrying one. This is the single biggest lever on cost per token, and it's why busy providers can charge less: fuller buses.
• Prompt caching: don't re-read what hasn't changed. If the start of your prompt is identical to last time — the system instructions, the document you're chatting about, the conversation so far — the provider can keep the prefill computation for that prefix and reuse it, skipping the re-read entirely. That's why price sheets list cached input at a fraction of the normal input price: you're paying for memory storage rather than fresh computation. (It's also why well-built apps keep the stable part of a prompt at the front.)

Free tiers, rate limits, and the economics made visible

Now every quirk of AI products decodes cleanly:

• Free tiers have quotas because every free message is GPU time someone pays for. The quota is the budget.
• Rate limits protect the provider's fixed pool of GPUs — the machines batch beautifully, but they don't stretch. Too many simultaneous users means someone waits.
• "You've been switched to a faster model" means a smaller, cheaper model — a fraction of the parameters, a fraction of the cost per token.
• Subscriptions vs API pricing are the same bill in two shapes: a flat fee with fair-use limits for humans, and a raw per-token meter for software. Heavy users are why "unlimited" plans quietly aren't.
• Success is expensive. Unlike normal software — where an extra user costs nearly nothing — an AI product's costs scale with usage. A viral week isn't just a happy traffic spike; it's an invoice.

Recap

• Per-token pricing mirrors the machine: one full model pass per generated token, so cost is proportional to tokens.
• Inference has two phases: prefill (reading your prompt, parallel, cheap) and decode (writing the reply, sequential, expensive) — which is why output tokens cost more than input.
• Models have no memory, so each turn re-reads the whole conversation — long chats cost more, and that's what input billing reflects.
• Batching (many users per GPU pass) and prompt caching (never re-read an unchanged prefix) are the two levers that make serving affordable — caching is why "cached input" is billed at a steep discount.
• Free-tier quotas, rate limits, and model fallbacks are all the same thing: a bill that scales with popularity, managed in public.

All of this metering happens on real machines in real buildings drawing real power — and the buildings have become a story of their own: data centres & power.