When people talk about the cost of AI, they’re often talking about two completely different things without realising it. One is the eye-watering cost of creating a model like GPT-4 or Gemini. The other is the cost of using that model every time someone asks it a question. These two phases have names: training and inference.
Understanding the difference clears up a lot of confusion — about why AI companies spend so much, why your AI bill grows with usage, and which of these costs your organisation is actually likely to pay. Let’s unpack both.
Training: where the model learns
Training is the phase where a model learns from data. For a large language model, that means processing an enormous amount of text and adjusting billions of internal settings (called parameters) until it gets good at predicting what comes next. We looked at how this learning works in How Does a Machine Learn?.
Training a frontier model is enormously expensive. Stanford’s 2024 AI Index estimated that OpenAI’s GPT-4 used around US$78 million worth of compute to train, and Google’s Gemini Ultra around US$191 million[1]. And that’s just the computing power — it doesn’t include salaries, data or the many experimental runs that never become products.
Why so much? Research has found that language model performance improves predictably as you increase model size, the amount of training data and the amount of compute used for training[2]. Later work from DeepMind showed that many models were actually undertrained for their size, and that model size and training data should grow together — their smaller “Chinchilla” model, trained on more data, outperformed much larger models[3]. Either way, the recipe for a better model has meant more data and more compute, which means bigger training bills.
Inference: where the model works
Inference is what happens after training, every time the model is actually used. You send a prompt, the trained model processes it and generates a response. NVIDIA puts it neatly: inference is where capabilities learned during training are put to work, applying what’s been learned to new, unseen data[4].
Each individual inference is cheap compared with training — often a fraction of a cent. But it happens every time anyone, anywhere, uses the model. Millions of users asking millions of questions adds up very quickly.
An analogy: training is like putting someone through years of university. Inference is that graduate answering questions at work every day. The degree is a big one-off cost. The daily work is a small cost that never stops.
Side by side
| Training | Inference | |
|---|---|---|
| What happens | The model learns patterns from data | The trained model answers a request |
| How often | Occasionally (per model version) | Every single time it’s used |
| Cost shape | Huge, up-front, one-off | Small per request, ongoing, grows with usage |
| Who usually pays | AI labs and large tech companies | Whoever uses the model — including you |
| How you see it | Built into the provider’s pricing | Per-token API charges or subscription fees |
Which one does your business pay for?
For almost every organisation, the answer is inference. Unless you’re an AI lab, you won’t train a large language model from scratch. You’ll use a model someone else has trained — through a product like ChatGPT or Copilot, or through an API that charges per token (see What Are Tokens?).
That changes how you should think about AI costs. You’re not facing one big build cost; you’re taking on a running cost that rises with every user and every request. That’s fine — it means you can start small — but it needs to be modelled properly before you scale. I’ve written about exactly this in The Hidden Risk of Pure AI Products: Token Cost at Scale, and the Cost Analyser lets you project it with your own numbers.
What about “training it on our data”?
When vendors talk about training a model on your data, they usually mean something much smaller: fine-tuning an existing model with a modest set of your examples, or more commonly, simply giving the model your documents at inference time so it can answer from them. The second approach is called retrieval-augmented generation, and I’ll cover it later in this series. It’s worth asking any vendor exactly which one they mean, because the cost, effort and data implications are very different.
Both phases run in data centres, and the electricity adds up. The International Energy Agency estimates data centres used about 1.5% of the world’s electricity in 2024 and projects that their consumption will more than double by 2030, with AI a major driver[5]. As inference volumes grow, the efficiency of everyday use matters more and more — another reason to use the smallest model that does the job well.
Why this distinction matters
- Budgeting: plan for an ongoing operating cost that scales with use, not a one-off project cost.
- Model choice: a smaller, cheaper model is often good enough for simple tasks, and saves on every single request.
- Speed: inference speed is what your users feel. Bigger models generally respond more slowly.
- Vendor conversations: knowing the difference helps you ask sharper questions about pricing and “training on your data”.
What I want you to take away
- Training is learning; inference is using. Training builds the model once; inference runs it every time someone asks a question.
- Training is massively expensive — tens to hundreds of millions of dollars of compute for frontier models — but it’s the AI labs that pay it.
- Your organisation almost certainly pays for inference, as a running cost that grows with usage.
- “Training on your data” usually means fine-tuning or retrieval, not building a model from scratch. Ask vendors which.
- Model the running cost before you scale, and use the smallest model that does the job well.
Working out where AI fits in your business?
I help Australian teams cut through the hype and pick AI projects that actually pay off. If you'd like a second opinion on an idea, I'm happy to talk it through.
Get in touch →References
- Stanford Institute for Human-Centered AI (2024). AI Index Report 2024. Stanford University. https://hai.stanford.edu/ai-index/2024-ai-index-report
- Kaplan, J., et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361. https://arxiv.org/abs/2001.08361
- Hoffmann, J., et al. (2022). Training Compute-Optimal Large Language Models. arXiv:2203.15556. https://arxiv.org/abs/2203.15556
- NVIDIA. What’s the Difference Between Deep Learning Training and Inference? NVIDIA Blog. https://blogs.nvidia.com/blog/difference-deep-learning-training-inference-ai/
- International Energy Agency (2025). Energy and AI: Executive summary. IEA, Paris. https://www.iea.org/reports/energy-and-ai/executive-summary