Ask an AI chatbot a question and you’ll get a fluent, confident, well-formatted answer. Most of the time it’s right. But every so often it will tell you something that is simply made up — a statistic that doesn’t exist, a policy your company never had, a book by an author who never wrote it — in exactly the same confident tone.
This is called hallucination, and it’s the single most important limitation to understand before you put AI anywhere near customers, legal documents or business decisions. In this post I’ll explain what hallucination is, why it happens, and the practical steps that keep it from causing real damage.
What is a hallucination?
In AI, a hallucination is when a model produces content that sounds plausible but is wrong or isn’t supported by the information it was given. A major survey of the topic describes large language models as prone to “generating plausible yet nonfactual content”[2]. The word plausible is the dangerous part. A hallucination doesn’t look like an error. It looks like every other answer.
Researchers tend to split hallucinations into two broad kinds[1][2]:
| Type | What happens | Business example |
|---|---|---|
| Getting the facts wrong | The answer conflicts with real-world facts, or invents facts that can’t be verified | Citing a regulation or court case that doesn’t exist |
| Not sticking to the source | The answer contradicts, or adds to, the document or instructions it was given | A summary of a contract that “includes” a clause the contract doesn’t contain |
The second kind surprises people most. Even when you give the model the exact document, it can still slip in details that aren’t there.
Why does it happen?
To understand hallucination, remember what a large language model actually does: it predicts the most likely next piece of text, over and over (see What is a large language model?). That has three consequences.
1. It’s optimised for plausible, not true
A language model learns the patterns of language from enormous amounts of text. It gets very good at producing text that sounds like a correct answer. But there’s no built-in fact-checker. If the most plausible-sounding continuation happens to be false, the model has no internal alarm that goes off.
2. Gaps get filled in
When the model doesn’t have the information — something obscure, something after its training data was collected, or something private to your organisation — it doesn’t necessarily stop. It fills the gap with something that fits the pattern.
3. Guessing gets rewarded
A 2025 paper from researchers at OpenAI and Georgia Tech offers a helpful way to think about this. They compare language models to “students facing hard exam questions” who guess when unsure rather than admit uncertainty[3]. Their argument is that the way models are trained and evaluated rewards guessing over saying “I don’t know” — much like a multiple-choice exam where a blank answer scores zero but a guess might score a point[3]. If the scoreboard rewards confident answers, you get confident answers, right or wrong.
Hallucination isn’t a bug that will be patched next month. It’s a side effect of how these models work. Newer models hallucinate less, but you should design every AI process on the assumption that some answers will be wrong.
How to reduce hallucinations
You can’t eliminate hallucination entirely, but you can make it much rarer and much easier to catch. These are the approaches I recommend, roughly in order of impact.
Ground the model in your own sources
Instead of asking the model to answer from memory, give it the relevant documents and tell it to answer only from them. This is the idea behind retrieval-augmented generation (RAG), which I’ll cover in detail later in this series. In the paper that introduced RAG, models that looked up relevant passages before answering produced more specific, diverse and factual language than a model relying on its internal memory alone[4].
Ask for sources, then check them
Ask the model to quote or cite the part of the document its answer is based on. That turns “trust me” into “here’s where I got it”, and a quick check of the quote catches many errors. Never trust a citation you haven’t clicked on.
Give it permission to say “I don’t know”
Add an explicit instruction: “If the answer isn’t in the documents provided, say you don’t know.” It sounds too simple to matter, but it directly counters the pressure to guess.
Keep a human in the loop where it counts
Match the level of checking to the level of risk. A first draft of an internal email needs a skim. Anything involving money, legal obligations, health, safety or a customer’s personal situation needs a qualified person to review it before it goes out.
| Risk level | Example use | Suggested check |
|---|---|---|
| Low | Brainstorming, first drafts, rewording | Quick read by the author |
| Medium | Internal summaries, research notes | Check key facts against the source |
| High | Customer advice, contracts, financial or legal content | Grounded answers + expert human review before use |
What I want you to take away
- A hallucination is a confident, plausible answer that is wrong or not supported by the source it was given.
- It happens because models predict plausible text, fill gaps with patterns, and have been rewarded for guessing rather than admitting uncertainty.
- Hallucinations look exactly like correct answers. Fluency and confidence are not evidence of accuracy.
- Grounding the model in your own documents is the biggest single fix, followed by asking for sources and allowing “I don’t know”.
- Match human review to the risk. The higher the stakes, the more checking you need before anything leaves the building.
Working out where AI fits in your business?
I help Australian teams cut through the hype and pick AI projects that actually pay off. If you'd like a second opinion on an idea, I'm happy to talk it through.
Get in touch →References
- Ji, Z., et al. (2023). Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 55(12). https://doi.org/10.1145/3571730
- Huang, L., et al. (2023). A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. arXiv:2311.05232. https://arxiv.org/abs/2311.05232
- Kalai, A. T., Nachum, O., Vempala, S. S., & Zhang, E. (2025). Why Language Models Hallucinate. arXiv:2509.04664. https://arxiv.org/abs/2509.04664
- Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 33. https://arxiv.org/abs/2005.11401