What is RAG (Retrieval-Augmented Generation)?

Ask a general AI chatbot about your company’s leave policy and you will get a confident, generic answer that may have nothing to do with your actual policy. The model has never seen your documents, so it fills the gap with something plausible. Retrieval-Augmented Generation, almost always shortened to RAG, is the most common way businesses fix that problem.

RAG sounds technical, but the idea behind it is simple, and once it clicks you will start noticing it everywhere — in customer support bots, internal knowledge assistants and “chat with your documents” tools. In this post I’ll walk through what RAG is, how it works step by step, and where it tends to go wrong.

The open-book exam

The analogy I use with clients is an exam. A plain language model is sitting a closed-book exam: it can only answer from what it memorised during training. If the answer isn’t in its memory, it may guess. RAG turns it into an open-book exam: before answering, the system looks up the relevant pages in your documents and puts them in front of the model, which then writes its answer using those pages.

The term comes from a 2020 research paper by Patrick Lewis and colleagues at Facebook AI Research, University College London and New York University. They combined a language model’s built-in knowledge — what they called parametric memory — with a searchable index of documents — non-parametric memory — and found the combination produced more specific and more factual answers than the language model alone.^[1]

How RAG works, step by step

Most RAG systems follow the same basic recipe. The first two steps happen once, when you set things up; the last three happen every time someone asks a question.

Why not just paste every document into the prompt? Because, as I covered in what is a context window, there is a limit to how much a model can take in, every token costs money, and models don’t pay equal attention to everything in a long input. Retrieval lets you send only the handful of passages that actually matter.

Embeddings, in plain English

Step 2 relies on something called an embedding. An embedding is a list of numbers that represents the meaning of a piece of text, so that passages about similar things end up with similar numbers. That lets the system find a section about “annual leave entitlements” even when the user typed “how many holidays do I get?” — there are no shared keywords, but the meaning is close.

This style of search took off around the same time as RAG itself. A 2020 study on Dense Passage Retrieval showed that meaning-based search using learned embeddings could outperform a strong traditional keyword-matching system called BM25 at finding the right passages for open-domain questions.^[2] Many production systems today use a mix of both: keyword search for exact terms like product codes, and embeddings for meaning.

Why businesses like RAG

• Answers grounded in your content. The model works from your policies, products and records instead of the general internet.
• Easy to keep current. Update a document and re-index it, and answers change straight away. There is no need to retrain the model.
• Citations. Because the system knows which passages it retrieved, it can show users where an answer came from so they can check it.
• Access control. You can restrict retrieval to documents a user is allowed to see.

People often ask whether they should use RAG or fine-tune a model instead — that is, train it further on their own data. As a rough guide, RAG is the better fit when the goal is to give the model access to facts that change, while fine-tuning is more about changing a model’s behaviour, such as its tone or output format. A widely cited survey of the field treats RAG as particularly well suited to knowledge-heavy tasks where information needs to stay up to date.^[3] For most of the business use cases I see, RAG is the place to start.

Where RAG goes wrong

RAG reduces made-up answers, but it doesn’t eliminate them. In my experience the problems almost always sit in the retrieval step, not the AI step. If the search pulls back the wrong passages, the model will write a fluent answer from the wrong material.

• Bad chunking. Split a document in the wrong place and a chunk can lose its meaning — a passage that says “revenue grew 3%” is useless if you can’t tell which company or quarter it refers to.
• Messy source documents. Out-of-date versions, duplicates and scanned PDFs with poor text quality all lead to poor answers.
• Missing content. If the answer isn’t in your documents, a good system should say so rather than improvise. That needs to be designed in.

The chunking problem is real enough that Anthropic published a technique called Contextual Retrieval, which adds a short explanation of each chunk’s context before indexing it. In their tests, combining this with both embedding and keyword search reduced failed retrievals by 49%, and by 67% when a reranking step was added.^[4] The broader lesson: retrieval quality is where most of the gains are.

What I want you to take away

1. RAG is an open-book exam for AI. The system searches your material first, then the model answers using what it found.
2. It follows a simple recipe. Chunk, index, retrieve, augment, generate.
3. Embeddings let search work on meaning, not just keywords. Most good systems combine both.
4. It is usually the right first step for “AI on our documents”. It stays current and can cite its sources, without retraining a model.
5. Retrieval quality decides answer quality. Clean documents and sensible chunking matter more than which model you choose.