What is RAG in AI? The lookup step that runs before the model answers
- glossary
- llm
- rag
RAG, retrieval-augmented generation, is the step where a system looks things up before the model answers: it fetches relevant text from a source you chose, puts that text in the prompt, and only then lets the model generate. The rest is what it costs, where it fails, and how it differs from handing an agent a tool.
- The plain definition. RAG is a technique, not a model. A retrieval step pulls passages from your documents, database or search index and adds them to the prompt before generation.
- Why it exists. A model knows two things: its training data and what is in its context right now. Retrieval fills the second with what the first lacks.
- How it works. Four steps: index the source, retrieve the pieces that match the question, augment the prompt with them, generate the answer.
- Where you feel it. Every retrieved passage spends context tokens, and the answer is only as good as what the retriever found.
- The honest limit. RAG lowers the hallucination rate. It does not delete it.
When the model fetches for itself through tools, the question becomes which servers to connect: the MCP servers review.
The plain definition
Retrieval-augmented generation is a pipeline with a lookup in front of the model. The user asks something. A retriever searches a source you control and returns the passages that match. Those passages go into the prompt next to the question. The model answers with that material in front of it.
AWS defines it as optimizing "the output of a large language model, so it references an authoritative knowledge base". Google Cloud calls it "an AI framework that combines the strengths of traditional information retrieval systems" with generative models. Wikipedia describes a technique that lets models "retrieve and incorporate new information from external data sources" and dates the term to a 2020 paper. The common thread: the knowledge lives outside the model, and something fetches it at answer time.
What is RAG vs LLM?
An LLM is the model that reads tokens and writes tokens. RAG is a way of feeding it. Nothing inside the model changes when you add retrieval; what changes is the prompt, which now carries passages fetched a moment ago. "RAG vs LLM" is like "library vs reader". One holds the material, the other does the reading.
What it means in practice
Indexing splits the source into chunks and stores them in a searchable form: a keyword index, database rows, or vectors, which allow search by meaning. Retrieval turns the question into a query and keeps the top matches. Augmentation pastes those matches into the prompt with instructions to use them. Generation is the ordinary model call.
Two things follow.
Retrieved text spends the window. Every passage you paste is tokens in the prompt, on every question. A handful of short passages is a small toll. A pile of them, from a retriever tuned for recall, is a large one, and past a point the model can miss the relevant sentence inside it.
The retriever sets the ceiling. If the right passage is not in the results, the model answers from its weights, with the false comfort of a system that looks grounded. Google Cloud's page says it plainly: "The retrieval mechanism in RAG is critically important." Its levers are the search engine, the source data, the chunking and the question itself. None of them is the model.
And the limit stays. IBM puts it in one sentence: "While RAG can reduce the risk of hallucinations, it cannot make a model error-proof." The model can misread a correct passage, blend two of them, or answer past the end of what was retrieved. That failure has its own entry, what an AI hallucination is. Retrieval shrinks the gap that fabrication fills. It does not close it.
What is RAG vs MCP?
Classic RAG decides for the model. A retriever picks the passages and adds them to the prompt as it is assembled, before any model call. MCP hands the model a tool and lets it decide what to fetch, when, and how often. Both are retrieval before generation. The difference is who chooses what gets retrieved.
That difference makes an MCP server a bigger decision than a vector index. The clearest case I run is the database server: the agent writes its own SQL, so the retriever is a query the model composed a second ago. Two costs carry over. The schema sits in context on every call, like retrieved passages do. And read-only has to be granted by the database, because a prompt asking the agent not to write is a request, not a control.
What are examples of RAG AI?
IBM's list is the standard one: customer support bots with current product knowledge, research over internal documents, knowledge engines for employees. Same shape each time, a model in front of a document store. The examples I can vouch for are smaller: the database agent above, the wiki in the next section, and the way this blog gets written.
Before every post about someone else's product I download the documentation and write from the file, not from memory. That is RAG done by hand, retrieve first, generate second. The last judge audit of this blog caught three errors, each where the text had gone from memory instead of from the source.
What people get wrong
Is ChatGPT a RAG model?
Not by default. "RAG model" is shorthand for a model with retrieval wired in front of it. If, before answering, a chat product searched the web or read a file you attached, that answer was retrieval-augmented. If it answered from its weights, it was not. The label applies per answer, not per product.
| People say | What is true |
|---|---|
| RAG is a type of model | A pipeline around a model. Same LLM with or without it; only the prompt changes |
| Documents in the prompt end hallucination | They lower the rate; IBM's page says RAG "cannot make a model error-proof". It can still misread, blend or ignore a passage |
| More retrieved text, better answer | Passages compete for the window. Past a point relevance drops and cost rises |
| RAG needs a vector database | Retrieval is any lookup that returns text: keyword search, SQL, a file path. Wikipedia's hybrid search section pairs "a traditional text search" with vector results |
| RAG replaces fine-tuning | Fine-tuning changes the model; retrieval changes what it has in front of it. IBM frames RAG as a way to "avoid high retraining costs" |
Use the RAG vs fine tuning decision worksheet to separate changing facts from output behavior and choose the evaluation for each failure.
One example from my own work
The example that fixed my view of retrieval was a wiki, not a chatbot. I pointed an open source tool at a corpus of my own documents, 5.95 MB, roughly 1.79 million tokens, and had it write three wikis. Then I sampled. In one wiki I checked ten file paths and seven content claims against the source: ten of ten and seven of seven held. The other two each came back ten of ten on paths and six of six on claims. Zero invented facts in the sample.
The value was the map: the wiki knew which file covered what, so an agent could open the right document instead of answering from a summary. A retrieval layer that knows where things are beats one that pretends to know everything.
The failure was quieter than a hallucination. The first pass skipped two of the seven systems in the corpus and said nothing about it. The omission was invisible from inside the output. One explicit requirement in the prompt fixed it. That is the RAG failure I now expect: not invented facts, but a retriever that did not fetch what it should have, and a model too fluent to notice.
Where to go deeper
Retrieved passages come out of the same budget as everything else; how to pack it is the context window entry. The agent side, where the model runs the retrieval itself, starts with what an AI agent is and continues in the two MCP posts from the RAG vs MCP section.
Sources: AWS, What is RAG; Google Cloud, What is RAG; IBM, What is RAG; Wikipedia, RAG. All read on 30 August 2026.