What is RAG?

Retrieval-Augmented Generation (RAG) is a way of building AI systems that look things up before they answer, instead of relying on memory alone. If you have ever asked a chatbot a question and got a confident answer that turned out to be wrong, you have met the problem RAG is designed to solve. This guide explains what RAG is from first principles, with no prior knowledge assumed.

The problem: models forget and models make things up

A large language model (LLM) is the kind of AI that powers tools like ChatGPT. It learns patterns from a huge amount of text, then predicts what to write next, one word at a time. That makes it fluent, but it creates two problems.

The first is staleness. A model only knows what was in its training data, and training happens at a fixed point in time. Ask it about an event from last week, a new product, or your company’s internal policy, and it has no way to know. Its knowledge is frozen.

The second is hallucination, which is when a model states something false as though it were true. Because the model is predicting plausible-sounding text rather than checking facts, it can invent a statistic, a quotation, or a source that never existed, and present it with total confidence.

You could try to fix this by retraining the model on fresh data, but that is slow and expensive, and you would have to do it again every time something changes. RAG takes a different approach: leave the model as it is, and give it the right information at the moment you ask the question.

The core idea: look it up, then answer

RAG combines two steps. First it retrieves relevant information from a collection of documents you trust. Then it hands that information to the model along with your question, so the model can generate an answer based on real, current text rather than on memory.

Think of the difference between asking a colleague a question off the top of their head, and asking them to check the handbook first and then reply. The second answer is grounded in a source you can point to. Grounding means the answer is tied to actual documents, so you can trace where it came from and check it.

The pipeline: chunk, embed, store, retrieve, generate

RAG works in five stages. The first three prepare your data ahead of time. The last two run every time someone asks a question.

Chunk

Your source documents might be long: a 40-page policy, a whole book, a year of support tickets. You split them into smaller passages, a step called chunking. A chunk is usually a few sentences or a paragraph, small enough to be about one idea but large enough to make sense on its own.

Embed

Next you convert each chunk into an embedding, which is a list of numbers that captures the meaning of the text. Passages about similar topics end up with similar numbers, even if they use different words. This is what lets a search for “how do I get my money back” find a passage titled “Refund policy”.

Store

You put all those embeddings into a vector database, a store built to hold embeddings and find the closest matches quickly. This is your searchable library, ready to answer questions.

Retrieve

When a question arrives, you embed the question the same way, then ask the vector database for the handful of chunks whose meaning is closest to it. This step is the retrieval in Retrieval-Augmented Generation. You typically pull back the top few passages, not the whole library.

Generate

Finally you give the model both the question and the retrieved passages, and ask it to write an answer using them. Because the model is working from real text you supplied, its answer stays current and is far less likely to be invented. Many systems also cite the passages, so you can check the source yourself.

RAG or fine-tuning: which do you need?

There is another way to change how a model behaves, called fine-tuning, which means continuing to train the model on your own examples so its underlying behaviour shifts. It is easy to confuse the two, so here is a rule of thumb.

Reach for RAG when you need the model to know facts that change or that live in your own documents: product details, policies, research papers, anything you might edit tomorrow. Updating is as easy as adding or replacing a document, with no retraining.

Reach for fine-tuning when you need to change how the model writes rather than what it knows: a consistent tone, a specific format, or a specialised style. The two are not rivals, and plenty of production systems use both, but for keeping answers accurate and up to date, RAG is usually the better first move.

What RAG needs from you: good data

A RAG system is only as good as the documents you feed it. Retrieval cannot surface a fact that is not in your collection, and it will happily retrieve rubbish if that is what you stored. Clean, well-organised, trustworthy source data is the foundation of the whole thing.

That is where we come in. You can browse the directory to find real, ready-to-use data sources for RAG, from open web archives to encyclopaedic knowledge bases and scientific literature.

Next steps

You now know the shape of RAG: the problems it solves, the five-stage pipeline, and when to choose it over fine-tuning. When you are ready to go deeper, the glossary defines every term in one place, and the directory is the best place to start gathering the data your own system will need.