Evaluating your RAG system: the benchmarks and datasets that matter

A Retrieval-Augmented Generation (RAG) system has two jobs: find the right passages, then write a grounded answer from them. When something goes wrong, the failure could be in either job, and a single “the answer was bad” impression tells you nothing about which. Evaluation is how you tell them apart. This post walks through the datasets and metrics that let you measure each layer, and ends with the advice that matters most: build your own small evaluation set. If you want the wider landscape of what is out there, the directory catalogues it, and the rag-benchmarks and retrieval-benchmarks categories collect the sets discussed here.

Two layers, measured separately

Keep the two layers distinct in your head and in your tooling:

Retrieval quality. Given a question, does your retriever surface the passages that actually contain the answer? This is a search problem, and it is measured against a corpus with known relevant documents.
Answer quality. Given the passages that were retrieved, does the model produce a correct, grounded, relevant answer? This is a generation problem, and it depends on the retriever having done its job first.

If you only measure the end-to-end answer, a good generator can paper over a weak retriever on easy questions and collapse on hard ones, and you will not know why. Measure retrieval on its own, then measure the whole pipeline.

Layer one: retrieval benchmarks

Retrieval evaluation uses datasets where each query is paired with the documents judged relevant to it. You run your retriever and check how many of those relevant documents it returns, and how highly it ranks them.

MS MARCO is the workhorse for passage ranking. Built from real search queries and a large collection of passages, it is the set most embedding and ranking models are trained and compared on. If you want a baseline sense of whether a retriever ranks relevant passages above irrelevant ones, this is the reference point.

BEIR (Benchmark for Information Retrieval) exists because a model that does well on one dataset can do poorly elsewhere. BEIR bundles many retrieval tasks across different domains, from fact-checking to scientific search, and evaluates in a zero-shot setting, meaning the model is not fine-tuned on each task. It is the standard way to ask “does this retriever generalise beyond the data it was trained on”, which is exactly the question that matters when your documents look nothing like the training set.

MTEB (Massive Text Embedding Benchmark) is where you should start when choosing an embedding model. It aggregates retrieval alongside clustering, classification and other tasks into one leaderboard, so you can compare embedders on a like-for-like basis rather than trusting a vendor’s own numbers. Use it to draw up a shortlist of candidate embedders, then verify them on your own data.

Beyond the general sets, several benchmarks target conditions that break naive assumptions:

CoIR (Code Information Retrieval) evaluates retrieval over source code and technical text, where the query and the target may share little natural-language vocabulary. If you are building over a codebase, general retrieval scores will mislead you.
MIRACL measures multilingual retrieval across many languages. A model that looks strong in English can degrade sharply in others, and MIRACL surfaces that.
LongEmbed probes long-context retrieval, where relevant information sits inside long documents. Many embedders truncate or lose signal past a certain length, and this is where you find out.

Pick the specialised set that matches your workload. There is no benefit in a model that tops the English leaderboard if your users write in five languages or search a code repository.

Retrieval metrics, plainly

Two numbers carry most of the weight:

Recall@k. Of all the documents that were relevant to a query, what fraction appeared in the top k results you retrieved? Recall@5 or recall@10 tells you whether the answer-bearing passage even reached the model. For RAG this is often the single most important retrieval metric, because a passage the model never sees cannot inform the answer.
nDCG (normalised discounted cumulative gain). This rewards putting relevant documents higher up, not just including them somewhere in the list. A retriever that buries the right passage at position ten scores worse than one that ranks it first, even if both technically “found” it. Ranking matters when you only pass the top few passages to the model.

Report both. Recall tells you whether the information is present; nDCG tells you whether it is prominent.

Layer two: end-to-end RAG benchmarks

Once retrieval is holding up, evaluate the whole pipeline: retrieve, then generate, then judge the answer.

Natural Questions is built from real questions people asked, paired with Wikipedia, and it remains a solid end-to-end test because the questions are genuine rather than constructed to be easy. It exercises the full path from question to grounded answer over a general knowledge corpus.

Open-RAGBench is a benchmark built specifically for RAG rather than adapted from a pure retrieval or QA task, pairing questions with a corpus so you can score retrieval and generation together under one harness.

FlashRAG is worth knowing as a toolkit rather than a single dataset. It bundles many RAG benchmarks and the plumbing to run them, so you can evaluate a pipeline against a range of established sets without wiring each one up by hand. If you want breadth quickly, start here.

Generation metrics, plainly

Answer quality does not reduce to a single accuracy figure, because a RAG answer can be fluent and confident while being wrong or unsupported. The commonly used triad, popularised by RAGAS-style evaluation, breaks it into three questions:

Faithfulness. Is every claim in the answer supported by the retrieved passages, or did the model invent something? This is your hallucination check.
Answer relevance. Does the answer actually address the question that was asked, rather than drifting to a related topic?
Context precision. Of the passages you retrieved and passed in, how many were actually useful? Low context precision means you are feeding the model noise, which raises cost and invites distraction.

These are typically scored by a model acting as a judge against the retrieved context, so treat them as directional signals rather than exact truth, and always spot-check the judgements by hand.

The most important step: build your own eval set

Public benchmarks are calibration, not the target. They tell you a model is competent in general; they cannot tell you it works on your contracts, your support tickets or your product manuals. Production quality is predicted by an evaluation set drawn from your own material.

Build one deliberately:

Collect real questions from users, support logs or the people who will actually use the system. Invented questions tend to be too clean.
For each question, note the passage in your own documents that answers it. This gives you ground truth for recall.
Write, or agree, a reference answer, so you can judge faithfulness and relevance against something concrete.
Keep it small but honest. Fifty to a couple of hundred well-chosen examples, including the awkward and ambiguous ones, will teach you more than any leaderboard.

Wire this into a repeatable harness so every change to chunking, embedding or prompting is scored the same way. To practise the mechanics before pointing it at your own data, rag-mini-wikipedia is a deliberately tiny corpus with paired questions: small enough to stand up an eval loop in an afternoon and confirm your metrics behave as expected.

The Learn section walks through the build itself, and the DIY RAG course lesson on improving your RAG picks up where this post leaves off. If you are still choosing a corpus to build on, choosing your first RAG dataset is the companion piece. Measure both layers, trust your own eval set over any leaderboard, and change one thing at a time.