Datasets specifically designed for evaluating RAG system performance, including question-answer pairs with retrieval corpora.
9sources
RAG-specific benchmarks are datasets built to measure how well a Retrieval-Augmented Generation system actually works. They pair a retrieval corpus (the documents your system searches) with question and answer sets, and often mark which passages hold the correct answer. That combination lets you test the whole pipeline end to end, and, just as importantly, separate a retrieval failure (the right passage was never found) from a generation failure (the passage was found but the model answered badly). You cannot improve what you cannot measure, and these are how you measure.
Choosing one starts with matching the benchmark to your job. Some test simple lookups where the answer sits in a single passage; others need multi-hop reasoning, meaning the system has to combine facts from several documents. Corpus size matters too: a small set like RAG-Mini-Wikipedia is perfect for a quick smoke test during development, while a broader collection gives you a sterner, more realistic exam.
Two things deserve care. First, contamination: if the benchmark’s answers were part of a model’s training data, strong scores can flatter a model that is simply remembering rather than retrieving, so favour newer or domain-specific sets when this worries you. Second, licensing, which is frequently research-only, so read the terms before you build evaluation into a commercial product.
The benchmarks below range from lightweight test sets you can run in minutes to fuller evaluation suites for serious, repeatable comparison.
Meta's factual question answering benchmark of 4,409 QA pairs across five domains, paired with mock web-search and knowledge-graph retrieval APIs. It is distinct from "Corrective RAG (CRAG)", which is a retrieval method, not a dataset.
A benchmark for evaluating LLMs and retrieval systems on questions about public financial filings. The full set holds 10,231 question, answer and evidence triplets; a 150 example sample is released openly.
A research toolkit that bundles standardised, pre-processed benchmark datasets for RAG evaluation across many scenarios. Every dataset comes in a single unified JSONL format, so you can swap benchmarks in and out without rewriting your data loading each time.
A Google evaluation set of 824 challenging questions that tests factuality, multi-hop retrieval and reasoning together, each paired with a gold answer and the Wikipedia articles needed to answer it. Built to measure end-to-end RAG.
A RAG evaluation benchmark from Vectara built on 1,000 arXiv papers, with multimodal content extraction and 3,045 question-and-answer pairs across scientific domains. Designed to test retrieval and answer quality on real research documents rather than short, simplified passages.
A curated collection of datasets and resources assembled by PleIAs for building and evaluating RAG applications. A useful starting point when you want ready-made material to test retrieval pipelines without hunting for datasets one by one.
A small evaluation set that pairs 918 question-and-answer pairs with a matching corpus of 3,200 Wikipedia passages. Built specifically for testing RAG systems, it is small enough to run quick evaluation loops while still covering realistic retrieval questions.
A word-level hallucination corpus of roughly 18,000 LLM responses generated in a RAG setting, each manually annotated for hallucinated spans across question answering, summarisation, and data-to-text tasks. A reference corpus for training and evaluating faithfulness detectors.
A 2025 benchmark of 23,088 context-independent question, context and answer triples over more than 7,300 financial documents that mix text and tables. Each question maps to exactly one ground-truth document, making it purpose-built for numerical, table-aware financial RAG.