FRAMES (Fact, Fetch, and Reason)
FRAMES (Fact, Fetch, and Reason) is a compact but demanding benchmark from Google for evaluating retrieval-augmented generation as a whole rather than in isolated parts. It contains 824 questions, each written to require pulling together information from several documents and then reasoning over it. Every item ships with a reference answer and a list of the Wikipedia articles that contain the supporting facts, along with a label for the kind of reasoning involved, such as numerical, tabular, temporal, or multiple-constraint reasoning.
Access is straightforward. The dataset is hosted on Hugging Face as a single test split in CSV, with an auto-converted Parquet copy, so you can pull it with the datasets library or download the files directly. There is no API or bulk archive to manage: the whole set is under a thousand rows.
For RAG and AI work it is best used as an evaluation harness, not as training or knowledge-base material. Because each question names the Wikipedia pages that hold the answer, you can test a retriever, a reasoning chain, and final answer synthesis end to end, and compare naive prompting against retrieval and multi-step approaches. The Google paper reports large gaps between these settings, which makes FRAMES useful for spotting where a pipeline fails.
Watch-outs: this is an English, Wikipedia-grounded set, so it does not reflect private or domain-specific corpora, and its small size means results can be noisy. The dataset itself is Apache 2.0, but the underlying facts trace back to Wikipedia, whose text carries its own CC BY-SA terms, so treat the reference content accordingly if you reproduce it. If you need broader or complementary coverage, benchmarks such as Natural Questions or HotpotQA sit alongside FRAMES in a RAG evaluation toolkit.
Related sources
CRAG (Comprehensive RAG Benchmark)
Meta's factual question answering benchmark of 4,409 QA pairs across five domains, paired with mock web-search and knowledge-graph retrieval APIs. It is distinct from "Corrective RAG (CRAG)", which is a retrieval method, not a dataset.
FinanceBench
A benchmark for evaluating LLMs and retrieval systems on questions about public financial filings. The full set holds 10,231 question, answer and evidence triplets; a 150 example sample is released openly.
FlashRAG Benchmark Datasets
A research toolkit that bundles standardised, pre-processed benchmark datasets for RAG evaluation across many scenarios. Every dataset comes in a single unified JSONL format, so you can swap benchmarks in and out without rewriting your data loading each time.
Open RAGBench (Vectara)
A RAG evaluation benchmark from Vectara built on 1,000 arXiv papers, with multimodal content extraction and 3,045 question-and-answer pairs across scientific domains. Designed to test retrieval and answer quality on real research documents rather than short, simplified passages.