CRAG (Comprehensive RAG Benchmark)
CRAG, the Comprehensive RAG Benchmark, is a factual question answering benchmark released by Meta to test retrieval-augmented generation systems on realistic, varied queries. It contains 4,409 question-answer pairs spread across five domains: finance, sports, music, movies, and open-domain encyclopaedic knowledge. The questions span eight categories and deliberately mix popular and long-tail entities, along with facts that change over timescales from years down to seconds, so a system is tested on both stable and fast-moving information.
What sets CRAG apart from a plain QA set is that it ships with retrieval content and mock APIs. Each question comes with up to 50 full HTML pages simulating a web search, and a set of mock knowledge-graph APIs simulating structured lookups. This lets you evaluate the whole RAG pipeline, retrieval plus generation, rather than only the language model. The benchmark underpinned the Meta KDD Cup 2024 challenge, which drew thousands of participants across three tasks (web retrieval, structured queries, and an end-to-end setting).
Access is through the project's GitHub repository. The data is distributed as bzip2-compressed JSONL files, with the larger task-3 set split into parts you concatenate before extracting; the schema is documented in the repo. For RAG or AI work it is best used as an evaluation harness: measure retrieval quality, answer accuracy, and, importantly, hallucination and "I don't know" behaviour, since CRAG scores wrong answers more harshly than abstentions.
The main watch-out is licensing. CRAG is released under CC BY-NC 4.0, so use is limited to non-commercial research and evaluation, and the bundled HTML retrieval content carries the terms of its original publishers. Attribution is required. If you need a commercially usable evaluation set, consider Natural Questions or the retrieval corpora behind BEIR instead.
Related sources
FinanceBench
A benchmark for evaluating LLMs and retrieval systems on questions about public financial filings. The full set holds 10,231 question, answer and evidence triplets; a 150 example sample is released openly.
FlashRAG Benchmark Datasets
A research toolkit that bundles standardised, pre-processed benchmark datasets for RAG evaluation across many scenarios. Every dataset comes in a single unified JSONL format, so you can swap benchmarks in and out without rewriting your data loading each time.
FRAMES (Fact, Fetch, and Reason)
A Google evaluation set of 824 challenging questions that tests factuality, multi-hop retrieval and reasoning together, each paired with a gold answer and the Wikipedia articles needed to answer it. Built to measure end-to-end RAG.
Open RAGBench (Vectara)
A RAG evaluation benchmark from Vectara built on 1,000 arXiv papers, with multimodal content extraction and 3,045 question-and-answer pairs across scientific domains. Designed to test retrieval and answer quality on real research documents rather than short, simplified passages.