Natural Questions
Natural Questions takes genuine questions that people typed into Google and pairs each with a Wikipedia page and the passage that answers it. Because the questions come from real search behaviour rather than being written for the benchmark, they read the way people actually ask for information, warts and all. Each example carries two kinds of answer annotation: a long answer, usually the paragraph or table that contains the response, and a short answer, the specific span or entity, where one exists. The full release runs to roughly 300,000 training examples plus smaller development and test splits.
The original data is published on Google's research site and mirrors the full Wikipedia page for each question, which makes the raw files large. For retrieval work most people reach for a lighter packaging instead: the BEIR benchmark ships an "nq" subset that reduces the task to questions matched against a corpus of Wikipedia passages, which drops straight into a vector database and standard evaluation scripts. It is also available through Hugging Face Datasets and TensorFlow Datasets, so you can stream it rather than download everything at once.
Natural Questions is best for measuring open-domain retrieval and end-to-end RAG question answering: does your retriever surface the passage that holds the answer, and does your generator use it correctly? It suits benchmarking a general-purpose assistant rather than a niche domain.
Watch the licence: the dataset is released under CC BY-SA 3.0, so any derivative you share must carry the same share-alike licence and credit the source, which matters if you redistribute a processed version. The Wikipedia snapshot is also frozen at collection time, so answers reflect the encyclopaedia as it was, not today. It sits alongside MS MARCO as one of the two default general-domain retrieval benchmarks, with TriviaQA and HotpotQA useful when you want harder or multi-hop questions.
Related sources
BEIR
A collection of 18 information retrieval datasets spanning ad-hoc web search, question answering, fact verification, and duplicate question retrieval. Built by aggregating existing datasets, some originally created for other tasks and converted to retrieval format. Tests how well retrieval models generalise to unseen domains without fine-tuning, which matters for real-world RAG.
BRIGHT
A reasoning-intensive retrieval benchmark. It requires genuine reasoning to connect queries with relevant documents rather than keyword or semantic overlap, revealing model weaknesses that BEIR misses.
CoIR (Code Retrieval Benchmark)
A code retrieval benchmark spanning diverse programming tasks and languages. Built for evaluating how well models retrieve relevant source code, which matters if you are building RAG over codebases.
FreshStack
A 2025 framework and benchmark for retrieval over fast-moving technical documentation. It pairs real Stack Overflow questions and answers with chunked GitHub code and docs across five niche domains, giving code and docs RAG a hard, contamination-resistant test.