MS MARCO
MS MARCO started as a machine-reading dataset built from real, anonymised Bing search queries and grew into one of the most heavily used retrieval benchmarks in the field. Its passage-ranking track pairs roughly a million queries with a collection of about 8.8 million short passages, together with human judgements of which passage answers which query. Related tasks cover document ranking and question answering, all grounded in genuine search behaviour rather than invented questions.
You can pull the data from the official site or from Hugging Face, usually as tab-separated or JSON files. The passages are already short and retrieval-sized, so you can embed the collection into a vector database and start evaluating retrievers and rerankers almost straight away. Because it is so widely adopted, nearly every retrieval library and tutorial assumes you have it to hand.
MS MARCO is at its best as a training and evaluation set for passage ranking: fine-tuning cross-encoders and dense retrievers, then measuring them against a realistic, general-domain workload. It is a benchmark, not a knowledge base to serve to end users.
Two caveats matter. The relevance judgements are sparse, often just one passage marked relevant per query while other genuinely relevant passages go unlabelled, which understates recall and produces misleading false negatives if you read the scores too literally. More important for production, MS MARCO carries a non-commercial licence, and some model developers deliberately keep it out of their training data to avoid the restriction. If you are building anything commercial, read the terms before you rely on it.
It is usually cited alongside Natural Questions, which is openly licensed and a clean complement, and tested against the broader BEIR suite, with MIRACL for multilingual work, when you want to look beyond a single general-domain benchmark.
Related sources
BEIR
A collection of 18 information retrieval datasets spanning ad-hoc web search, question answering, fact verification, and duplicate question retrieval. Built by aggregating existing datasets, some originally created for other tasks and converted to retrieval format. Tests how well retrieval models generalise to unseen domains without fine-tuning, which matters for real-world RAG.
BRIGHT
A reasoning-intensive retrieval benchmark. It requires genuine reasoning to connect queries with relevant documents rather than keyword or semantic overlap, revealing model weaknesses that BEIR misses.
CoIR (Code Retrieval Benchmark)
A code retrieval benchmark spanning diverse programming tasks and languages. Built for evaluating how well models retrieve relevant source code, which matters if you are building RAG over codebases.
FreshStack
A 2025 framework and benchmark for retrieval over fast-moving technical documentation. It pairs real Stack Overflow questions and answers with chunked GitHub code and docs across five niche domains, giving code and docs RAG a hard, contamination-resistant test.