The standard benchmarks for measuring retrieval and embedding quality, so you can compare models and pipelines on a level footing.
13sources
Retrieval benchmarks are the standard tests for measuring how well a system finds the right passage for a query, and they are the closest thing you have to an objective answer to “is my retrieval any good?”. A RAG system lives or dies on whether it fetches relevant context before the model writes anything, so before you commit to an embedding model or a search pipeline, these datasets let you compare options on the same queries and the same judged answers rather than on vibes.
Choosing between them is really about matching the benchmark to your use case. Broad suites like BEIR and MTEB (the Massive Text Embedding Benchmark) score a model across many tasks and domains at once, which is perfect for a first shortlist. Focused sets are better once you know your shape of problem: MIRACL for multilingual retrieval, CoIR for code, LongEmbed for long documents, BRIGHT for reasoning-heavy queries. A model that tops the general leaderboard can still trail on your specific domain, so pick the benchmark that looks like your real traffic.
The pitfall to watch is overfitting to the leaderboard. Popular benchmarks are so widely used that some models are tuned to score well on them, which flatters their numbers without helping your data. Treat a strong result as a reason to shortlist, not a guarantee. Check the licence too, since a few sets, like MS MARCO, allow research use but restrict commercial deployment.
The benchmarks below cover broad multi-task suites and the focused tests for specific retrieval challenges.
A collection of 18 information retrieval datasets spanning ad-hoc web search, question answering, fact verification, and duplicate question retrieval. Built by aggregating existing datasets, some originally created for other tasks and converted to retrieval format. Tests how well retrieval models generalise to unseen domains without fine-tuning, which matters for real-world RAG.
A reasoning-intensive retrieval benchmark. It requires genuine reasoning to connect queries with relevant documents rather than keyword or semantic overlap, revealing model weaknesses that BEIR misses.
A code retrieval benchmark spanning diverse programming tasks and languages. Built for evaluating how well models retrieve relevant source code, which matters if you are building RAG over codebases.
A 2025 framework and benchmark for retrieval over fast-moving technical documentation. It pairs real Stack Overflow questions and answers with chunked GitHub code and docs across five niche domains, giving code and docs RAG a hard, contamination-resistant test.
The first open benchmark for the retrieval step of legal RAG. It offers 6,858 human-annotated query-answer pairs over a corpus of roughly 79 million characters, with character-level ground-truth spans across contracts and privacy policies, assembled from CUAD, MAUD, ContractNLI and PrivacyQA.
A retrieval benchmark focused on very long documents, with an average length above 5,500 words. It addresses a gap in standard benchmarks, which mostly use short documents of at most 512 tokens (the chunks of text a model reads at once).
A benchmark for evaluating legal text embedding and retrieval models, built from ten expert-annotated datasets spanning six jurisdictions and three task types (search, zero-shot classification, and question answering).
A multilingual retrieval benchmark built from human-annotated relevance judgements over Wikipedia articles, covering 18 languages. Widely used for evaluating retrieval within a single language across a broad set of languages.
One of the most widely used general-domain retrieval benchmarks, built from real Bing search queries with human-annotated passage relevance judgements. It carries a non-commercial licence: some model developers deliberately exclude MS MARCO from training because of its terms, so check carefully before any commercial use.
The Massive Text Embedding Benchmark, a common framework for comparing text embedding models (the numeric vectors that represent meaning for search and retrieval). It brings together SemEval, BEIR, and many other datasets across tasks such as retrieval, classification, and clustering. The MMTEB extension widens the scope to over 250 languages and more than 500 tasks.
A general-domain retrieval benchmark of real, anonymised Google search queries paired with Wikipedia passages that contain the answers. Alongside MS MARCO, one of the two most widely used retrieval benchmarks for general-domain evaluation.
The MTEB team's retrieval-focused embedding benchmark, launched in beta in October 2025 as a new retrieval section of the MTEB leaderboard. It spans 20 languages and enterprise domains such as law, healthcare, finance and code, and deliberately mixes open datasets with held-out private ones to measure genuine generalisation rather than training-set memorisation.
A benchmark for OCR-free, page-image document retrieval, introduced with the ColPali paper. It bundles page-level retrieval tasks across several domains and languages so vision-based retrievers can be evaluated directly on document images.