The standard benchmarks for measuring retrieval and embedding quality, so you can compare models and pipelines on a level footing.
8sources
A collection of 18 information retrieval datasets spanning ad-hoc web search, question answering, fact verification, and duplicate question retrieval. Built by aggregating existing datasets, some originally created for other tasks and converted to retrieval format. Tests how well retrieval models generalise to unseen domains without fine-tuning, which matters for real-world RAG.
A reasoning-intensive retrieval benchmark. It requires genuine reasoning to connect queries with relevant documents rather than keyword or semantic overlap, revealing model weaknesses that BEIR misses.
A code retrieval benchmark spanning diverse programming tasks and languages. Built for evaluating how well models retrieve relevant source code, which matters if you are building RAG over codebases.
A retrieval benchmark focused on very long documents, with an average length above 5,500 words. It addresses a gap in standard benchmarks, which mostly use short documents of at most 512 tokens (the chunks of text a model reads at once).
A multilingual retrieval benchmark built from human-annotated relevance judgements over Wikipedia articles, covering 18 languages. Widely used for evaluating retrieval within a single language across a broad set of languages.
One of the most widely used general-domain retrieval benchmarks, built from real Bing search queries with human-annotated passage relevance judgements. It carries a non-commercial licence: some model developers deliberately exclude MS MARCO from training because of its terms, so check carefully before any commercial use.
The Massive Text Embedding Benchmark, a common framework for comparing text embedding models (the numeric vectors that represent meaning for search and retrieval). It brings together SemEval, BEIR, and many other datasets across tasks such as retrieval, classification, and clustering. The MMTEB extension widens the scope to over 250 languages and more than 500 tasks.
A general-domain retrieval benchmark of real, anonymised Google search queries paired with Wikipedia passages that contain the answers. Alongside MS MARCO, one of the two most widely used retrieval benchmarks for general-domain evaluation.