BEIR (Benchmarking Information Retrieval) gathers 18 datasets into a single evaluation suite covering tasks as varied as web search, question answering, and fact-checking. Many of the datasets were built for other purposes and reshaped into a retrieval format, giving broad coverage of domains and query styles in one place.
Its main value is zero-shot testing: it measures how well a model retrieves relevant passages in domains it was never trained on, which is a good proxy for how a retrieval model will behave on your own data. The framework is released under Apache 2.0, but the individual datasets carry their own licences, so check each one before using it in a commercial product.
Related sources
BRIGHT
A reasoning-intensive retrieval benchmark. It requires genuine reasoning to connect queries with relevant documents rather than keyword or semantic overlap, revealing model weaknesses that BEIR misses.
CoIR (Code Retrieval Benchmark)
A code retrieval benchmark spanning diverse programming tasks and languages. Built for evaluating how well models retrieve relevant source code, which matters if you are building RAG over codebases.
LongEmbed
A retrieval benchmark focused on very long documents, with an average length above 5,500 words. It addresses a gap in standard benchmarks, which mostly use short documents of at most 512 tokens (the chunks of text a model reads at once).
MIRACL
A multilingual retrieval benchmark built from human-annotated relevance judgements over Wikipedia articles, covering 18 languages. Widely used for evaluating retrieval within a single language across a broad set of languages.