BRIGHT
BRIGHT is designed to test retrieval where finding the right document takes actual reasoning, not just matching words or surface meaning between the query and the text. Many benchmarks reward models that spot shared keywords or similar phrasing, which hides how they perform when the connection between question and answer is less direct.
By focusing on reasoning-heavy queries, BRIGHT exposes weaknesses that broad benchmarks such as BEIR do not surface, making it a useful stress test for retrieval in technical or analytical domains. It is released under CC BY 4.0, so you can use it commercially as long as you credit the source.
Related sources
BEIR
A collection of 18 information retrieval datasets spanning ad-hoc web search, question answering, fact verification, and duplicate question retrieval. Built by aggregating existing datasets, some originally created for other tasks and converted to retrieval format. Tests how well retrieval models generalise to unseen domains without fine-tuning, which matters for real-world RAG.
CoIR (Code Retrieval Benchmark)
A code retrieval benchmark spanning diverse programming tasks and languages. Built for evaluating how well models retrieve relevant source code, which matters if you are building RAG over codebases.
LongEmbed
A retrieval benchmark focused on very long documents, with an average length above 5,500 words. It addresses a gap in standard benchmarks, which mostly use short documents of at most 512 tokens (the chunks of text a model reads at once).
MIRACL
A multilingual retrieval benchmark built from human-annotated relevance judgements over Wikipedia articles, covering 18 languages. Widely used for evaluating retrieval within a single language across a broad set of languages.