CoIR (Code Retrieval Benchmark)
CoIR (Code Information Retrieval) brings together a range of programming tasks and languages into one benchmark aimed squarely at code retrieval. Standard text retrieval benchmarks say little about how a model handles source code, where structure, syntax, and identifiers carry meaning that plain prose does not.
If you are building a RAG system over codebases, whether for code search, documentation lookup, or developer assistance, CoIR gives you a way to compare retrieval models on the kind of content you will actually be searching. It is released under the MIT licence.
Related sources
BEIR
A collection of 18 information retrieval datasets spanning ad-hoc web search, question answering, fact verification, and duplicate question retrieval. Built by aggregating existing datasets, some originally created for other tasks and converted to retrieval format. Tests how well retrieval models generalise to unseen domains without fine-tuning, which matters for real-world RAG.
BRIGHT
A reasoning-intensive retrieval benchmark. It requires genuine reasoning to connect queries with relevant documents rather than keyword or semantic overlap, revealing model weaknesses that BEIR misses.
LongEmbed
A retrieval benchmark focused on very long documents, with an average length above 5,500 words. It addresses a gap in standard benchmarks, which mostly use short documents of at most 512 tokens (the chunks of text a model reads at once).
MIRACL
A multilingual retrieval benchmark built from human-annotated relevance judgements over Wikipedia articles, covering 18 languages. Widely used for evaluating retrieval within a single language across a broad set of languages.