CoIR (Code Retrieval Benchmark)
CoIR (Code Information Retrieval) brings together a range of programming tasks and languages into one benchmark aimed squarely at code retrieval. It spans several kinds of retrieval that a code assistant actually does: finding code from a natural-language description, finding code that is similar to other code, answering code questions, and retrieving the right snippet to complete a function. Standard text-retrieval benchmarks say little about any of this, because source code carries meaning in its structure, syntax, and identifiers that plain prose does not.
CoIR ships as a Python package with a small, consistent evaluation harness, and the underlying task datasets are on HuggingFace, so you point it at an embedding or retrieval model and get comparable scores back. If you have used BEIR or MTEB for text, the workflow will feel familiar: same idea, tuned for code.
The important thing to be clear about is what CoIR is for. It is a measuring tape, not a knowledge base. You do not put CoIR into your production index. You use it to choose the embedding model that will power a RAG system over your own codebases, whether that is code search, documentation lookup, or an in-editor assistant, before you commit to indexing millions of lines. Picking the wrong retriever is expensive to discover later, and this is how you avoid it.
A couple of caveats. Benchmark scores are a guide, not a guarantee, and your private code may differ from the public repositories these tasks are built from, so treat the ranking as a shortlist and confirm on a sample of your own data. There is also a risk that popular models have seen some of this public code in training, which can flatter their scores. On licensing CoIR is clean: it is released under the MIT licence, so commercial use is fine.
Reach for CoIR alongside general-purpose suites like MTEB and BEIR for prose, and LongEmbed when your documents run very long.
Related sources
BEIR
A collection of 18 information retrieval datasets spanning ad-hoc web search, question answering, fact verification, and duplicate question retrieval. Built by aggregating existing datasets, some originally created for other tasks and converted to retrieval format. Tests how well retrieval models generalise to unseen domains without fine-tuning, which matters for real-world RAG.
BRIGHT
A reasoning-intensive retrieval benchmark. It requires genuine reasoning to connect queries with relevant documents rather than keyword or semantic overlap, revealing model weaknesses that BEIR misses.
FreshStack
A 2025 framework and benchmark for retrieval over fast-moving technical documentation. It pairs real Stack Overflow questions and answers with chunked GitHub code and docs across five niche domains, giving code and docs RAG a hard, contamination-resistant test.
LegalBench-RAG
The first open benchmark for the retrieval step of legal RAG. It offers 6,858 human-annotated query-answer pairs over a corpus of roughly 79 million characters, with character-level ground-truth spans across contracts and privacy policies, assembled from CUAD, MAUD, ContractNLI and PrivacyQA.