BEIR (Benchmarking Information Retrieval) is not a knowledge base you deploy but a yardstick for measuring how good your retrieval is. It bundles 18 datasets into one evaluation suite spanning very different tasks: ad-hoc web search, question answering, fact verification, citation prediction, duplicate-question finding, and more. Many were originally built for other purposes and reshaped into a common retrieval format, so in one place you get scientific text (SciFact, TREC-COVID), general question answering (Natural Questions, HotpotQA), argument and fact-checking corpora, and Q&A from Stack Exchange, each with a corpus of passages, a set of queries, and human relevance judgements.
The reason it matters is zero-shot generalisation. Each dataset is meant to be evaluated without training on it, so BEIR tells you how a retriever behaves on domains it has never seen, which is a far better proxy for your own data than a score on the one benchmark a model was tuned for. In practice you use it to choose and sanity-check the retrieval half of your stack: swap embedding models, or compare dense retrieval against classic BM25 keyword search, and see which holds up across domains rather than winning on a single set.
Getting started is easy. The beir Python library on GitHub downloads any of the datasets in a standard format and runs the evaluation loop for you, reporting the usual information-retrieval metrics (nDCG, recall, and MAP at various cut-offs). A widely repeated finding from BEIR is that no single model wins everywhere and that a strong BM25 baseline is surprisingly hard to beat, which is a useful reality check before you commit to an expensive embedding pipeline.
On licensing, the BEIR framework itself is Apache 2.0, but the datasets inside it keep their own individual licences, some of which restrict commercial use or redistribution, so treat the suite as an evaluation tool and check each dataset's terms before you reuse its content in a product. For broader embedding evaluation across tasks and languages, MTEB (which incorporates BEIR) is the natural next step.
Related sources
BRIGHT
A reasoning-intensive retrieval benchmark. It requires genuine reasoning to connect queries with relevant documents rather than keyword or semantic overlap, revealing model weaknesses that BEIR misses.
CoIR (Code Retrieval Benchmark)
A code retrieval benchmark spanning diverse programming tasks and languages. Built for evaluating how well models retrieve relevant source code, which matters if you are building RAG over codebases.
FreshStack
A 2025 framework and benchmark for retrieval over fast-moving technical documentation. It pairs real Stack Overflow questions and answers with chunked GitHub code and docs across five niche domains, giving code and docs RAG a hard, contamination-resistant test.
LegalBench-RAG
The first open benchmark for the retrieval step of legal RAG. It offers 6,858 human-annotated query-answer pairs over a corpus of roughly 79 million characters, with character-level ground-truth spans across contracts and privacy policies, assembled from CUAD, MAUD, ContractNLI and PrivacyQA.