RTEB (Retrieval Embedding Benchmark)
RTEB (Retrieval Embedding Benchmark) is a retrieval-focused benchmark from Hugging Face and the MTEB community, launched in beta on 1 October 2025 as a new retrieval section of the MTEB leaderboard. Like MTEB and BEIR, it is not a corpus you retrieve over but a yardstick for choosing the embedding model that powers retrieval, and it is built specifically around the accuracy that matters for RAG. It covers 20 languages, from common ones such as English and Japanese to rarer ones such as Bengali and Finnish, and spans enterprise domains including law, healthcare, finance and code.
Its defining idea is a hybrid of open and private datasets. The open sets publish their full corpus, queries and relevance labels, so you can reproduce the scores yourself. The private sets are held back and evaluated only by MTEB maintainers, who commit not to train or publish models on them; for transparency each private set ships descriptive statistics, a description and a few sample (query, document, relevance) triplets. The aim is to measure genuine generalisation to unseen data rather than scores inflated by training on the test set. Around half of the retrieval datasets are repurposed from question-answering collections, and the default leaderboard metric is NDCG@10.
You use it the way you use MTEB: shortlist embedders on the leaderboard, favouring the domains and languages closest to your own, then confirm on your own corpus. Access is through the MTEB leaderboard on Hugging Face, with the evaluation code in the open mteb Python library on GitHub (Apache 2.0). The caveat is built into the design: scores on the private sets cannot be independently reproduced, so you are trusting the maintainers' process, and the individual datasets keep their own licences. Treat RTEB as a sharper, generalisation-focused complement to our MTEB and BEIR entries.
Related sources
BEIR
A collection of 18 information retrieval datasets spanning ad-hoc web search, question answering, fact verification, and duplicate question retrieval. Built by aggregating existing datasets, some originally created for other tasks and converted to retrieval format. Tests how well retrieval models generalise to unseen domains without fine-tuning, which matters for real-world RAG.
BRIGHT
A reasoning-intensive retrieval benchmark. It requires genuine reasoning to connect queries with relevant documents rather than keyword or semantic overlap, revealing model weaknesses that BEIR misses.
CoIR (Code Retrieval Benchmark)
A code retrieval benchmark spanning diverse programming tasks and languages. Built for evaluating how well models retrieve relevant source code, which matters if you are building RAG over codebases.
FreshStack
A 2025 framework and benchmark for retrieval over fast-moving technical documentation. It pairs real Stack Overflow questions and answers with chunked GitHub code and docs across five niche domains, giving code and docs RAG a hard, contamination-resistant test.