MTEB (Massive Text Embedding Benchmark)
MTEB unifies datasets from many different embedding tasks into one evaluation framework, so you can compare embedding models on a like-for-like basis rather than trusting each model's own headline numbers. It folds in SemEval and BEIR alongside a wide range of other datasets to give a rounded view of performance.
The MMTEB extension pushes coverage to more than 250 languages and over 500 tasks, making it the most comprehensive public reference for embedding quality. The leaderboard is hosted on Hugging Face and updated as new models are submitted.
Related sources
BEIR
A collection of 18 information retrieval datasets spanning ad-hoc web search, question answering, fact verification, and duplicate question retrieval. Built by aggregating existing datasets, some originally created for other tasks and converted to retrieval format. Tests how well retrieval models generalise to unseen domains without fine-tuning, which matters for real-world RAG.
BRIGHT
A reasoning-intensive retrieval benchmark. It requires genuine reasoning to connect queries with relevant documents rather than keyword or semantic overlap, revealing model weaknesses that BEIR misses.
CoIR (Code Retrieval Benchmark)
A code retrieval benchmark spanning diverse programming tasks and languages. Built for evaluating how well models retrieve relevant source code, which matters if you are building RAG over codebases.
LongEmbed
A retrieval benchmark focused on very long documents, with an average length above 5,500 words. It addresses a gap in standard benchmarks, which mostly use short documents of at most 512 tokens (the chunks of text a model reads at once).