MTEB (Massive Text Embedding Benchmark)

MTEB, the Massive Text Embedding Benchmark, is not a dataset you retrieve over but the standard yardstick for choosing the embedding model that will power your retrieval. Embeddings are the numeric vectors that represent meaning, and small differences between models translate into real differences in what your RAG system finds. MTEB pulls dozens of datasets into one framework spanning tasks such as retrieval, reranking, classification, clustering, and semantic similarity, folding in established suites like BEIR and SemEval so you can compare models on a like-for-like basis.

The public leaderboard is hosted on Hugging Face and updated as new models are submitted. Behind it sits an open Python library: you can install it, point it at any embedding model, and reproduce the scores or evaluate a model that is not yet listed, including one you have fine-tuned yourself. The MMTEB extension widens coverage to more than 250 languages and over 500 tasks, making it the most comprehensive public reference for embedding quality.

For RAG work, this is where you start when picking an embedder. The single most useful column is the retrieval score, since that is closest to what a RAG pipeline actually does, followed by reranking. Use the leaderboard to draw up a shortlist, then weigh the practical tradeoffs it does not capture: vector dimension (which sets your storage and memory), model size and latency, maximum input length, and licensing.

Treat the rankings as a guide, not gospel. Models can be tuned to score well on public benchmarks without matching that on your data, and a top-10 model on generic English text may lag on your domain, your language, or your document lengths. The honest move is to shortlist with MTEB, then run a small evaluation on your own corpus. The framework itself is Apache 2.0, so you can fold that evaluation into your own tooling freely.