Massive Legal Embedding Benchmark (MLEB)
The Massive Legal Embedding Benchmark (MLEB) is an evaluation suite for legal embedding and retrieval models, published by Isaacus in 2025. It gathers ten datasets covering cases, legislation, regulatory guidance, contracts, and legal literature drawn from the United States, the United Kingdom, the European Union, Australia, Ireland, and Singapore. The constituent sets include Bar Exam QA, SCALR, Singaporean Judicial Keywords, GDPR Holdings Retrieval, Australian Tax Guidance Retrieval, Irish Legislative Summaries, UK Legislative Long Titles, Contractual Clause Retrieval, License TL;DR Retrieval, and Consumer Contracts QA. Between them they cover three problem types: retrieval, zero-shot classification, and question answering, at a range of difficulties from lexical matching to tasks that need genuine legal reasoning.
Each dataset is hosted on Hugging Face under the isaacus organisation, typically as query and corpus splits with relevance labels, auto-served as Parquet and readable as JSON. The evaluation code lives in a public GitHub repository and runs with Python and the uv package manager, writing results in MTEB format so scores line up with the wider embedding-benchmark ecosystem. You supply API keys for the embedding providers you want to test.
MLEB is an evaluation resource rather than a knowledge base to index, so its value for RAG work is measuring which embedding or reranking model retrieves legal text most reliably before you commit to one in a pipeline. Watch the licensing: Isaacus aims to license the benchmark and its parts as permissively as possible, but individual datasets carry their own terms (SCALR, for example, is CC BY 4.0) and the evaluation code is MIT, so check each set before reuse and expect attribution obligations on some. If you need broader or general-purpose comparisons, MTEB and BEIR sit alongside this, and LegalBench covers legal reasoning tasks rather than embedding retrieval.
Related sources
BEIR
A collection of 18 information retrieval datasets spanning ad-hoc web search, question answering, fact verification, and duplicate question retrieval. Built by aggregating existing datasets, some originally created for other tasks and converted to retrieval format. Tests how well retrieval models generalise to unseen domains without fine-tuning, which matters for real-world RAG.
BRIGHT
A reasoning-intensive retrieval benchmark. It requires genuine reasoning to connect queries with relevant documents rather than keyword or semantic overlap, revealing model weaknesses that BEIR misses.
CoIR (Code Retrieval Benchmark)
A code retrieval benchmark spanning diverse programming tasks and languages. Built for evaluating how well models retrieve relevant source code, which matters if you are building RAG over codebases.
FreshStack
A 2025 framework and benchmark for retrieval over fast-moving technical documentation. It pairs real Stack Overflow questions and answers with chunked GitHub code and docs across five niche domains, giving code and docs RAG a hard, contamination-resistant test.