RAG evaluation gets specific: the 2026 wave of domain benchmarks
General leaderboards like BEIR and MTEB told you which retriever was good in the abstract. A wave of 2024 to 2026 benchmarks asks a harder question: good at what, for whom? A tour of domain, task, and contamination-resistant RAG evaluation, and how to use it.
For years, picking a retriever or an embedding model meant glancing at two leaderboards. MS MARCO for passage ranking, BEIR for zero-shot generalisation, and later MTEB for embeddings, were the shared yardsticks the whole field quoted. They were a genuine advance: before them, everyone measured on their own private set and no two numbers were comparable. But a top BEIR score has a way of evaporating the moment you point a system at real contracts, real financial filings, or a codebase that changed last week. The benchmarks released since 2024 are a response to exactly that gap. They are narrower, harder, and far more honest about what they do and do not tell you.
Why the general leaderboards stopped being enough
Three problems pushed the field past generic evaluation.
The first is the domain gap. General benchmarks are built from broad web and encyclopaedic text, so they reward models that match surface meaning across everyday language. Legal, financial, biomedical, and technical corpora use vocabulary, structure, and reasoning that those sets never exercise. A retriever that tops BEIR can still miss the one clause that answers a contract question.
The second is contamination. Public benchmarks are, by definition, public, so they end up in training data. When a model has seen the test set, its score measures memorisation rather than generalisation, and the leaderboard slowly stops predicting anything. This is the quiet reason a state-of-the-art number often fails to reproduce on your own data.
The third is that generic information-retrieval metrics miss the ways RAG actually fails. Recall@k says nothing about whether the model then hallucinated, whether the answer needed three hops to assemble, whether the evidence lived in a table or a chart, or whether the relevant document was published yesterday. The new benchmarks each take aim at one of those blind spots.
Domain-specific retrieval
The clearest trend is benchmarks that fix a single professional domain and test retrieval inside it.
LegalBench-RAG, released in 2024, was the first open benchmark built specifically for the retrieval step of legal RAG. It offers 6,858 human-annotated query and answer pairs over a corpus of roughly 79 million characters, with character-level ground-truth spans drawn from contract and privacy-policy datasets. Character-level spans matter: in law, retrieving the right document is not enough, you need the exact clause, and a benchmark that grades at that granularity measures something a document-level recall score cannot.
Finance has its own pair. T2-RAGBench provides 23,088 context-independent question, context and answer triples over more than 7,300 financial documents that deliberately mix text and tables, with each question mapping to exactly one ground-truth document. It is built for numerical, table-aware retrieval, the kind that trips up text-only pipelines. It sits naturally alongside FinanceBench, which tests end-to-end answers over public company filings.
FreshStack, from 2025, tackles fast-moving technical documentation. It pairs real Stack Overflow questions and answers with chunked GitHub code and docs across five niche domains, giving code and documentation RAG a deliberately hard test. Its design also addresses contamination, which we return to below.
Task- and failure-specific benchmarks
A second group fixes not a domain but a failure mode.
Faithfulness is the headline one. RAGTruth is a word-level hallucination corpus of roughly 18,000 LLM responses generated in a RAG setting, each manually annotated for hallucinated spans across question answering, summarisation, and data-to-text tasks. It exists because a system can retrieve the right passage and still write an answer the passage does not support, and measuring that gap is its own problem. We looked at this in depth in measuring RAG faithfulness.
Reasoning is another. BRIGHT is a reasoning-intensive retrieval benchmark: finding the relevant document requires genuine reasoning rather than keyword or semantic overlap, which is precisely the setting where strong embedders collapse and where a good score on BEIR tells you nothing. We unpacked why dense retrieval struggles here in reasoning-intensive retrieval.
Then there is modality. ViDoRe, introduced with the ColPali work, evaluates OCR-free retrieval directly over document page images, so tables, figures, and layout are part of what is retrieved rather than noise an OCR step discards. And for end-to-end factual answering, CRAG offers 4,409 question and answer pairs across five domains with mock retrieval APIs, while FRAMES folds factuality, multi-hop retrieval, and reasoning into a single test. CRAG is worth a licence note: it is released under a non-commercial licence, so it is fine for internal evaluation but not for a commercial product pipeline without separate permission.
Built to resist memorisation
The most forward-looking benchmarks treat contamination as a first-class design problem rather than an afterthought.
RTEB, launched in beta by the MTEB team in October 2025 as a new retrieval section of the leaderboard, is the clearest example. It spans 20 languages and enterprise domains such as law, healthcare, finance, and code, and it deliberately mixes open datasets with held-out private ones. The private half is the point: scores on data the model cannot have trained on are a far better proxy for how an embedder will generalise to your corpus. The trade-off is that those private results are not independently reproducible, which is an honest price for measuring generalisation rather than recall of the test set.
FreshStack attacks the same problem from the other direction, by drawing on fast-moving technical content that keeps moving, so a model cannot have memorised the answers during training. Between them they mark a shift in what a benchmark is for: not a number to top, but a measurement designed to stay meaningful after the model has seen the rest of the internet.
What this means for your own evaluation
The practical lesson is not that you should chase the newest leaderboard. It is that evaluation has become something you assemble to fit your problem.
Start by matching the benchmark to your failure mode. If your documents are full of tables, a table-aware set like T2-RAGBench tells you more than a general one. If hallucination is your risk, RAGTruth and a faithfulness metric matter more than recall. If your corpus changes weekly, a contamination-resistant set like FreshStack is the honest mirror. Browse the RAG benchmarks and retrieval benchmarks categories to find the one closest to your domain.
Then separate the two questions. Retrieval quality (did the right passage reach the context window) and end-to-end quality (was the final answer correct and supported) fail for different reasons and need different tests. Measuring them together hides which half is broken, a point we made in our guide to evaluating your RAG system, and one that interacts directly with reranking: a reranker can only reorder what the first stage surfaced.
Finally, treat every public benchmark as a shortlist, not a verdict. The single most reliable thing you can do is build a small evaluation set from your own documents and real user questions, because that is the only test guaranteed not to be in anyone’s training data. The 2026 wave of benchmarks is a better shortlist than the field has ever had. It is still not your production traffic.