Reasoning-intensive retrieval: where dense search breaks

Dense retrieval has a comfortable failure mode: it works beautifully right up to the point where it stops working, and the point where it stops is easy to miss. On the benchmarks most teams use to pick an embedding model, the numbers look reassuring. In production, on a certain kind of query, recall quietly collapses. The reason is not a bug in your pipeline. It is a property of what embedding models were trained to do, and a benchmark called BRIGHT makes it impossible to ignore.

What embedders actually learned

An embedding model maps text into a vector space where nearness means similarity. Trained on hundreds of millions of query-document pairs, it learns that a question and its answer tend to share vocabulary, entities, and topic. Ask “what is the capital of Australia” and the passage containing “Canberra is the capital” sits close by. This is the regime that standard evaluation rewards. On MS MARCO (a passage-ranking dataset built from Bing queries and web answers), and across the tasks aggregated in MTEB, the Massive Text Embedding Benchmark, the relevant document usually looks like the query. Semantic or even keyword overlap is enough.

That assumption holds for a large share of real retrieval, which is why dense search became the default. It also means the benchmarks that certify these models are, structurally, tests of surface matching. BEIR, the zero-shot retrieval suite that preceded MTEB, samples diverse domains but keeps the same information-seeking shape: the answer is a passage that states the answer. You can browse the whole family in our retrieval benchmarks category, and the pattern repeats. When people report an embedding model’s score, they are almost always reporting how well it matches meaning, not how well it reasons about relevance.

The BRIGHT gap

BRIGHT (Su et al., 2024) was built to break that assumption on purpose. The benchmark collects roughly 1,400 real-world queries where finding the right document requires multi-step reasoning rather than similarity. The queries come from naturally occurring human data across mathematics, coding, economics, psychology, robotics, and the natural sciences: StackExchange threads, competition maths problems, and LeetCode-style programming questions. Crucially, the truly relevant document often shares almost no vocabulary with the query. A hard maths problem is best served by a passage stating a general theorem or technique; a coding question is best served by documentation of an underlying principle, not a passage that repeats its words.

The headline result is stark. SFR-Embedding-Mistral, a model near the top of the MTEB leaderboard with a score around 59.0 nDCG@10 there, scores just 18.3 nDCG@10 on BRIGHT. That is not a modest regression. It is a model performing on reasoning-intensive queries at roughly a third of its headline competence, and the pattern holds across other strong retrievers. The BRIGHT source page collects the paper, leaderboard, and dataset; the contrast with the MTEB entry and the BEIR entry is the whole point. A number that predicts production quality on MS MARCO-shaped traffic tells you very little about the reasoning tail.

Why the vectors miss

The failure is legible once you accept what the model optimised for. Relevance in BRIGHT is often a derived relationship, not a stated one. To know that a passage about modular arithmetic answers a particular competition problem, you have to do the mathematics first, at least far enough to recognise which principle applies. The embedding of the raw query lands nowhere near the embedding of the principle, because on the surface they are about different things. No amount of better contrastive training on similarity pairs teaches a model to close that gap, because the gap is not a similarity gap. It is an inference gap.

This is also why lexical retrieval refuses to die. On BRIGHT, BM25 (a sparse keyword scorer with no learned semantics) is competitive with, and on some domains better than, expensive dense models. When relevance depends on a term of art that both query and document happen to name, exact matching can beat a smooth semantic space that has blurred the distinction away. The lesson is not that embeddings are bad. It is that similarity and relevance are different quantities, and standard benchmarks let teams conflate them for years without penalty.

Reranking cannot rescue what was never retrieved

The instinct, when top-k quality disappoints, is to add a reranker: a heavier cross-encoder or an LLM that reads each candidate against the query and reorders them. Two-stage retrieval is genuinely powerful, and we cover it in depth in reranking and two-stage retrieval. But it sits strictly downstream of the first-stage retriever, and that ordering is the trap here.

A reranker can only reorder the shortlist it is given. If reasoning-intensive relevance never made the initial candidate set, no reranker recovers it, because the right document is not in the list to be promoted. The BRIGHT authors observe exactly this asymmetry: LLM reranking adds only a few points on top of a first stage that has already surfaced the right neighbourhood, whereas fixing the first stage moves the numbers far more. Reranking improves precision within a good shortlist. It does nothing for recall failures upstream of it. On reasoning-intensive queries, recall is precisely what breaks, so the money has to go into the retriever, not the reorderer.

Bridging the query and the document

The most effective mitigations all do the same thing: they spend inference to close the reasoning gap before the vector search happens, so the query that hits the index already resembles the target document.

The simplest version is reasoning-augmented query expansion. Instead of embedding the raw query, you ask a language model to think about it first, drafting the reasoning steps or the likely form of the answer, then retrieve using that expanded text. On BRIGHT, using chain-of-thought reasoning from a strong model as the query improves retrieval by up to 12.2 nDCG points, with sparse retrievers like BM25 gaining the most, because the reasoning trace introduces the vocabulary of the principle the query was missing. ThinkQE (Lei et al., 2025) formalises this as an evolving thinking process that refines the expansion using feedback from the corpus itself, rather than a single fixed rewrite.

A second line trains the retriever to reason rather than bolting reasoning on at query time. RaDeR (Das et al., 2025) builds reasoning-aware dense retrieval models using retrieval-augmented reasoning trajectories from an LLM solving maths problems, generating hard negatives that force the model to learn derived relevance. ReasonEmbed (Chen et al., 2025) pushes this further with synthetic data designed to avoid trivial pairs and a training scheme weighted by each sample’s reasoning intensity; its strongest configuration reaches 38.1 nDCG@10 on BRIGHT, roughly double the off-the-shelf embedder and a sign the gap is closable, if not yet closed.

All of these share a cost: an extra LLM call, or a heavier model, in the hot path of retrieval. That is a real budget decision, and it belongs in your evaluation from the start rather than as a later patch. This is the same discipline we argue for in evaluating your RAG system: measure on queries that look like your hardest traffic, not on the benchmark that flatters your model.

What to take back to your own system

Most RAG workloads are not reasoning-intensive, and for them a good embedding model over a clean corpus, with the datasets in our directory as raw material, is the right and cheap answer. The failure BRIGHT exposes is specific: queries where the useful document is connected to the question by inference rather than by shared meaning. Maths, code, law, medicine, and anything where a general principle answers a specific case are the domains to watch.

Three practical checks follow. First, know which kind of retrieval you are actually doing, because a strong MTEB score does not certify the reasoning tail. Second, if that tail matters, spend on the first stage: reasoning-augmented queries, a reasoning-aware retriever, or hybrid retrieval that keeps BM25 in the mix, before you spend on a reranker that can only sort what the retriever already found. Third, build a small evaluation set of your own hardest queries and measure recall on it directly. BRIGHT is the warning that a benchmark can look solved while your real problem is untouched. The number that matters is the one measured on the queries you actually get.