Reranking and two-stage retrieval: precision after recall

Retrieval quality sets the ceiling on everything a Retrieval-Augmented Generation (RAG) system can do. If the passage that answers the question never reaches the context window, no amount of prompt engineering or model scale will recover it. The two-stage retrieval pattern exists because the single model that would rank a whole corpus perfectly is too expensive to run over a whole corpus. So you split the job: cast a wide net cheaply, then sort the catch carefully.

The bi-encoder trade-off

Most first-stage retrievers are bi-encoders, also called dual encoders. A bi-encoder passes the query through one tower and each document through another, producing a single dense vector per input. Relevance is then a dot product or cosine similarity between two vectors. Because every document embedding can be computed offline and indexed, retrieval at query time is a nearest-neighbour lookup: fast, and scalable to billions of vectors. Dense Passage Retrieval (Karpukhin et al., 2020) is the canonical demonstration that a learned dual encoder beats BM25 on open-domain question answering.

The compression is also the weakness. Squeezing a passage into one vector forces the model to decide, ahead of time and without seeing the query, which of the passage’s meanings to encode. Fine-grained signals get averaged away. A query term that matters intensely to relevance but only appears once in the document is easy to lose. The query and the document never interact until the final similarity score, so the model cannot condition its reading of one on the other.

BM25 sits in the same architectural slot for different reasons. It is a sparse lexical scorer with no learned semantics at all, but it is fast, needs no training, and remains a stubbornly strong first stage, especially for rare terms, codes, and exact phrases that dense models blur. Many production systems run both and fuse the results.

The cross-encoder

A cross-encoder inverts the design. It concatenates the query and a candidate document into a single sequence and feeds the pair jointly through a transformer, so every query token can attend to every document token. The output is one relevance score. This full interaction is exactly what the bi-encoder gives up, and it is consistently more accurate. monoBERT (Nogueira and Cho, 2019) showed that a BERT cross-encoder reranking passage candidates produced a large jump in ranking quality on the MS MARCO passage task. monoT5 (Nogueira et al., 2020) reframed the same idea as sequence-to-sequence generation, training the model to emit a relevance token whose logit serves as the score, with strong zero-shot transfer to collections it was never trained on.

The cost is the mirror image of the benefit. Because the score depends on the query, nothing can be precomputed. Ranking a corpus of ten million documents would mean ten million forward passes through a large transformer per query. That is a non-starter online. A cross-encoder is only viable over a shortlist.

Retrieve, then rerank

Hence the standard pattern. Stage one retrieves the top k candidates cheaply, with a bi-encoder, BM25, or a hybrid of both. Stage two reranks those k candidates with a cross-encoder and returns the reordered top n to the generator. The first stage optimises recall: it only has to get the right documents somewhere into the shortlist. The second stage optimises precision: it decides the order that actually reaches the model.

The division of labour is what makes the economics work. You pay for full query-document interaction on k candidates, not on the corpus. If k is a few hundred and n is a handful, the reranker runs a bounded, predictable number of forward passes regardless of how large the underlying index grows.

Choosing k is the central tuning decision. Too small and the first stage’s recall failures become unrecoverable: the reranker can only reorder what it is given, so a gold passage ranked 200th by the retriever is invisible if k is 100. Too large and reranking latency climbs without improving the final answer, because the extra candidates were never plausible. In practice teams often start around k of 100 to 200 for the retriever and rerank down to a top n of 3 to 10 for the context window, then sweep those numbers against their own evaluation set. The right values depend on first-stage recall, reranker throughput, and how many passages the generator can actually use.

Late interaction: a middle ground

Between the two extremes sits late interaction, which asks whether you can keep some of the cross-encoder’s fine-grained matching without giving up precomputation. ColBERT (Khattab and Zaharia, 2020) is the reference design. Instead of collapsing a document to one vector, ColBERT keeps a contextual embedding for every token. At query time it does the same for the query, then scores a document with a MaxSim operation: for each query token, find its maximum similarity against any document token, and sum those maxima. The token embeddings are still computed independently, so document representations can be indexed offline, but the matching step recovers much of the term-level interaction that a single vector throws away.

The catch is storage. A vector per token is far larger than a vector per passage, and the index grows accordingly. ColBERTv2 (Santhanam et al., 2021) addresses this directly, pairing residual compression of the token vectors with a denoised supervision strategy, and reports substantial reductions in footprint while holding or improving quality. Late interaction has since become a practical option for teams that want stronger matching than a bi-encoder but cannot afford a cross-encoder pass over every query, and it can serve as either a first stage or a reranker depending on the pipeline.

None of these are strict tiers. A common architecture runs BM25 or a dense retriever first, a ColBERT-style late-interaction pass second, and a heavy cross-encoder only on a small final shortlist. Each stage narrows the field for the more expensive one behind it.

When the extra stage pays off

Reranking is not free and does not always earn its latency. It pays off most clearly when the first stage returns many plausible candidates whose ordering matters: queries with subtle intent, near-duplicate passages, or corpora where lexical overlap alone misranks results. It helps least when the first stage is already precise, when only one document could possibly answer, or when your latency budget cannot absorb an extra model call in the request path.

Weigh it against the alternatives before reaching for it. Sometimes a better first-stage embedding model, a hybrid of dense and lexical retrieval, or fusing multiple retrievers closes enough of the gap that a reranker adds little. The way to know is to measure recall at your chosen k separately from the final ranking quality, so you can see whether errors come from the net or from the sorting.

Latency is the other side of the ledger. A cross-encoder over 100 candidates is a real cost on the critical path, though it parallelises well and can be run on batched hardware. Late interaction is cheaper per candidate but carries the index-size cost instead. Distilled or smaller rerankers trade a little accuracy for meaningfully lower latency, and are often the pragmatic choice for interactive systems.

Measuring it honestly

You cannot tune a two-stage pipeline without a benchmark that separates the stages. MS MARCO, built from real Bing queries with human relevance labels, is the training and evaluation bedrock for most rerankers; its passage ranking task is where monoBERT, monoT5, and ColBERT were all measured. BEIR (Thakur et al., 2021) collects diverse retrieval datasets specifically to test zero-shot generalisation, and it is the standard check on whether a retriever or reranker holds up outside the domain it was trained on. MTEB (Muennighoff et al., 2022) extends the same spirit across embedding tasks, including a dedicated reranking track, and its leaderboard is the fastest way to compare candidate models before you commit to integrating one. For code-heavy retrieval, purpose-built suites like CoIR fill a gap the general benchmarks leave open.

Treat the public leaderboards as a shortlist, not a verdict. A model that tops BEIR or MTEB is a strong starting point, but your corpus, query distribution, and latency budget are not theirs. Browse the retrieval benchmarks and RAG-specific benchmarks categories to find an evaluation set close to your domain, then measure first-stage recall and final ranking quality on your own data. The rest of the directory and the wider research section cover the retrievers, embeddings, and corpora that feed the pipeline these two stages sit on top of.