Visual document retrieval: RAG over pages, not text

Ask most retrieval-augmented generation (RAG) pipelines to work over a PDF and they do the same thing: run optical character recognition (OCR) or a text extractor, split the resulting string into chunks, embed each chunk, and index it. It works well for prose. It fails quietly on everything else. A different line of work skips the text extraction entirely and embeds the rendered page image, letting a vision-language model match a query against layout, tables and figures directly. This piece looks at how that works, at ColPali and the ViDoRe benchmark that popularised it, and at when it is worth the cost.

Where the text pipeline breaks

The extract-then-chunk approach makes an assumption that a document is a linear stream of words. Real documents are not. A financial report puts the number you want inside a table with merged cells and a footnote. A scientific paper carries half its meaning in a figure. A scanned contract has no text layer at all, only pixels, and the OCR pass introduces errors before chunking even begins.

Each of these breaks a different part of the pipeline. Multi-column layouts get read in the wrong order, so sentences interleave. Tables collapse into a run of numbers with no rows or columns, and a query about a specific cell has nothing to match. Charts and diagrams vanish completely, because there is no text to extract. Even when OCR succeeds, it discards the spatial structure (what sits next to what, what is a heading, what is a caption) that a human reader uses without noticing.

None of this is a chunking problem, so no chunking strategy fixes it. The information is lost upstream, at the extraction step, before any chunk is ever cut. If the table structure never survived OCR, no amount of clever splitting will bring it back.

Embedding the page as an image

The alternative is to treat the page as what it is: an image. A vision-language model (VLM) encodes the rendered page directly, producing embeddings that reflect both the text on the page and its visual arrangement. Retrieval then matches the query against those embeddings, so a question about a chart or a table cell can hit the page that contains it without anyone having parsed the chart into words first.

Two 2024 papers framed the idea clearly. Document Screenshot Embedding (Ma et al., 2024) proposed encoding a document screenshot as a single unified input, with no content-extraction preprocessing, and reported large gains over OCR-text retrieval on a mixed-modality slide-retrieval task. VisRAG (Yu et al., 2024) built a full pipeline around the principle: embed the document as an image with a VLM, retrieve, then generate with a VLM, arguing that this retains information that text parsing throws away. Both point at the same conclusion, that the parsing step is where the loss happens, and removing it helps.

ColPali and late interaction over patches

The work that turned this into a practical recipe is ColPali (Faysse et al., 2024), presented at ICLR 2025. Its design borrows an idea from text retrieval that RAG practitioners will recognise from two-stage ranking.

Standard dense retrieval pools a whole passage into one vector. ColBERT (Khattab and Zaharia, 2020) instead keeps one vector per token and scores a query against a document with a late-interaction operator called MaxSim: for each query token, take the maximum similarity against any document token, then sum those maxima. This token-level matching is more expressive than comparing two pooled vectors, and it is the same mechanism that powers multi-vector rerankers. We covered how that fits into a pipeline in the piece on reranking and two-stage retrieval; ColPali is essentially that idea moved from text tokens to image patches.

ColPali runs a page through a VLM (the original built on a SigLIP vision encoder feeding a small language model) and keeps a separate low-dimensional embedding for each image patch, roughly a thousand vectors per page rather than one. At query time the text query is embedded token by token, and relevance is the MaxSim between query tokens and page patches. A query about “revenue by region” can align its tokens with the patches that render that part of a table, without the table ever being converted to text. Because the page is never OCR’d, scanned documents and figure-heavy pages are handled on the same footing as clean digital text.

Follow-ups swapped in stronger backbones. The ColQwen models, part of the same ColVision family, apply the ColPali late-interaction recipe on top of Qwen2-VL and Qwen2.5-VL, and report several nDCG@5 points of improvement over the original ColPali on ViDoRe. The pattern is now a template: take a capable open VLM, add a projection to per-patch multi-vectors, train with a contrastive objective, and score with MaxSim.

ViDoRe: measuring it

ColPali arrived with its own yardstick, the Visual Document Retrieval Benchmark (ViDoRe), a set of page-level retrieval tasks spanning several domains, languages and document types, scored with nDCG@5. It exists because the standard text-retrieval benchmarks do not test any of this. BEIR measures zero-shot text retrieval and MTEB measures text embedding quality across tasks, but both assume the document is already text; neither can tell you whether a system reads a chart. ViDoRe fills that gap, and it is catalogued here under retrieval benchmarks alongside those text-first suites. You can find its entry on the ViDoRe benchmark page, and it sits within the broader multimodal data collection in the directory.

The first ViDoRe release was quickly saturated as models improved, so ViDoRe Benchmark V2 (Macé et al., 2025) raised the bar with harder, non-overlapping tasks, multilingual queries and more realistic query complexity, and no in-domain training data. Reported state-of-the-art nDCG@5 on V2 sits well below the near-saturated V1 numbers, which is the point: a benchmark that everyone scores 85 on has stopped discriminating between systems. When you read a ViDoRe number, check which version it refers to, because V1 and V2 are not comparable.

The trade-offs are real

Visual retrieval is not free, and the cost lands almost entirely on the index. A single-vector text embedding stores one vector per chunk. ColPali-style retrieval stores roughly a thousand vectors per page. Even projected down to a small dimension, that is on the order of a few hundred kilobytes per page, so a million pages can mean hundreds of gigabytes of raw vectors before any compression. That storage has to be searched at query time too, and MaxSim over per-patch multi-vectors is heavier than a single dot product.

This has become an active research area precisely because the naive index is expensive. Work on quantisation, patch pooling and pruning aims to cut the footprint by large factors (int8 quantisation alone gives a 4x reduction, and more aggressive schemes claim far more), and serving systems have shown the approach scaling to very large collections with the right engineering. But the honest summary is that visual retrieval trades index size and compute for recall on visually rich content.

That trade only pays off when the content is actually visual. If your corpus is clean, born-digital text (documentation, wiki pages, transcripts, most of the datasets that are already plain prose) then OCR-free retrieval buys you little and costs you a lot; a good text embedding and a reranker will match it for a fraction of the index. Visual retrieval earns its keep on scanned archives, financial and regulatory filings, slide decks, scientific papers, forms and anything where tables, charts and layout carry the answer. Many real systems are mixed, and a sensible pattern is to route: send the visually rich document classes through a ColPali-style index and keep everything else on cheap text retrieval.

The direction of travel is clear. Embedding the page instead of a lossy transcription of it removes a failure mode that no downstream chunking or reranking could ever repair, and the benchmarks now exist to measure it. Whether it belongs in your stack comes down to one question: does the answer live in the pixels, or in the text you could have extracted anyway?

Further reading

• ColPali: Efficient Document Retrieval with Vision Language Models (Faysse et al., 2024). arXiv:2407.01449
• ViDoRe Benchmark V2: Raising the Bar for Visual Retrieval (Macé et al., 2025). arXiv:2505.17166
• VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents (Yu et al., 2024). arXiv:2410.10594
• Unifying Multimodal Retrieval via Document Screenshot Embedding (Ma et al., 2024). arXiv:2406.11251
• ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT (Khattab and Zaharia, 2020).