ViDoRe (Visual Document Retrieval Benchmark)
ViDoRe (Visual Document Retrieval) is an evaluation benchmark for retrieving document pages by their visual appearance rather than by extracted text. It was released alongside the ColPali paper by ILLUIN Technology and is designed to test systems that embed page images directly, skipping the usual OCR-to-text pipeline. The suite gathers a set of page-level retrieval tasks spanning domains such as scientific figures, infographics, industrial and financial documents, and government reports, with queries in several languages including English, French, Spanish, and German.
Access is open. The evaluation code and task definitions live on GitHub, while the task data is published as individual datasets under the vidore organisation on the Hugging Face Hub. Each task provides document page images plus queries and relevance judgements, offered in both a QA format and a BEIR-style corpus and queries format, so you can plug them into standard retrieval evaluation loops. A public leaderboard tracks reported scores.
For RAG and AI work, ViDoRe is best used as a yardstick rather than a knowledge base: it is the reference benchmark for late-interaction visual retrievers such as ColPali and ColQwen, and it is a good way to compare a vision retriever against a classic OCR-plus-text-embedding pipeline on layout-heavy PDFs. Watch-outs: the evaluation code is MIT licensed, but the underlying task datasets are drawn from separate academic sources (for example DocVQA, InfoVQA, and arXiv figures) and carry their own, differing licences, so check each dataset card before any commercial reuse. Note also that a newer ViDoRe V2 and V3 exist with harder, longer-document tasks, so confirm which version a score refers to. If you need broader multimodal or text retrieval baselines already likely in a directory, consider BEIR and the MTEB retrieval tracks as complements.
Related sources
BEIR
A collection of 18 information retrieval datasets spanning ad-hoc web search, question answering, fact verification, and duplicate question retrieval. Built by aggregating existing datasets, some originally created for other tasks and converted to retrieval format. Tests how well retrieval models generalise to unseen domains without fine-tuning, which matters for real-world RAG.
BRIGHT
A reasoning-intensive retrieval benchmark. It requires genuine reasoning to connect queries with relevant documents rather than keyword or semantic overlap, revealing model weaknesses that BEIR misses.
CoIR (Code Retrieval Benchmark)
A code retrieval benchmark spanning diverse programming tasks and languages. Built for evaluating how well models retrieve relevant source code, which matters if you are building RAG over codebases.
FreshStack
A 2025 framework and benchmark for retrieval over fast-moving technical documentation. It pairs real Stack Overflow questions and answers with chunked GitHub code and docs across five niche domains, giving code and docs RAG a hard, contamination-resistant test.