REAL-MM-RAG
REAL-MM-RAG is a real-world multimodal retrieval benchmark published by IBM Research (with collaborators at the Weizmann Institute of Science) to test systems that retrieve whole document pages by their visual appearance rather than by extracted text. It was introduced in a February 2025 paper and targets four properties the authors argue matter for realistic retrieval: genuinely multimodal pages, higher difficulty, queries that resemble what users actually ask, and accurate relevance labelling. The full benchmark spans roughly 8,000 pages split into four sub-domains, two financial and two technical, so a retriever must separate many near-identical pages from the same organisation rather than pick an obvious match. This dataset card is the FinReport part, built from IBM annual financial reports, and the sibling parts (financial slides, technical reports and technical slides) sit under the same ibm-research organisation on the Hub.
Access is open and programmatic. The data is hosted on the Hugging Face Hub as Parquet, pairing each page image with a question, a reference answer and up to three large-language-model rephrasings of the question, giving four levels of query phrasing of increasing distance from the source wording. You can pull it with the datasets library, Dask or the Croissant metadata, or browse it in the dataset viewer.
For RAG and AI work it is a yardstick, not a knowledge base: it is well suited to measuring page-image retrievers of the ColPali and ColQwen family, probing robustness to query rephrasing, and stress-testing performance on table-heavy finance pages where text-only pipelines tend to fail. The licence is the CDLA-Permissive-2.0, a permissive, commercial-friendly data agreement with no share-alike obligation, and because the documents are IBM's own published reports the content position is clean, which makes it a useful commercially-safe complement to ViDoRe (which we also list), whose underlying task datasets carry mixed, differing licences. Confirm which sub-domain a reported score refers to before comparing results.
Related sources
BEIR
The BEIR benchmark is a collection of 18 information retrieval datasets spanning ad-hoc web search, question answering, fact verification, and duplicate question retrieval. Built by aggregating existing datasets, some originally created for other tasks and converted to retrieval format. Tests how well retrieval models generalise to unseen domains without fine-tuning, which matters for real-world RAG.
BRIGHT
A reasoning-intensive retrieval benchmark. It requires genuine reasoning to connect queries with relevant documents rather than keyword or semantic overlap, revealing model weaknesses that BEIR misses.
CoIR (Code Retrieval Benchmark)
A code retrieval benchmark spanning diverse programming tasks and languages. Built for evaluating how well models retrieve relevant source code, which matters if you are building RAG over codebases.
FreshStack
A 2025 framework and benchmark for retrieval over fast-moving technical documentation. It pairs real Stack Overflow questions and answers with chunked GitHub code and docs across five niche domains, giving code and docs RAG a hard, contamination-resistant test.