Open RAGBench (Vectara)

Open RAGBench is an evaluation benchmark from Vectara, built to test retrieval and answer quality on genuine scientific documents rather than the short, tidy passages many older benchmarks lean on. It draws 1,000 papers from arXiv, the open preprint server for research, and pairs them with 3,045 question-and-answer items across several scientific domains. The content extraction is multimodal, meaning it captures not only body text but also material from tables and figures, which mirrors how awkward real papers are to parse.

The dataset lives on Hugging Face and ships as Parquet, the columnar format that loads cleanly with pandas or the datasets library, so folding it into an evaluation harness takes little effort. A sensible workflow is to index the extracted document passages in your own vector database, run the questions through your retrieval and generation pipeline, then score the answers against the provided references. Because the source documents are long and technical, this stresses retrieval far harder than a Wikipedia-based set, where the right evidence is often one detail buried deep in a paper.

It is at its best when your target use case is scientific, technical, or academic document search and you want a benchmark that behaves like production. If you are tuning chunk sizes, comparing retrievers, or picking embedding models for research-heavy content, this is a realistic yardstick.

Keep its scope in view. This is a test and evaluation set, not training data or a knowledge base to serve to users, and 1,000 papers is a focused sample rather than broad coverage, so a strong score here signals competence on dense scientific text, not general-purpose RAG. The Apache 2.0 licence is refreshingly clear: commercial use is allowed, there is no share-alike obligation, and attribution is appreciated rather than strictly required, which makes it easy to drop into an internal pipeline.

Use it alongside broader suites: general question-and-answer benchmarks for open-domain retrieval, and Open RAGBench when you specifically need to prove out long, technical documents.