FinanceBench
FinanceBench is a benchmark built by Patronus AI to test how well language models and retrieval pipelines answer questions grounded in real financial documents. The full dataset comprises 10,231 question, answer and evidence triplets drawn from public filings for 40 United States listed companies, including 10-K, 10-Q and 8-K filings, earnings reports and earnings call transcripts. Each question is paired with a gold evidence span extracted verbatim from the source document and indexed by page number, so answers can be traced back to their origin. Many questions are directly extractable, while others require light arithmetic or reasoning across a filing.
Access comes in two tiers. An open sample of 150 annotated examples is published on GitHub and Hugging Face as JSONL question and metadata files alongside the source PDF documents and an evaluation notebook. The full 10,231 question set is not freely downloadable: Patronus AI asks that you contact them to evaluate models against it, which is why this entry is marked as limited access rather than open.
For RAG work the benchmark is most useful as an evaluation harness rather than a knowledge base. The evidence spans let you measure retrieval accuracy and answer faithfulness separately, which is valuable given the paper reported that strong models paired with retrieval failed a large share of sample questions. The main watch-out is the licence: the open sample carries a CC BY-NC 4.0 licence, so commercial use is not permitted, and the underlying filings retain their own terms as company disclosures. Treat it as a research and internal evaluation resource, not a commercial training corpus.
If you need a fully open financial QA set, consider FinQA or TAT-QA, both of which target numerical reasoning over financial reports and ship complete training and test splits under more permissive terms.
Related sources
CRAG (Comprehensive RAG Benchmark)
Meta's factual question answering benchmark of 4,409 QA pairs across five domains, paired with mock web-search and knowledge-graph retrieval APIs. It is distinct from "Corrective RAG (CRAG)", which is a retrieval method, not a dataset.
FlashRAG Benchmark Datasets
A research toolkit that bundles standardised, pre-processed benchmark datasets for RAG evaluation across many scenarios. Every dataset comes in a single unified JSONL format, so you can swap benchmarks in and out without rewriting your data loading each time.
FRAMES (Fact, Fetch, and Reason)
A Google evaluation set of 824 challenging questions that tests factuality, multi-hop retrieval and reasoning together, each paired with a gold answer and the Wikipedia articles needed to answer it. Built to measure end-to-end RAG.
Open RAGBench (Vectara)
A RAG evaluation benchmark from Vectara built on 1,000 arXiv papers, with multimodal content extraction and 3,045 question-and-answer pairs across scientific domains. Designed to test retrieval and answer quality on real research documents rather than short, simplified passages.