RepLiQA
RepLiQA (Repository of Likely Question-Answer) is a contamination-resistant question answering benchmark published by ServiceNow Research. Its premise is simple: standard QA benchmarks leak into model training data, so a high score may reflect memorisation rather than genuine retrieval and grounding. RepLiQA sidesteps this by using documents that human annotators wrote about invented people, places and events that do not appear anywhere on the web, paired with questions whose answers can only be found by reading the supplied text. Each document carries five questions, and roughly a fifth of them are deliberately unanswerable, so a model is expected to say when the document does not contain the answer rather than guess.
The dataset is released in five splits (repliqa_0 to repliqa_4), together holding about 90,000 question-answer pairs over roughly 3,600 documents per split, spanning 17 topic categories in English. ServiceNow held later splits back and released them in stages across 2024 and 2025, so that a freshly published split postdates a given model's training cut-off, preserving the clean-room property over time. You access it through the Hugging Face datasets library, pandas or the dataset viewer. The documents arrive as Parquet with the source PDFs alongside, and the whole release is about 1.31 GB.
For RAG and AI work it is most useful as a diagnostic. Because the knowledge genuinely is not in the model's parameters, it isolates how well a system retrieves the right passage and grounds its answer in it, and it rewards honest abstention on unanswerable questions, which matters for production systems. It is best for stress-testing retrieval and grounding rather than as a knowledge base to deploy. The licence is CC BY 4.0, so commercial use is permitted provided you credit the source, and because the documents are human-authored about fictional scenarios there are no entangled third-party content rights. It sits naturally beside the other retrieval benchmarks we list, such as BEIR and BRIGHT, as the contamination-control member of that set.
Related sources
CRAG (Comprehensive RAG Benchmark)
Meta's factual question answering benchmark of 4,409 QA pairs across five domains, paired with mock web-search and knowledge-graph retrieval APIs. It is distinct from "Corrective RAG (CRAG)", which is a retrieval method, not a dataset.
FinanceBench
A benchmark for evaluating LLMs and retrieval systems on questions about public financial filings. The full set holds 10,231 question, answer and evidence triplets; a 150 example sample is released openly.
FlashRAG Benchmark Datasets
A research toolkit that bundles standardised, pre-processed benchmark datasets for RAG evaluation across many scenarios. Every dataset comes in a single unified JSONL format, so you can swap benchmarks in and out without rewriting your data loading each time.
FRAMES (Fact, Fetch, and Reason)
A Google evaluation set of 824 challenging questions that tests factuality, multi-hop retrieval and reasoning together, each paired with a gold answer and the Wikipedia articles needed to answer it. Built to measure end-to-end RAG.