RAGTruth
RAGTruth is a human-annotated corpus built to study hallucination in retrieval-augmented generation. It contains roughly 17,790 responses generated by a range of LLMs over about 2,965 source instances, spanning three task types: question answering, news and document summarisation, and data-to-text generation. Annotators marked hallucinations at the span (word) level rather than only labelling whole responses, recording around 14,289 hallucination spans and noting the intensity and type of each. The corpus ships with a fixed train and test split (roughly 15,090 training and 2,700 test items), which makes results comparable across papers.
Access is straightforward. The data lives in the ParticleMedia GitHub repository as JSONL files, chiefly a response file holding the generated text and its span annotations and a source-info file holding the query and the retrieved documents each response was conditioned on. There is no API or bulk download service: you clone or download the repository and join the two files on their shared identifiers.
For RAG and AI work it is best used as a labelled benchmark for hallucination and faithfulness detectors, whether you are training a span-level classifier, evaluating an LLM-as-judge, or measuring how often a grounding method reduces unsupported claims. Because responses come from several different models, it is also useful for comparing how faithfulness varies by generator and by task.
Watch-outs: the retrieved documents are drawn from other public datasets, so the MIT licence covers RAGTruth's own annotations and compilation rather than granting fresh rights over that underlying source text, check the original corpora if redistribution matters. The scale is modest by pretraining standards, and the responses reflect the LLMs available in 2023 to 2024. If you need alternatives already in this directory, HaluEval and TruthfulQA cover adjacent hallucination and truthfulness evaluation.
Related sources
CRAG (Comprehensive RAG Benchmark)
Meta's factual question answering benchmark of 4,409 QA pairs across five domains, paired with mock web-search and knowledge-graph retrieval APIs. It is distinct from "Corrective RAG (CRAG)", which is a retrieval method, not a dataset.
FinanceBench
A benchmark for evaluating LLMs and retrieval systems on questions about public financial filings. The full set holds 10,231 question, answer and evidence triplets; a 150 example sample is released openly.
FlashRAG Benchmark Datasets
A research toolkit that bundles standardised, pre-processed benchmark datasets for RAG evaluation across many scenarios. Every dataset comes in a single unified JSONL format, so you can swap benchmarks in and out without rewriting your data loading each time.
FRAMES (Fact, Fetch, and Reason)
A Google evaluation set of 824 challenging questions that tests factuality, multi-hop retrieval and reasoning together, each paired with a gold answer and the Wikipedia articles needed to answer it. Built to measure end-to-end RAG.