The best open datasets for RAG in 2026
A curated tour of the open datasets we reach for most when building RAG and AI systems, from web-scale corpora and encyclopaedic knowledge to legal text and evaluation benchmarks.
Ask ten teams for the best open dataset and you will get ten answers, because “best” is not a property of a dataset on its own. It is a property of the match between a dataset and the job you are trying to do. A petabyte-scale web crawl is the wrong starting point for a legal assistant, and a tidy corpus of court judgments will not help a general chatbot. So this roundup is organised by the job to be done, not by size or fame. For each source we give one or two sentences: what it is, and when to actually reach for it. Two things run through every recommendation: a licence you can genuinely build on, and RAG-readiness, meaning how much cleaning, chunking, and structuring you will have to do before the data earns its place in a retrieval pipeline.
If you want the full catalogue rather than a shortlist, the directory holds everything with filters for licence and access. This post is the opinionated tour.
Broad general knowledge
When you need a dependable base layer of facts that a model can ground itself against, start with Wikipedia. It is well written, broad, regularly updated, and openly licensed, which makes it the default encyclopaedic corpus for almost any RAG (Retrieval-Augmented Generation) system that answers general questions. The article structure also chunks cleanly: sections and paragraphs map naturally onto retrieval units, so you spend less time fighting the source into shape.
Where Wikipedia gives you prose, Wikidata gives you structure. It is a large, openly licensed knowledge graph of entities and the relationships between them: people, places, organisations, dates, and the links that tie them together. Reach for it when you need exact facts rather than paragraphs, or when you want to enrich retrieved text with reliable identifiers and attributes. The two pair well: Wikipedia for explanation, Wikidata for the precise value. A common design is to answer from Wikipedia passages and use Wikidata to verify or fill in the specific numbers, dates, and identifiers that free text tends to state loosely. Both live in the encyclopaedic category if you want to see what else sits alongside them.
Web-scale coverage and pretraining
Some projects need coverage that no curated corpus can match, or they are training and fine-tuning a model rather than only retrieving. That is the territory of Common Crawl, a nonprofit that has been archiving the open web for years and publishes its snapshots freely. It is the raw material behind most large open pretraining corpora. The catch is in the name: it is raw. Expect to invest heavily in filtering, deduplication, and text extraction before it is usable.
If you would rather someone else did that work, FineWeb is a cleaned and filtered corpus derived from Common Crawl, built for language-model pretraining. It keeps the breadth of the open web while removing a great deal of the boilerplate and low-quality text that makes raw crawl data painful. As a rule: choose Common Crawl when you want maximum control over your own cleaning pipeline, and FineWeb when you want a sensible, ready-made version of the web without rebuilding that pipeline yourself. Both, along with the other large corpora, sit under web-corpora.
A note on licences here. Web crawl data carries the terms of the original pages it captured, which is not the same as the dataset itself being freely reusable for every purpose. It is worth reading our licensing trap piece before you commit a crawl-derived corpus to a commercial product.
Scholarly and scientific literature
For technical accuracy, retrieval over the research literature is hard to beat, and the open options are strong. arXiv is the preprint server for physics, mathematics, computer science, and related fields: full-text papers, openly available, and the first place to look for a research-grounded assistant in those domains. Reach for it when your users ask questions that live at the frontier rather than in a textbook.
To navigate the literature rather than read it in full, two catalogues stand out. Semantic Scholar offers a large corpus of paper metadata and abstracts with a citation graph, which is useful when you want to rank, cluster, or follow the thread of related work. OpenAlex is an open index of scholarly works, authors, and institutions, and it is a good backbone when you need coverage across every discipline rather than one field. A common pattern is to retrieve candidate papers from one of these indexes, then pull full text from arXiv or an open repository for the passages you actually cite. The wider academic category lists the full-text archives and citation databases together.
Domain depth: legal as the worked example
General knowledge gets you a competent generalist. Domain depth gets you an expert, and legal work is the clearest illustration of why the distinction matters. Legal answers must cite the right instrument, in the right jurisdiction, at the right point in time, so the corpus has to be authoritative and current, not merely large.
- Pile of Law is a large, openly available collection of legal and administrative text: case law, statutes, and related documents, assembled with training and retrieval in mind. Reach for it when you want breadth across many types of legal writing in one place.
- Legislation.gov.uk publishes UK legislation as official, structured, openly licensed text. It is the source to reach for when a UK answer has to reflect the statute as enacted or amended, rather than a paraphrase of it.
The lesson generalises beyond law. Whatever your domain, look for the primary, authoritative publisher first, then supplement with broader corpora. The legal category collects jurisdictions and instrument types, and the same instinct, primary source before aggregation, applies to biomedical, government, and financial work too.
Code and developer Q&A
If your system answers programming questions, retrieval over developer discussion is often more useful than reference documentation alone, because it captures how people actually phrase and solve problems. Stack Exchange publishes its network of question-and-answer sites, including Stack Overflow, under an open licence, complete with accepted answers, votes, and tags. Reach for it when you want grounded, practical answers with real-world phrasing, and lean on the votes and accepted-answer flags as a built-in quality signal when you rank retrieved passages. It sits in the code category alongside source-code corpora and technical documentation sets.
Public-domain text
When you need long-form prose that is unambiguously free to use, Project Gutenberg is the standby: tens of thousands of public-domain books, mostly older literature, in clean plain text. It is a poor choice for current facts, but a good one for narrative style, historical language, or any application where licence certainty matters more than recency. Because the text is public domain, you avoid most of the reuse questions that shadow web and social data. It is also a convenient test bed: the documents are long and coherent, which makes it a good place to experiment with chunking strategies before you point them at your real corpus. If chunking is new to you, our lesson on how to chunk your documents covers the trade-offs.
Evaluation: knowing whether any of this works
Choosing data is only half the task. You also need to measure whether retrieval is finding the right passages, and that calls for benchmarks rather than more training data. BEIR is a heterogeneous benchmark that bundles many retrieval tasks and domains, and it is the standard way to check whether an embedding model or retriever generalises beyond the one dataset you tuned it on. MS MARCO, built from real search queries with relevance judgments, is a workhorse for training and evaluating passage retrieval and ranking. Use BEIR to test breadth and MS MARCO to sharpen the core ranking. Both live in the rag-benchmarks category, and it is worth wiring at least one of them into your pipeline before you trust any retrieval numbers.
The discovery hub
No single roundup can keep pace with what gets published, and much of it flows through one place. HuggingFace Datasets is the hub where a large share of open datasets are hosted, versioned, and documented, with a consistent loading interface across all of them. Reach for it when you are hunting for something specific, comparing variants of a corpus, or want to load data without writing a bespoke parser. Treat it as the index to search, and this directory as the curated shortlist that tells you which entries are worth your time.
Where to go next
The right dataset is the one that fits your use case, your licence constraints, and how much preparation you can afford, in that order. Start narrow with the source that matches your job, add a broad corpus for coverage, and keep an evaluation set close so you can tell whether changes help. If you are still deciding, choosing your first RAG dataset walks through the trade-offs, and the DIY RAG course covers the pipeline from raw data to answers. For the complete picture, browse the directory or dip into a category that matches your domain and see what fits.