Massive web-crawled datasets used as foundation data for LLM pretraining and broad-coverage RAG knowledge bases. These range from raw crawl archives to heavily filtered and deduplicated corpora.
20sources
Web corpora are the raw material behind most large language models, and they double as broad, general-purpose knowledge bases for RAG. They range from raw crawl archives, where you get the web more or less as it was found, to heavily filtered and deduplicated corpora that have already thrown away the boilerplate, spam, and near-duplicate pages. The right choice depends far more on how much cleaning you want to do yourself than on raw size.
For a RAG system, bigger is rarely better. You almost never need petabytes of unfiltered crawl data; you need clean, relevant passages. That usually makes a filtered corpus like FineWeb or C4 a better starting point than raw Common Crawl, because someone has already done the expensive quality work. Reach for the raw archives only when you need full control over filtering, or coverage that the curated sets deliberately trimmed.
Two things deserve a close look before you commit. First, language: several of these corpora are English-first, so if you are building for a multilingual audience, check the language breakdown or start from a multilingual set. Second, licensing: the dataset’s own licence (often ODC-By) usually covers the compilation, while the underlying page content stays subject to each original site’s terms. That gap matters if you are building something commercial, so treat “open dataset” and “open to reuse however you like” as different questions.
The sources below span that whole spectrum, from the foundational crawl everything else is built on, through the cleaned and deduplicated corpora most teams actually reach for.
A 750 GB English corpus derived from Common Crawl using heuristic cleaning to keep natural language and drop gibberish, boilerplate, and placeholder text. Built to train Google's T5 and later used for MPT-7B and others.
An open, multilingual pretraining corpus of roughly 2.27 trillion tokens, built only from public-domain and permissively licensed text with documented provenance for every document. Maintained by PleIAs.
A nonprofit that crawls the web and freely provides its archives and datasets. Petabytes of raw web data from billions of pages, with a new snapshot each month. Used in the training of GPT-3, LLaMA, T5, and many other large language models.
An 8 TB corpus of public domain and openly licensed text drawn from 30 sources, assembled by EleutherAI as a licence-transparent alternative to web-scraped pretraining data. Used to train the Comma family of language models.
A filtered English web dataset from the DataComp-LM benchmark project, produced by running model-based quality filtering over Common Crawl. Built to show which data-curation choices most improve language model training.
A 3T token open corpus from Allen AI (its name stands for "Data for Open Language Models' Appetite") combining web text, scientific papers, code, public-domain books, Reddit posts, and Wikipedia. Built to train the OLMo models.
A 24-trillion-token web corpus (about 23.6 billion documents) from Essential AI where every document carries a twelve-category taxonomy covering topic, format, complexity and quality. The labels let you carve out domain or quality subsets with simple filters, without training your own classifiers.
HuggingFace's corpus built entirely from PDFs: roughly 3 trillion tokens across 475 million documents in 1,733 languages, drawn from Common Crawl and the open web. It captures dense long-form content (reports, manuals, papers) that HTML web corpora miss, and is the PDF counterpart to FineWeb.
A 15T token English web corpus distilled from Common Crawl by HuggingFace, filtered and deduplicated for language model training. FineWeb-Edu is a subset filtered for educational content. One of the higher-quality open web datasets for pretraining and broad-coverage RAG.
The educational-quality subset of FineWeb, about 1.3 trillion English tokens kept by an educational-quality classifier, with a larger 5.4T token variant at a looser threshold. Built by HuggingFace as a denser, more teachable base for pretraining and broad-coverage RAG.
A multilingual extension of FineWeb from HuggingFace, built with an open curation pipeline that adapts filtering and deduplication across languages. Useful when you need clean web text beyond English for pretraining or multilingual RAG.
A permissive-first, open pretraining corpus of roughly 422 billion tokens, drawn from public-domain, permissively licensed, and civic or government text, with sources organised into provenance-based risk tiers.
NVIDIA's rebuilt Common Crawl corpus for LLM pretraining, adding eight fresh 2024 to 2025 snapshots, synthetic rephrasing, and multilingual synthetic question and answer data, with mathematics and code content preserved.
A multilingual web corpus (its name stands for Open Super-large Crawled Aggregated coRpus) extracted from Common Crawl with per-document language classification. Covers more than 150 languages, useful for non-English and cross-lingual RAG.
An open reproduction of the LLaMA training data from Together. V1 aggregates Wikipedia, books, arXiv, GitHub, Common Crawl, and Stack Exchange; V2 is a web-only corpus of more than 100T tokens with 46 quality signals per document.
A filtered English web dataset from the Technology Innovation Institute, creators of the Falcon models. It showed that carefully cleaned web-only data can match curated multi-source corpora for language model training.
A cleaned and deduplicated version of RedPajama-V1 from Cerebras. It drops short documents and removes near-duplicate content, cutting 1.2T tokens down to a denser 627B tokens without losing coverage.
An 825 GB curated English text dataset from EleutherAI, made of 22 sub-datasets spanning books, academic papers, code, web content, and more. Built as a diverse training corpus and widely used to train early open language models.
An open pre-training corpus of 15 trillion-plus tokens from LLM360, built by globally deduplicating 99 Common Crawl snapshots and blending in 14 curated domains such as FreeLaw, PG-19, Wikipedia, and scientific papers. Ships with a fully documented, reproducible processing recipe.
Zyphra's five trillion token English pretraining mixture, built by filtering and cross-deduplicating DCLM, FineWeb-Edu, Zyda-1 and Dolma's Common Crawl portion. A ready high-quality drop-in base that Zyphra reports beats its component datasets on downstream evaluation. Published on HuggingFace as Parquet under an open licence.