FineWeb / FineWeb-Edu
FineWeb takes raw Common Crawl snapshots and runs them through an open cleaning pipeline: text extraction, quality filtering, and deduplication (removing duplicate and near-duplicate pages so you are not storing the same text twice). The result is a 15T token English corpus that HuggingFace has shown matches or beats other open web datasets on downstream benchmarks.
FineWeb-Edu is a filtered slice of the same data, keeping only pages a classifier scores as educational. It is smaller but denser in explanatory, textbook-style prose, which makes it a useful base for knowledge-focused RAG and instruction tuning.
Both are published as Parquet files on HuggingFace and stream directly with the `datasets` library, so you can pull a shard without downloading the full 45 TB. The filtering and deduplication code is open, so you can trace exactly how each document survived.
Related sources
C4 (Colossal Clean Crawled Corpus)
A 750 GB English corpus derived from Common Crawl using heuristic cleaning to keep natural language and drop gibberish, boilerplate, and placeholder text. Built to train Google's T5 and later used for MPT-7B and others.
Common Crawl
A nonprofit that crawls the web and freely provides its archives and datasets. Petabytes of raw web data from billions of pages, with a new snapshot each month. Used in the training of GPT-3, LLaMA, T5, and many other large language models.
DCLM-Baseline (DataComp-LM)
A filtered English web dataset from the DataComp-LM benchmark project, produced by running model-based quality filtering over Common Crawl. Built to show which data-curation choices most improve language model training.
Dolma
A 3T token open corpus from Allen AI (its name stands for "Data for Open Language Models' Appetite") combining web text, scientific papers, code, public-domain books, Reddit posts, and Wikipedia. Built to train the OLMo models.