RefinedWeb
RefinedWeb was built to test a simple idea: if you filter and deduplicate Common Crawl aggressively enough, do you still need books, papers, and code mixed in? The Technology Innovation Institute (TII) found the answer was largely no, and trained its Falcon models mostly on web data processed this way.
The pipeline leans on strict text extraction, heuristic quality filtering, and heavy deduplication (removing both exact and near-duplicate pages), producing clean English prose at scale. The public release on HuggingFace is a roughly 600B token subset of the full internal corpus, published as Parquet and streamable with the `datasets` library.
It is a strong, well-documented example of a web-only cleaning recipe, and a good source of general English text for pretraining or broad RAG knowledge bases.
Related sources
C4 (Colossal Clean Crawled Corpus)
A 750 GB English corpus derived from Common Crawl using heuristic cleaning to keep natural language and drop gibberish, boilerplate, and placeholder text. Built to train Google's T5 and later used for MPT-7B and others.
Common Crawl
A nonprofit that crawls the web and freely provides its archives and datasets. Petabytes of raw web data from billions of pages, with a new snapshot each month. Used in the training of GPT-3, LLaMA, T5, and many other large language models.
DCLM-Baseline (DataComp-LM)
A filtered English web dataset from the DataComp-LM benchmark project, produced by running model-based quality filtering over Common Crawl. Built to show which data-curation choices most improve language model training.
Dolma
A 3T token open corpus from Allen AI (its name stands for "Data for Open Language Models' Appetite") combining web text, scientific papers, code, public-domain books, Reddit posts, and Wikipedia. Built to train the OLMo models.