RefinedWeb

RefinedWeb began as an experiment by the Technology Innovation Institute (TII), the Abu Dhabi lab behind the Falcon family of models. The team wanted to know whether aggressively cleaned web data alone could match the curated mixes of books, papers, and code that most pretraining recipes rely on. It could, and Falcon was trained largely on the result, which is why the dataset is worth a second look even now that newer web corpora exist.

What you get is filtered, English-language prose pulled out of Common Crawl. The pipeline runs strict text extraction to recover readable content from raw HTML, heuristic quality filters to drop boilerplate and low-value pages, and heavy deduplication that removes both exact and near-duplicate documents (pages that are almost identical, not just byte-for-byte copies). The public release on Hugging Face is a subset of roughly 600B tokens (a token is roughly a word or word-piece), a fraction of the full internal corpus but still very large.

Access is straightforward. The data ships as Parquet files and streams cleanly with the Hugging Face `datasets` library, so you can iterate over it without downloading the whole thing first. In practice, set `streaming=True`, sample a slice to sanity-check quality, then run your own topic or language filter before you chunk and embed. That keeps a retrieval index focused rather than diluted by general web chatter.

RefinedWeb suits projects that need broad, general English text: pretraining or continued pretraining of a base model, or a wide open-domain RAG corpus where coverage matters more than depth. Because it is web-only and English-only, it will not give you specialist domain material or multilingual reach, and the cleaning, though strong, cannot strip every trace of spam, SEO filler, or dated content.

On licensing, RefinedWeb is released under ODC-By 1.0, which permits commercial use but requires attribution, so credit the dataset in anything you ship. If you want a similar web-only corpus with different trade-offs, weigh it against FineWeb, C4, and Cerebras's SlimPajama, which mixes in books, papers, and code rather than staying web-only.