FineWeb / FineWeb-Edu

FineWeb is HuggingFace's cleaned rebuild of Common Crawl: dozens of monthly snapshots run through an open pipeline of text extraction, quality filtering, and deduplication (removing pages that are duplicates or near-duplicates so the same text is not stored twice). The result is roughly 15 trillion tokens of English web text, about 45 TB, which HuggingFace has shown matches or beats other open web corpora on downstream benchmarks.

FineWeb-Edu is a filtered slice of the same data. A classifier scores each page for how educational it looks, and only the higher-scoring pages are kept. It is much smaller than the full set but far denser in explanatory, textbook-style writing, which makes it a strong base when you care about factual, teachable content rather than sheer volume.

Both live on HuggingFace as Parquet files and stream directly with the `datasets` library, so you can pull a single shard and start chunking without downloading tens of terabytes first. The filtering and deduplication code (built on HuggingFace's `datatrove` library) is open, so you can trace exactly why any given document survived, or rerun the pipeline with your own thresholds.

For RAG, FineWeb is a sensible default when you want broad, reasonably clean English web coverage without building your own crawl-cleaning pipeline. Reach for FineWeb-Edu when the retrieval quality of each passage matters more than covering every corner of the web, for example a study assistant or a general-knowledge question-answering system.

Two things to keep in mind. First, it is English only; for other languages you want FineWeb2, the multilingual sibling. Second, the licence is ODC-By 1.0, which permits commercial use with attribution to the dataset, but the underlying web pages keep their own terms, so treat the text as filtered web content rather than material cleared for any use. Next to C4 it is larger and more carefully deduplicated; next to raw Common Crawl it saves you the entire extraction and filtering effort.