FineWeb2

FineWeb2 is HuggingFace's multilingual successor to FineWeb, and it is large: a filtered, deduplicated web corpus covering many hundreds of language and script combinations, all distilled from Common Crawl snapshots. Instead of applying rules tuned for English everywhere, the pipeline adapts its quality filtering and deduplication (stripping out duplicate and near-duplicate pages) language by language, so a lower-resource language is judged on its own terms rather than discarded by heuristics designed for English.

The data ships as Parquet files on HuggingFace, split by language, so you take only the configurations you need. With the `datasets` library you can stream a single language without downloading the whole corpus, which matters a great deal at this scale. Every record pairs the text with metadata such as the source URL and detected language, giving you room to filter harder, trace where a passage came from, or blend languages on purpose.

FineWeb2 earns its keep the moment your work has to reach past English. It is a solid base for a non-English or cross-lingual knowledge store, for reinforcing a base model's grip on a particular language, or for assembling a multilingual retrieval corpus without standing up your own crawl-and-clean pipeline. The per-language curation is what makes the smaller languages usable rather than raw.

It is still web text, so the usual realities apply: boilerplate creeps in, quality is uneven between languages, and no single page is guaranteed accurate, which makes it better as broad background than as a citable authority. On licensing, the dataset is released under ODC-By 1.0, which permits commercial use with attribution, but the underlying page content remains under each original site's terms, so the content licence differs from the dataset licence and you should redistribute the raw text with care.

For English specifically, reach for the original FineWeb or C4; for the widest low-resource coverage, pair FineWeb2 with GlotCC.