OSCAR
OSCAR runs Common Crawl snapshots through a language classifier and splits the text by language, giving you clean per-language slices rather than one mixed pile. This makes it one of the more practical starting points when you need web text in a specific language, especially languages that larger English-first datasets barely cover.
It spans more than 150 languages and is refreshed as new Common Crawl snapshots and pipeline versions land. Later releases add document-level annotations, such as quality and content flags, so you can filter before you index or train.
Access is through the OSCAR project site and HuggingFace, with some larger or more sensitive language sets gated behind a quick access request. The metadata is released under CC0, while the underlying page content stays subject to its original terms.
Related sources
C4 (Colossal Clean Crawled Corpus)
A 750 GB English corpus derived from Common Crawl using heuristic cleaning to keep natural language and drop gibberish, boilerplate, and placeholder text. Built to train Google's T5 and later used for MPT-7B and others.
Common Crawl
A nonprofit that crawls the web and freely provides its archives and datasets. Petabytes of raw web data from billions of pages, with a new snapshot each month. Used in the training of GPT-3, LLaMA, T5, and many other large language models.
DCLM-Baseline (DataComp-LM)
A filtered English web dataset from the DataComp-LM benchmark project, produced by running model-based quality filtering over Common Crawl. Built to show which data-curation choices most improve language model training.
Dolma
A 3T token open corpus from Allen AI (its name stands for "Data for Open Language Models' Appetite") combining web text, scientific papers, code, public-domain books, Reddit posts, and Wikipedia. Built to train the OLMo models.