Dolma
Dolma pulls together several sources into one documented corpus: web text from Common Crawl, scientific papers from Semantic Scholar, code from GitHub, public-domain books, Reddit conversations, and Wikipedia. It spans more than 4 billion documents totalling 3T tokens, so the mix inside it is genuinely varied, hence its `mixed` readiness rating.
What sets Dolma apart is transparency. The Allen Institute for AI (Allen AI) published the full toolkit used to build it, including the filtering, deduplication, and mixing steps, so you can see and reproduce how every part was assembled. It was created as the training data for the OLMo family of fully open models.
It is hosted on HuggingFace and downloadable in shards. Note the licensing is split: ODC-By 1.0 alongside the AI2 ImpACT License, so check the terms for the subset you plan to use.
Related sources
C4 (Colossal Clean Crawled Corpus)
A 750 GB English corpus derived from Common Crawl using heuristic cleaning to keep natural language and drop gibberish, boilerplate, and placeholder text. Built to train Google's T5 and later used for MPT-7B and others.
Common Crawl
A nonprofit that crawls the web and freely provides its archives and datasets. Petabytes of raw web data from billions of pages, with a new snapshot each month. Used in the training of GPT-3, LLaMA, T5, and many other large language models.
DCLM-Baseline (DataComp-LM)
A filtered English web dataset from the DataComp-LM benchmark project, produced by running model-based quality filtering over Common Crawl. Built to show which data-curation choices most improve language model training.
FineWeb / FineWeb-Edu
A 15T token English web corpus distilled from Common Crawl by HuggingFace, filtered and deduplicated for language model training. FineWeb-Edu is a subset filtered for educational content. One of the higher-quality open web datasets for pretraining and broad-coverage RAG.