Dolma
Dolma (the name spells out "Data for Open Language Models' Appetite") is Allen AI's bid to build a serious pretraining corpus entirely in the open. It stitches six different kinds of source into one documented whole: web text from Common Crawl, scientific papers from Semantic Scholar, code from GitHub, public-domain books, Reddit conversations, and Wikipedia. Together they come to more than 4 billion documents and roughly 3T tokens, and because the texture of the text changes so much from one subset to the next, we rate its readiness as `mixed`.
What sets Dolma apart is transparency. The Allen Institute for AI (Allen AI) published not only the data but the full toolkit that produced it, covering the tagging, filtering, deduplication, and mixing at every step. You can read exactly how each subset was cleaned, and if you want you can rerun the pipeline or point the same tools at your own raw data. It was assembled to train the OLMo family of fully open models, so the entire chain from raw crawl to finished model is open to inspection, which is rare.
It is hosted on HuggingFace and shipped as `JSONL` shards (one JSON document per line), so you can pull just the subsets you need rather than the whole corpus and stream them into a pipeline. Because it is already partitioned by source, you can be deliberate: take the books and papers for a factual knowledge base, or the web slice when you want broad coverage.
As a RAG or pretraining input it is a strong choice for a broad, mixed-domain corpus, and an excellent study case if you want to see how a large dataset is actually built. The catch is licensing. Dolma sits under a split of ODC-By 1.0 and the AI2 ImpACT License, the latter adding use-based conditions and attribution requirements, and individual subsets inherit their own sources' terms. We mark commercial use as unclear, so check the conditions for the specific subset you plan to ship rather than treating the corpus as uniformly open.
Compare it with The Pile and RedPajama, which take a similar multi-source approach, and with FineWeb when you want a purely web-based, more heavily filtered corpus.
Related sources
C4 (Colossal Clean Crawled Corpus)
A 750 GB English corpus derived from Common Crawl using heuristic cleaning to keep natural language and drop gibberish, boilerplate, and placeholder text. Built to train Google's T5 and later used for MPT-7B and others.
Common Corpus
An open, multilingual pretraining corpus of roughly 2.27 trillion tokens, built only from public-domain and permissively licensed text with documented provenance for every document. Maintained by PleIAs.
Common Crawl
A nonprofit that crawls the web and freely provides its archives and datasets. Petabytes of raw web data from billions of pages, with a new snapshot each month. Used in the training of GPT-3, LLaMA, T5, and many other large language models.
Common Pile v0.1
An 8 TB corpus of public domain and openly licensed text drawn from 30 sources, assembled by EleutherAI as a licence-transparent alternative to web-scraped pretraining data. Used to train the Comma family of language models.