Dolma 3

Dolma 3 is the Allen Institute for AI's (Ai2) third-generation open pretraining corpus, built to train the OLMo 3 family of fully open models. It follows the original Dolma, which we also list, and reflects what Ai2 learned across two earlier generations about assembling, filtering and documenting a large corpus in the open. The headline artefact is the Dolma 3 Pool, a dataset of over 9 trillion tokens (the source tables total roughly 9.3T tokens across about 10 billion documents, and 17.5 TB of files). It draws on Common Crawl web pages, academic PDFs processed with olmOCR, code from StackEdu, arXiv papers with LaTeX, FineMath maths web pages, and Wikipedia and Wikibooks. From this pool Ai2 constructs the Dolma 3 Mix, a roughly 5.9T-token (about 6T) pretraining mixture that weights code and mathematics more heavily than earlier Dolma releases and applies much stronger decontamination through extensive deduplication, quality filtering and careful control of the data mix.

You access it on the Hugging Face Hub. The pool ships as compressed JSONL shards (.jsonl.zst) with a Parquet viewer, and loads through the datasets library, with streaming and per-file filtering supported for selective downloads. Note that the pool hosts only the Common Crawl and olmOCR PDF documents directly; the other sources are fetched from their own linked repositories, and the exact training mixture lives in a separate mix repository.

For RAG and AI work it is best treated as a pretraining and research corpus rather than a retrieval-ready knowledge base. It is broad, mixed-domain and partitioned by source, so you can take the slices you need. The licence on Ai2's curation is ODC-BY, which requires attribution but imposes no share-alike condition, and the card notes the data is intended for research and educational use. Because the underlying web, code and book content carries its own upstream terms, we flag that content licences differ and mark commercial use as restricted: check the terms for the specific subset you plan to ship. Compare it with Dolma, FineWeb, The Pile and RedPajama.