RedPajama
RedPajama-V1 recreated the recipe described in the original LLaMA paper, mixing Common Crawl web text with Wikipedia, public-domain and other books, arXiv papers, GitHub code, and Stack Exchange Q&A. It gave the open community a roughly 1.2T token dataset with a known composition, and became the base for several open models.
RedPajama-V2 is a different beast: a web-only corpus of more than 100T tokens drawn from 84 monthly Common Crawl snapshots across five languages. Instead of pre-filtering, it ships 46 precomputed quality signals per document (things like language scores, repetition rates, and heuristic flags), so you can apply your own filtering thresholds rather than accept someone else's.
The pipeline code is on GitHub under Apache 2.0, while the underlying data keeps the licences of its original sources. The data itself is hosted on HuggingFace. V2 is best treated as raw material you filter down, not a ready-to-train corpus.
Related sources
C4 (Colossal Clean Crawled Corpus)
A 750 GB English corpus derived from Common Crawl using heuristic cleaning to keep natural language and drop gibberish, boilerplate, and placeholder text. Built to train Google's T5 and later used for MPT-7B and others.
Common Crawl
A nonprofit that crawls the web and freely provides its archives and datasets. Petabytes of raw web data from billions of pages, with a new snapshot each month. Used in the training of GPT-3, LLaMA, T5, and many other large language models.
DCLM-Baseline (DataComp-LM)
A filtered English web dataset from the DataComp-LM benchmark project, produced by running model-based quality filtering over Common Crawl. Built to show which data-curation choices most improve language model training.
Dolma
A 3T token open corpus from Allen AI (its name stands for "Data for Open Language Models' Appetite") combining web text, scientific papers, code, public-domain books, Reddit posts, and Wikipedia. Built to train the OLMo models.