SlimPajama
SlimPajama takes RedPajama-V1 and tightens it up. Cerebras removed very short and low-content documents, then ran additional fuzzy deduplication (finding pages that are near-identical, not just exact matches) across the whole corpus. The result keeps the same mix of web text, books, papers, code, and Q&A but with far less repetition.
Because duplicate content is removed, every token you train or index on tends to add new information, which makes SlimPajama efficient for its size. It is published on HuggingFace as a 627B token dataset you can stream with the `datasets` library, and the cleaning pipeline is open so you can reproduce or adjust it.
It is a good default when you want the breadth of RedPajama without the storage and compute cost of the full V1 dump.
Related sources
C4 (Colossal Clean Crawled Corpus)
A 750 GB English corpus derived from Common Crawl using heuristic cleaning to keep natural language and drop gibberish, boilerplate, and placeholder text. Built to train Google's T5 and later used for MPT-7B and others.
Common Crawl
A nonprofit that crawls the web and freely provides its archives and datasets. Petabytes of raw web data from billions of pages, with a new snapshot each month. Used in the training of GPT-3, LLaMA, T5, and many other large language models.
DCLM-Baseline (DataComp-LM)
A filtered English web dataset from the DataComp-LM benchmark project, produced by running model-based quality filtering over Common Crawl. Built to show which data-curation choices most improve language model training.
Dolma
A 3T token open corpus from Allen AI (its name stands for "Data for Open Language Models' Appetite") combining web text, scientific papers, code, public-domain books, Reddit posts, and Wikipedia. Built to train the OLMo models.