SlimPajama
SlimPajama takes RedPajama-V1 and tightens it up. Cerebras dropped very short and low-content documents, then ran additional fuzzy deduplication (finding pages that are near-identical, not just exact byte-for-byte matches) across the whole corpus. The result keeps the same familiar mix of web text, books, academic papers, code, and Q&A drawn from Common Crawl, C4, GitHub, arXiv, Wikipedia, Stack Exchange, and books, but with far less repetition. The 1.2T tokens of RedPajama-V1 come down to a denser 627B.
Because so much duplicate content is gone, most tokens you train or index on add genuinely new information, which makes SlimPajama efficient for its size. It is published on Hugging Face as 627B tokens of `JSONL`, split into shards and tagged by source so you can filter to just the domains you care about. You can stream it with the `datasets` library rather than pulling the full download, and the cleaning pipeline is open, so you can reproduce it or tune the thresholds for your own data.
For RAG, SlimPajama works best as broad background knowledge: pretraining or continued pretraining of a base model, or a wide open-domain corpus where you want varied, multi-source text without curating the mix yourself. If you only need one domain, the source tags let you carve out, say, just the arXiv or Stack Exchange slice rather than indexing everything.
The caveats are the usual ones for a broad web-plus-books corpus: it is general-purpose English, so it will not give you deep specialist coverage or multilingual reach, and even after deduplication some noise and dated content remain. On licensing, Cerebras releases the dataset under Apache 2.0, which permits commercial use with no share-alike obligation, but the underlying documents keep their own terms, so treat redistribution of the raw text with care. Compared with the web-only RefinedWeb, FineWeb, or C4, SlimPajama's strength is that mixed diet: it is the leaner, deduplicated way to get RedPajama's breadth without the storage and compute cost of the full V1 dump.
Related sources
C4 (Colossal Clean Crawled Corpus)
A 750 GB English corpus derived from Common Crawl using heuristic cleaning to keep natural language and drop gibberish, boilerplate, and placeholder text. Built to train Google's T5 and later used for MPT-7B and others.
Common Corpus
An open, multilingual pretraining corpus of roughly 2.27 trillion tokens, built only from public-domain and permissively licensed text with documented provenance for every document. Maintained by PleIAs.
Common Crawl
A nonprofit that crawls the web and freely provides its archives and datasets. Petabytes of raw web data from billions of pages, with a new snapshot each month. Used in the training of GPT-3, LLaMA, T5, and many other large language models.
Common Pile v0.1
An 8 TB corpus of public domain and openly licensed text drawn from 30 sources, assembled by EleutherAI as a licence-transparent alternative to web-scraped pretraining data. Used to train the Comma family of language models.