RedPajama
RedPajama, from Together, arrives in two very different versions. V1 recreated the training recipe set out in the original LLaMA paper, mixing Common Crawl web text with Wikipedia, books, arXiv papers, GitHub code, and Stack Exchange question-and-answer threads into roughly 1.2 trillion tokens with a documented, known composition. It handed the open community a transparent alternative to closed training sets and seeded several early open models.
V2 is a different beast: a web-only corpus of more than 100 trillion tokens, drawn from 84 monthly Common Crawl snapshots across five languages (English, French, Spanish, German, and Italian). Rather than pre-filter, it ships 46 precomputed quality signals for each document, including language- identification scores, repetition and perplexity measures, and heuristic flags, so you set your own thresholds instead of inheriting someone else's editorial choices.
Both versions live on the Hugging Face Hub as JSONL and Parquet, with the pipeline code on GitHub. The datasets library is the natural way in, streaming rather than downloading given the scale. For retrieval with V2, the smart move is to treat the quality signals as your filter: load the metadata, pick thresholds that suit your domain (stricter for a polished knowledge base, looser for breadth), keep only the documents that pass, then deduplicate before you chunk and embed. V1 is easier if you want a ready, balanced mix without building a filter of your own.
The main caveat is that V2 is raw material, not a finished corpus: skip the filtering and you will index spam, boilerplate, and duplicates. On licensing, the pipeline code is Apache 2.0, but the underlying text keeps the licences of its original web sources, so commercial reuse of the content is not automatically clear and needs the same care as Common Crawl itself.
If you would rather not filter, SlimPajama is a cleaned, deduplicated 627-billion-token cut of V1, while Dolma and FineWeb are other filtered Common Crawl derivatives. Reach for RedPajama V2 when you specifically want full control over quality filtering at web scale.
Related sources
C4 (Colossal Clean Crawled Corpus)
A 750 GB English corpus derived from Common Crawl using heuristic cleaning to keep natural language and drop gibberish, boilerplate, and placeholder text. Built to train Google's T5 and later used for MPT-7B and others.
Common Corpus
An open, multilingual pretraining corpus of roughly 2.27 trillion tokens, built only from public-domain and permissively licensed text with documented provenance for every document. Maintained by PleIAs.
Common Crawl
A nonprofit that crawls the web and freely provides its archives and datasets. Petabytes of raw web data from billions of pages, with a new snapshot each month. Used in the training of GPT-3, LLaMA, T5, and many other large language models.
Common Pile v0.1
An 8 TB corpus of public domain and openly licensed text drawn from 30 sources, assembled by EleutherAI as a licence-transparent alternative to web-scraped pretraining data. Used to train the Comma family of language models.