Common Pile v0.1

The Common Pile v0.1 is a large text corpus built entirely from public domain and openly licensed material, released in 2025 by EleutherAI with a wide group of academic and industry collaborators. It gathers roughly 8 TB of text across 30 sources spanning open-source code, legal and government documents, encyclopaedic text such as Wikipedia, academic papers, public domain books, educational resources, and audio transcripts. It is positioned as a licence-transparent successor to earlier web-scraped corpora, with provenance and licensing documented rather than assumed.

Access is straightforward. The raw sources and a filtered, deduplicated pretraining version are published as separate collections on Hugging Face under the common-pile organisation, stored as Parquet (converted from the original JSON), and the tooling used to build and filter the corpus is on GitHub. You can stream or bulk download individual sources or work with the filtered release intended for model training. EleutherAI also released the Comma v0.1-1T and Comma v0.1-2T models trained on this data.

For RAG and AI work it is best suited to pretraining, continued pretraining, and fine-tuning where clear licensing matters, and as a broad knowledge base you can chunk and embed without the provenance doubts attached to general web crawls. The main watch-out is the licence itself: the compilation aggregates many different open licences, including permissive licences, public domain, and copyleft share-alike terms such as CC BY-SA, so obligations vary by source and you should check the licence recorded on each item before redistribution or commercial use. If you need similarly scaled but differently governed corpora, Common Crawl and FineWeb are the obvious neighbours already in this directory, though neither offers the same per-item licence clarity.