Common Corpus

Common Corpus is one of the largest openly licensed text collections assembled for training language models, containing 2,267,302,720,836 tokens. It is organised into five broad streams: OpenCulture (books, newspapers and cultural heritage material), OpenGovernment (administrative and legal documents), OpenSource (code), OpenScience (scholarly articles) and OpenWeb. More than half of the text predates the 21st century, reflecting the heavy weighting towards public-domain cultural heritage. It is genuinely multilingual: eight languages carry more than 10 billion tokens (including German, Spanish, Italian, Polish, Greek and Latin) and 33 languages carry more than 1 billion.

The dataset is distributed as Parquet files on Hugging Face, so you can stream it with the datasets library or pull the shards directly for bulk processing. Every record keeps documented contextual metadata and source information, which is unusual at this scale and useful when you need to trace where a passage came from.

For RAG and AI work, Common Corpus is best treated as a broad-coverage pretraining or domain-mixing corpus, and as a provenance-clean base you can filter down into a retrieval index for history, law, government or public-domain literature. Because it deliberately excludes copyrighted material, it is a sensible starting point when licensing risk matters.

Watch-outs: the licence is not a single blanket grant. Individual documents carry their own terms (CC0, various Creative Commons variants, MIT, the French Licence ouverte and others), so the compilation licence differs from the item-level licences, and any share-alike or attribution obligations must be checked per source before redistribution. Toxicity and PII filtering were applied but are imperfect. If you need alternatives already common in a directory, compare it with FineWeb for filtered web text and Common Crawl for raw web archives.