CC-100
CC-100 is a multilingual corpus built from Common Crawl snapshots, widely used as pretraining data for multilingual language models. It covers a broad set of languages and is straightforward to download directly from the statmt.org mirror.
One quirk is worth knowing before you use it: the data is stored split into individual paragraphs rather than whole documents. That makes it a paragraph-level corpus by default. If your RAG pipeline relies on document structure or longer context windows, you may need to reassemble or re-chunk the text rather than treating each entry as a full document. The content is drawn from Common Crawl and remains subject to the original sites' terms.
Related sources
AI4Bharat (IndicCorp)
A collection of corpora, models, and benchmarks for Indian languages, produced by IIT Madras. It covers major Indic languages with monolingual corpora, parallel translation data, and evaluation sets.
CulturaX
A multilingual text dataset of 6.3 T tokens across 167 languages, built for training large language models. It combines cleaned versions of mC4 and OSCAR, two large web corpora, after a multi-stage filtering pipeline.
GlotCC
An open, broad-coverage corpus and processing pipeline built from Common Crawl, targeting minority and low-resource languages that larger corpora tend to miss. It fills gaps left by datasets focused on higher-resource languages.
HPLT (High Performance Language Technologies)
An EU-funded project building open language resources from web crawls and the Internet Archive. Its MonoHPLT dataset contains monolingual text in 75 languages, released as openly as licensing allows.