CulturaX
CulturaX gives you broad-coverage web text in 167 languages, ready for pretraining or for building non-English and multilingual retrieval systems. It merges two well-known web corpora, mC4 and OSCAR, then runs the combined data through a pipeline of language identification, URL-based filtering, metric-based cleaning, document refinement, and deduplication (removing duplicate and near-duplicate text so you are not storing the same passage twice).
The dataset is published on Hugging Face, so you can stream it or download it in parts rather than pulling all 6.3 T tokens at once. The licence is ODC-By, which permits commercial use with attribution, but the data still inherits the underlying mC4 and OSCAR terms, so check those if your use case is sensitive.
Related sources
AI4Bharat (IndicCorp)
A collection of corpora, models, and benchmarks for Indian languages, produced by IIT Madras. It covers major Indic languages with monolingual corpora, parallel translation data, and evaluation sets.
CC-100
A high-quality multilingual corpus derived from Common Crawl. It splits every document into separate paragraphs, so it is effectively a paragraph-level rather than document-level corpus, which matters for your chunking strategy.
GlotCC
An open, broad-coverage corpus and processing pipeline built from Common Crawl, targeting minority and low-resource languages that larger corpora tend to miss. It fills gaps left by datasets focused on higher-resource languages.
HPLT (High Performance Language Technologies)
An EU-funded project building open language resources from web crawls and the Internet Archive. Its MonoHPLT dataset contains monolingual text in 75 languages, released as openly as licensing allows.