MADLAD-400
MADLAD-400 (Multilingual And Document-level Large Audited Dataset) covers 419 languages, including many low-resource ones that most web corpora skip. It was built from Common Crawl and then manually audited, meaning people reviewed samples per language to catch mislabelled or low-quality text rather than relying on automated filters alone.
Two versions are available. The noisy version applies only language identification, so you get more data but with more junk. The cleaned version adds more extensive filtering for higher quality. Pick the noisy set if you want volume and plan to filter yourself, or the cleaned set if you want text that is closer to ready. Both are released under ODC-By 1.0, which permits commercial use with attribution.
Related sources
AI4Bharat (IndicCorp)
A collection of corpora, models, and benchmarks for Indian languages, produced by IIT Madras. It covers major Indic languages with monolingual corpora, parallel translation data, and evaluation sets.
CC-100
A high-quality multilingual corpus derived from Common Crawl. It splits every document into separate paragraphs, so it is effectively a paragraph-level rather than document-level corpus, which matters for your chunking strategy.
CulturaX
A multilingual text dataset of 6.3 T tokens across 167 languages, built for training large language models. It combines cleaned versions of mC4 and OSCAR, two large web corpora, after a multi-stage filtering pipeline.
GlotCC
An open, broad-coverage corpus and processing pipeline built from Common Crawl, targeting minority and low-resource languages that larger corpora tend to miss. It fills gaps left by datasets focused on higher-resource languages.