ROOTS
ROOTS was assembled by BigScience, the open research collaboration behind the BLOOM language model, to serve as that model's training corpus. It spans 46 natural languages and 13 programming languages, drawing on sources chosen by the community for quality and provenance, combined with the OSCAR web corpus.
The dataset is released under the BigScience RAIL licence, a responsible-AI licence that permits research and many uses but places restrictions on certain applications, and the terms vary by subset. Because of those use restrictions, treat commercial use as restricted and read the licence for the specific subset you plan to use rather than assuming blanket permission.
Related sources
AI4Bharat (IndicCorp)
A collection of corpora, models, and benchmarks for Indian languages, produced by IIT Madras. It covers major Indic languages with monolingual corpora, parallel translation data, and evaluation sets.
CC-100
A high-quality multilingual corpus derived from Common Crawl. It splits every document into separate paragraphs, so it is effectively a paragraph-level rather than document-level corpus, which matters for your chunking strategy.
CulturaX
A multilingual text dataset of 6.3 T tokens across 167 languages, built for training large language models. It combines cleaned versions of mC4 and OSCAR, two large web corpora, after a multi-stage filtering pipeline.
GlotCC
An open, broad-coverage corpus and processing pipeline built from Common Crawl, targeting minority and low-resource languages that larger corpora tend to miss. It fills gaps left by datasets focused on higher-resource languages.