AI4Bharat (IndicCorp)
AI4Bharat is a research effort based at IIT Madras that builds open resources for Indian languages, which are underserved by most mainstream datasets. Its IndicCorp corpora provide large amounts of monolingual text across major Indic languages, alongside parallel data (sentence pairs aligned across languages) and benchmark sets for measuring model quality.
If you are building RAG or language tools for Indian audiences, this is a strong starting point because it bundles training data, evaluation sets, and models from one group with consistent methodology. Licensing varies by dataset but is mostly permissive, so check the specific corpus or benchmark you want, since commercial terms are not guaranteed to be the same across the collection.
Related sources
CC-100
A high-quality multilingual corpus derived from Common Crawl. It splits every document into separate paragraphs, so it is effectively a paragraph-level rather than document-level corpus, which matters for your chunking strategy.
CulturaX
A multilingual text dataset of 6.3 T tokens across 167 languages, built for training large language models. It combines cleaned versions of mC4 and OSCAR, two large web corpora, after a multi-stage filtering pipeline.
GlotCC
An open, broad-coverage corpus and processing pipeline built from Common Crawl, targeting minority and low-resource languages that larger corpora tend to miss. It fills gaps left by datasets focused on higher-resource languages.
HPLT (High Performance Language Technologies)
An EU-funded project building open language resources from web crawls and the Internet Archive. Its MonoHPLT dataset contains monolingual text in 75 languages, released as openly as licensing allows.