Masakhane
Masakhane is a community-driven research network focused on African languages, many of which have little or no presence in mainstream datasets. Its contributors, spread across the continent and beyond, produce datasets, evaluation benchmarks, and models covering dozens of languages, with a strong emphasis on involving native speakers in the work.
For RAG builders targeting African audiences, Masakhane is often the best or only source of quality data for a given language. Licensing varies across its many releases but is mostly CC BY (attribution required) or Apache 2.0, both of which allow commercial use. Because the terms differ from one release to the next, check the licence on the specific dataset you plan to use.
Related sources
AI4Bharat (IndicCorp)
A collection of corpora, models, and benchmarks for Indian languages, produced by IIT Madras. It covers major Indic languages with monolingual corpora, parallel translation data, and evaluation sets.
CC-100
A high-quality multilingual corpus derived from Common Crawl. It splits every document into separate paragraphs, so it is effectively a paragraph-level rather than document-level corpus, which matters for your chunking strategy.
CulturaX
A multilingual text dataset of 6.3 T tokens across 167 languages, built for training large language models. It combines cleaned versions of mC4 and OSCAR, two large web corpora, after a multi-stage filtering pipeline.
GlotCC
An open, broad-coverage corpus and processing pipeline built from Common Crawl, targeting minority and low-resource languages that larger corpora tend to miss. It fills gaps left by datasets focused on higher-resource languages.