Large-scale text corpora covering languages beyond English, including low-resource and regional languages. Essential for building RAG systems for non-English or multilingual audiences.
11sources
Multilingual and regional corpora are large text collections that cover languages beyond English, including low-resource languages, meaning ones with far less digitised text and fewer ready-made tools than English or Mandarin. If your RAG system, which retrieves passages to ground a language model’s answers, serves a non-English or mixed-language audience, an English-first corpus will quietly let those users down. Building on data in their own languages is what closes that gap.
The main choice is breadth against depth. Sets like MADLAD-400 or CulturaX span hundreds of languages, which is ideal for wide coverage, but the amount and cleanliness of text per language varies enormously. A corpus focused on one region or family, such as AI4Bharat for Indian languages or Masakhane for African ones, usually offers deeper, better-curated coverage of the languages it does include. Decide whether you need many languages shallowly or a few languages well.
Quality is where care pays off. For low-resource languages, the same web-scale filtering that works for English lets more noise through, so mislabelled languages, machine-translated text, and encoding or script problems are all more common. Check how the corpus was deduplicated and language-identified before trusting it. Licensing needs a second look too, particularly for corpora drawn from subtitles or scraped web pages, where the compilation may be openly licensed while the underlying content is not.
The corpora below run from broad hundred-language collections to focused regional projects built by and for specific language communities.
A collection of corpora, models, and benchmarks for Indian languages, produced by IIT Madras. It covers major Indic languages with monolingual corpora, parallel translation data, and evaluation sets.
A high-quality multilingual corpus derived from Common Crawl. It splits every document into separate paragraphs, so it is effectively a paragraph-level rather than document-level corpus, which matters for your chunking strategy.
A multilingual text dataset of 6.3 T tokens across 167 languages, built for training large language models. It combines cleaned versions of mC4 and OSCAR, two large web corpora, after a multi-stage filtering pipeline.
A high-quality multilingual pretraining corpus of roughly the top 10% of FineWeb2 documents in each of 20 languages, selected by a model-based quality classifier. Built by EPFL and released on Hugging Face.
An open, broad-coverage corpus and processing pipeline built from Common Crawl, targeting minority and low-resource languages that larger corpora tend to miss. It fills gaps left by datasets focused on higher-resource languages.
An EU-funded project building open language resources from web crawls and the Internet Archive. Its MonoHPLT dataset contains monolingual text in 75 languages, released as openly as licensing allows.
A manually audited, general-domain monolingual dataset of 3 T tokens spanning 419 languages, built from Common Crawl by Google DeepMind and Google Research. It comes in a noisy version and a more heavily filtered clean version.
A grassroots research community building natural language processing resources for African languages. It produces datasets, benchmarks, and models across dozens of African languages that are otherwise poorly served.
NVIDIA's 2025 post-training set of prompts and synthetic responses for supervised fine-tuning and reinforcement learning, covering mathematics, code, STEM, reasoning, and instruction following, with the instruction data expanded into five additional languages.
A large corpus of translated movie and TV subtitles. The latest version covers 60 languages with 2.6 billion sentences in total. It is valuable for conversational and colloquial language, and for parallel translation data.
The Responsible Open-Science Open-Collaboration Text Sources corpus, a 1.6 TB dataset spanning 46 natural languages and 13 programming languages, released by BigScience. Built from community-selected sources plus OSCAR, it trained BLOOM.