Multilingual RAG: where to get non-English data

Most RAG (Retrieval-Augmented Generation) tutorials assume your knowledge base is in English. Real systems rarely are. If you are building for users in India, West Africa, continental Europe, or anywhere else, you need retrieval corpora in the languages your users actually read and speak. This post is a practical map of open non-English data: large multilingual web text, regional and community efforts that fill the gaps, speech for voice and multimodal work, and one dataset built specifically for measuring multilingual retrieval quality. Everything here has its own page in our directory, which carries the outbound links, licence notes, and download pointers.

Before the list, one honest caveat that shapes every choice below: language coverage is deeply uneven. English, and to a lesser degree a handful of high-resource European and East Asian languages, dominate almost every corpus. Low-resource languages have far less data, and what exists tends to be noisier: more boilerplate, more misclassified text, more machine translation masquerading as native writing. Quality and licence also vary within a single dataset, language by language. Treat any headline “covers N languages” claim as a starting point for your own inspection, not a guarantee.

Large multilingual web text

The broadest coverage comes from cleaned web crawls. These are your default foundation if you need a lot of text across many languages.

CulturaX is a large multilingual corpus assembled from earlier crawl-based datasets, deduplicated and filtered, spanning a wide set of languages. It is a common starting point when you want scale and reasonable cleaning in one place. FineWeb2 takes the FineWeb pipeline, which was originally English-focused, and extends the filtering and deduplication approach to many more languages, with per-language processing rather than a single global pass. Both are good candidates when you want the cleaning work already done.

If you want to compare against, or reprocess from, the more established building blocks, several sit underneath these newer corpora:

• CC-100 reconstructs monolingual data in many languages from web crawl snapshots. It is widely used and easy to reason about, though the cleaning is lighter than the newer corpora above.
• OSCAR is another crawl-derived multilingual corpus with language identification and filtering, distributed in versioned releases so you can pin a specific one.
• MADLAD-400 aims for very broad language coverage, reaching into the long tail of lower-resource languages that the bigger corpora often drop, with document-level and per-language quality signals.
• GlotCC focuses specifically on wide language coverage from web data, again reaching languages that mainstream pipelines under-serve.

For the long-tail options in particular, sample the specific languages you care about before committing. A dataset that nominally covers a language may contain only a few thousand usable documents for it, and a noticeable fraction may be mistranslated or misidentified. It is also worth checking how each corpus handles scripts and encodings: right-to-left languages, non-Latin scripts, and languages with several writing systems in active use are all places where a generic pipeline can silently mangle text or split the same language across incompatible representations.

If you would rather skip the ingestion and embedding stage entirely, Cohere’s multilingual Wikipedia embeddings offer a ready-to-use option: Wikipedia in many languages, already chunked and turned into vectors. That saves real work, but it ties you to a specific embedding model, so it suits prototypes and Wikipedia-centred knowledge bases more than a bespoke corpus. If the trade-offs there are unfamiliar, our lesson on how to create embeddings walks through what choosing a model commits you to. You can browse the full set of options on the multilingual category page.

Regional and community-driven data

Big global crawls are shaped by what is abundant on the open web, which means they systematically under-represent languages that are widely spoken but under-published online. This is where regional and community projects matter, because they deliberately gather, clean, and sometimes create data for languages the large crawls miss or handle poorly.

AI4Bharat concentrates on Indic languages, the languages of the Indian subcontinent, producing corpora, benchmarks, and models built with attention to scripts and language varieties that a generic pipeline tends to flatten. Masakhane is a community effort focused on African languages, with datasets and evaluation resources built by and with speakers of those languages rather than scraped at a distance.

Two things make these worth your attention even when a global corpus nominally covers the same language:

• The data is often cleaner for its target languages, because people who read the language did the filtering and know what boilerplate, spam, and misclassification look like.
• Coverage extends to varieties, dialects, and scripts that automated language identification frequently mislabels or discards.

The trade-off is that licences and formats are less uniform than the big corpora. Some resources are permissively licensed, others carry research-only or attribution conditions, and a few mix several licences across sub-datasets. Check the terms per resource, not per project. We wrote about why this matters in The licensing trap, which is worth reading before you build anything you intend to ship.

Speech, for voice and multimodal RAG

If your system takes spoken input, answers by voice, or retrieves over transcripts and recordings, you need audio paired with text. Two open speech corpora anchor the speech-audio category.

Common Voice is a crowd-sourced, multilingual collection of read speech with transcripts, contributed and validated by volunteers across a growing set of languages. Because it is community-recorded, the language spread is uneven in the same way text corpora are: some languages have many hours, others only a little. People’s Speech is a large English-centred corpus of transcribed speech drawn from openly licensed sources, useful when you need volume of real-world audio rather than breadth of languages.

For multilingual voice RAG specifically, Common Voice is usually the more relevant starting point on language coverage, while People’s Speech gives you scale for English pipelines. As with text, sample the exact languages you need and check recording quality, accent spread, and transcript accuracy before you rely on any of it. Speech data that mixes languages or has inconsistent transcription conventions will quietly degrade retrieval further downstream.

Measuring multilingual retrieval

Building the corpus is only half the job. You still need to know whether retrieval actually works in each language, and per-language quality can differ enormously even with the same embedding model and index. This is where an evaluation set earns its place.

MIRACL is built for exactly this: a multilingual retrieval benchmark with queries and relevance judgements across many languages, so you can measure whether your retriever surfaces the right passages rather than assuming that English-level performance carries over. Running an evaluation like this early tends to expose weak languages before your users do, and it gives you a concrete target when you tune chunking, embeddings, or the retriever itself. It also helps you decide where to spend effort: if a handful of languages carry most of your traffic, you may reasonably invest in cleaner regional data for those and accept thinner coverage elsewhere, rather than chasing uniform quality across a long list of languages that few users query. If your retrieval scores look poor for a given language, our guide on how to improve your RAG covers the usual levers.

Putting it together

A workable approach for a multilingual build is to start broad, then patch the gaps. Use CulturaX or FineWeb2 for general coverage, reach for MADLAD-400 or GlotCC when you need lower-resource languages, and layer in AI4Bharat, Masakhane, or another regional source for the languages that matter most to your users and that the global crawls handle badly. Add speech from Common Voice if you are going multimodal, and hold out MIRACL, or your own labelled set, to check that retrieval genuinely works language by language.

Above all, inspect before you trust. The single biggest difference between a multilingual RAG system that works and one that quietly fails for half its users is whether someone actually looked at the data in each language. If you are still choosing a foundation, Choosing your first RAG dataset is a good companion to this roundup, and the wider directory lets you filter by access type and licence as you shortlist.