MADLAD-400

MADLAD-400 (Multilingual And Document-level Large Audited Dataset) is a web-text corpus of roughly 3 trillion tokens spanning 419 languages, built by Google DeepMind and Google Research from Common Crawl. Its defining trait sits in the name: rather than trusting automated filters alone, the team manually audited samples in each language, so many low-resource languages that other web corpora mislabel or skip entirely turn up here with a known level of quality.

It comes in two flavours, both hosted on Hugging Face as gzipped JSON Lines, one JSON document per line. The noisy version applies only language identification, giving you more volume but more junk; the cleaned version layers on heavier filtering for higher quality. You can stream it through the datasets library instead of downloading the lot, which matters at this scale, and pull just the languages you want by their codes. Choose the noisy set when you want maximum data and plan to filter yourself, or the cleaned set when you want text closer to ready.

It is at its best for multilingual and especially low-resource work: pretraining or continued pretraining of multilingual models, assembling a knowledge base in languages that English-centric corpora ignore, or building the retrieval corpus for a non-English or cross-lingual RAG system. If your users do not work in English, this is one of the few places to find broad coverage of their languages at scale.

The caveats are those of any Common Crawl derivative. Even the cleaned version is web text, so expect boilerplate, duplication, and uneven quality, and the smaller a language's slice, the more variable it tends to be, so per-language filtering still pays off. The licence is ODC-By 1.0, which permits commercial use provided you attribute; as a database drawn from the open web, the usual caution about the underlying pages' own rights applies when you redistribute content.

Among its neighbours, monolingual English corpora like FineWeb or C4 are cleaner if you only need English, while multilingual sets such as CulturaX, OSCAR, and GlotCC cover overlapping ground. MADLAD-400 stands out for the sheer breadth of languages and the human auditing behind them.