CulturaX

CulturaX gives you broad-coverage web text in 167 languages, totalling 6.3 trillion tokens, and it is built specifically for training large language models. It merges two well-known web corpora, mC4 and OSCAR, then runs the combined data through a multi-stage pipeline: language identification, URL-based filtering to drop junk and adult domains, metric-based cleaning that scores documents on signals like perplexity, document-level refinement, and fuzzy deduplication that removes duplicate and near-duplicate text so you are not storing the same passage twice. The result is meaningfully cleaner than either parent corpus on its own.

The dataset is published on Hugging Face and stored as Parquet, split by language. That layout matters in practice: you can stream it with the `datasets` library, or download only the languages and shards you need rather than pulling all 6.3 trillion tokens, which run to many terabytes. If you only want, say, Welsh or Vietnamese, you fetch just those files.

For RAG (Retrieval-Augmented Generation), CulturaX is most useful as raw material for the languages the English-first datasets neglect: pretraining or continued-pretraining a multilingual base model, training or fine-tuning a multilingual embedding model, or seeding a non-English knowledge base. It is broad web text, not curated reference content, so for a factual retrieval corpus you will still want to filter down to the domains and quality band you care about; coverage and cleanliness also thin out for the lower-resource languages in the tail.

On licensing, CulturaX is released under ODC-By, which permits commercial use with attribution, but the data inherits the underlying mC4 and OSCAR terms too, so review those if your use case is sensitive. If your work is English-only, a focused corpus like FineWeb will serve you better; reach for CulturaX when breadth across many languages is the whole point.