CC-100

CC-100 is a multilingual text corpus built from Common Crawl, created to reproduce the training data behind the XLM-R multilingual model. It covers more than 100 languages, including many lower-resource ones that are hard to find at scale elsewhere, with the larger languages running to tens or hundreds of gigabytes of plain text and the smaller ones much slimmer. Each language comes as its own compressed plain-text file, so you can take only the languages you care about.

You download it directly from the statmt.org mirror, one file per language, and unpack the plain `.txt`. There is no API or streaming layer: you pull the files and process them yourself, which is refreshingly simple. Because it is already extracted text with no HTML or markup to strip, you can move fairly quickly to cleaning, deduplication, and chunking.

One quirk matters for RAG, and it is easy to miss. The data is stored split into individual paragraphs rather than whole documents, so although it reads like continuous text, it is effectively a paragraph-level corpus. If your pipeline depends on full-document structure or long context windows, you will need to reassemble or re-chunk rather than treat each block as a complete document.

CC-100 is best for multilingual and lower-resource language work: pretraining or fine-tuning multilingual embedding models, or building a retrieval corpus in a language where clean data is scarce. Because it is derived from Common Crawl, the content stays subject to the original sites' terms, so treat commercial reuse and redistribution with the same care you would the underlying crawl. If you need document-level structure or heavier filtering already done, a corpus like OSCAR, CC-Net, or the multilingual slices of FineWeb may fit better; CC-100's strength is breadth of language coverage in a form that is trivial to download.