GlotCC
GlotCC is a broad-coverage web corpus built for the long tail of the world's languages. While most web-derived datasets crowd around a small set of high-resource languages, GlotCC pushes clean coverage out to more than a thousand languages, including many minority and low-resource ones that larger corpora barely register. It is derived from Common Crawl and cleaned with a language-aware pipeline that puts heavy weight on accurate language identification, so text is far less likely to be filed under the wrong language.
The corpus is published on HuggingFace and arranged by language, and the project releases its processing pipeline alongside the data, so you can reproduce the cleaning or extend it to newer crawl snapshots yourself. Records arrive as JSON carrying the text and its metadata, and you can stream or download one language at a time rather than pulling the entire release, which matters given the sheer number of languages. Working language by language keeps the memory footprint sensible.
This is the source to reach for when your RAG or model work needs to serve communities the mainstream datasets overlook: building retrieval or generation in a minority language, widening the language coverage of a multilingual system, or researching low-resource natural language processing. If a language is absent from FineWeb2 or the major corpora, there is a fair chance GlotCC holds at least some of it.
Stay realistic about volume and quality at the far tail. For the rarest languages there may be only a modest amount of text, and even careful filtering leaves some noise and the occasional misidentified page, so sample and spot-check before you commit to a given language. On licensing, the pipeline is released under CC0 (effectively public domain, no rights reserved), but the text comes from Common Crawl and stays subject to the original sites' terms, so the content licence differs from the pipeline licence and you should handle the underlying data with the same caution you would give raw crawl.
Run it alongside FineWeb2 and CC-100 for multilingual coverage, with GlotCC plugging the low-resource gaps the others leave open.
Related sources
AI4Bharat (IndicCorp)
A collection of corpora, models, and benchmarks for Indian languages, produced by IIT Madras. It covers major Indic languages with monolingual corpora, parallel translation data, and evaluation sets.
CC-100
A high-quality multilingual corpus derived from Common Crawl. It splits every document into separate paragraphs, so it is effectively a paragraph-level rather than document-level corpus, which matters for your chunking strategy.
CulturaX
A multilingual text dataset of 6.3 T tokens across 167 languages, built for training large language models. It combines cleaned versions of mC4 and OSCAR, two large web corpora, after a multi-stage filtering pipeline.
FineWeb2-HQ
A high-quality multilingual pretraining corpus of roughly the top 10% of FineWeb2 documents in each of 20 languages, selected by a model-based quality classifier. Built by EPFL and released on Hugging Face.