GlotCC
GlotCC is aimed squarely at the long tail of world languages. Where most web-derived corpora concentrate on a handful of high-resource languages, GlotCC extends coverage to minority and low-resource ones, making it useful if you are building retrieval or generation for communities that are usually underserved by mainstream datasets.
The project releases both the corpus and the pipeline that produced it, so you can reproduce or extend the cleaning process yourself. The pipeline is licensed under CC0 (effectively public domain, no rights reserved), but the text itself comes from Common Crawl and remains subject to the original sites' terms, so the content licence differs from the pipeline licence and you should treat the underlying data accordingly.
Related sources
AI4Bharat (IndicCorp)
A collection of corpora, models, and benchmarks for Indian languages, produced by IIT Madras. It covers major Indic languages with monolingual corpora, parallel translation data, and evaluation sets.
CC-100
A high-quality multilingual corpus derived from Common Crawl. It splits every document into separate paragraphs, so it is effectively a paragraph-level rather than document-level corpus, which matters for your chunking strategy.
CulturaX
A multilingual text dataset of 6.3 T tokens across 167 languages, built for training large language models. It combines cleaned versions of mC4 and OSCAR, two large web corpora, after a multi-stage filtering pipeline.
HPLT (High Performance Language Technologies)
An EU-funded project building open language resources from web crawls and the Internet Archive. Its MonoHPLT dataset contains monolingual text in 75 languages, released as openly as licensing allows.