FineWeb2-HQ

FineWeb2-HQ is a quality-filtered slice of FineWeb2, the large multilingual web corpus derived from Common Crawl. The EPFL Machine Learning and Optimization Laboratory trained a classifier on XLM-RoBERTa embeddings to score documents for structured, knowledge-rich content, then kept roughly the top 10% in each of 20 languages that span several writing systems. The result is about 380 million documents totalling around 6.3 TB, distributed as Parquet.

Access is straightforward: the data lives on the Hugging Face Hub, so you can stream it, load it with the datasets library, or pull the Parquet files in bulk. A companion GitHub repository publishes the selection code from the accompanying paper, which was accepted to the datasets track at NeurIPS 2025.

For RAG and AI work its main strength is pretraining and continued pretraining where multilingual coverage and document quality both matter. The EPFL evaluations report that models reach comparable quality with far fewer tokens than the unfiltered base, which makes it attractive for compute-constrained training runs. It can also seed a broad multilingual retrieval corpus, though as a raw web dataset you will want your own chunking, deduplication, and embedding step before indexing.

Watch-outs are the usual ones for web-scale crawl data. The content is unlabelled web text, so expect noise, uneven per-language volume, and possible personal data, and the maintainers advise additional PII and sensitive-content filtering before training. The dataset compilation is ODC-By 1.0, but use remains subject to Common Crawl's terms and to the terms of the underlying sites, so the compilation licence and the content licence are not the same thing. If you need the full unfiltered pool, the parent FineWeb2 is the obvious alternative; for English-only work, the original FineWeb serves a similar role.