DCLM-Baseline (DataComp-LM)
DataComp-LM (DCLM) is a benchmark where the model architecture stays fixed and the training data is the variable, so teams compete on curation rather than modelling. DCLM-Baseline is the reference dataset that came out of it: a Common Crawl slice cleaned with a model-based quality filter, meaning a small classifier scores each document and low-scoring pages are dropped.
That approach turned out to beat many hand-tuned heuristic pipelines on downstream benchmarks, which is the point the project set out to demonstrate. The dataset is released on HuggingFace as Parquet and streamable with the `datasets` library.
It is a good pick when you want high-quality English web text with a clear, reproducible curation method behind it, and a useful reference if you are designing your own filtering pipeline.
Related sources
C4 (Colossal Clean Crawled Corpus)
A 750 GB English corpus derived from Common Crawl using heuristic cleaning to keep natural language and drop gibberish, boilerplate, and placeholder text. Built to train Google's T5 and later used for MPT-7B and others.
Common Crawl
A nonprofit that crawls the web and freely provides its archives and datasets. Petabytes of raw web data from billions of pages, with a new snapshot each month. Used in the training of GPT-3, LLaMA, T5, and many other large language models.
Dolma
A 3T token open corpus from Allen AI (its name stands for "Data for Open Language Models' Appetite") combining web text, scientific papers, code, public-domain books, Reddit posts, and Wikipedia. Built to train the OLMo models.
FineWeb / FineWeb-Edu
A 15T token English web corpus distilled from Common Crawl by HuggingFace, filtered and deduplicated for language model training. FineWeb-Edu is a subset filtered for educational content. One of the higher-quality open web datasets for pretraining and broad-coverage RAG.