DCLM-Baseline (DataComp-LM)

DataComp-LM (DCLM) turns the usual benchmark on its head. Instead of competing on model design, everyone holds the architecture and training budget fixed and competes on the training data itself, so the whole contest is about curation. DCLM-Baseline is the reference corpus that fell out of that effort: a very large slice of Common Crawl (the open web-crawl archive most base models are trained on) put through a model-based quality filter, where a small, fast text classifier scores each document and the weak ones are dropped. What remains runs to trillions of tokens of English web text.

The filtering method is what makes it notable. Rather than writing rules by hand to strip boilerplate and spam, the team trained a lightweight classifier to spot the kind of clear, instructive prose that actually helps a model learn, then ran it across the crawl at scale. That approach beat many carefully tuned heuristic pipelines on downstream tests, which is precisely the finding the benchmark set out to demonstrate.

The corpus is hosted on HuggingFace as `Parquet` files and plays nicely with the `datasets` library, so you can stream it shard by shard rather than downloading the whole thing first, which matters at this size. For a RAG knowledge base it is a strong pick when you want high-quality English web text with a transparent, reproducible curation story behind it, and it doubles as a working template if you are designing your own quality filter.

A few honest caveats. It is English-only, so it does nothing for a multilingual system. It is still web-derived, so some noise, duplication, and dated content slips through, and any classifier-based curation quietly leans towards whatever the classifier was trained to like. On rights, DCLM-Baseline is released under ODC-By 1.0, which permits commercial use but requires attribution, so credit the dataset in anything you ship.

Compare it with FineWeb and its FineWeb-Edu variant, which chase similar quality goals by different routes, and with the raw Common Crawl it is distilled from when you would rather control the filtering yourself.