Common Crawl vs FineWeb vs C4 vs RedPajama: choosing a web corpus
The big open web corpora look alike from the outside. Here is how Common Crawl, FineWeb, C4, RedPajama and their cousins actually differ in filtering, size, cleanliness and licence, and how to choose between them.
The large open web corpora sound interchangeable when you read their headlines. Each one promises billions of pages of web text, each one has been used to train well known language models, and each one turns up in the same conversations about pretraining and retrieval. Look closer and they are not siblings but a family tree. One of them is the raw base layer, and the rest are progressively filtered and deduplicated derivatives of it. Understanding that lineage is the fastest way to pick the right one for your project, whether you are pretraining a model, fine tuning, or assembling a knowledge base for Retrieval-Augmented Generation (RAG, the pattern where a model retrieves documents at query time and generates an answer grounded in them).
This post walks the web corpora category from the raw source outward, and ends with a plain recommendation for each common situation. If you want the wider set of options, the full directory lists many more.
The base layer: Common Crawl
Almost every corpus below starts from the same place. Common Crawl is a non-profit that crawls the public web and releases the results for free, with a new snapshot roughly every month. Each snapshot is enormous, measured in hundreds of terabytes of compressed data, and it is genuinely raw. You get WARC files (the archival format that stores the full HTTP response, including headers and HTML), along with extracted plain text and metadata.
Raw is the important word. Common Crawl is not filtered for quality, not deduplicated, and not cleaned of boilerplate, navigation menus, cookie banners, or spam. That is a feature, not a flaw: it is the widest possible net, and every downstream corpus makes its own decisions about what to throw away. If you want maximal coverage and are prepared to build your own cleaning pipeline, you start here. If you want text that is ready to use with less work, you want one of the derivatives.
One more thing to hold in mind as you read on: every corpus below inherits its coverage and its biases from Common Crawl. The crawler reaches what is public and linkable, which skews towards popular, English heavy, and commercially indexed pages. Filtering can improve quality, but it cannot add content the crawl never saw. That ceiling is worth remembering when you assume a web corpus covers your domain.
C4: the classic cleaned baseline
C4, the Colossal Clean Crawled Corpus, was one of the first widely used attempts to turn Common Crawl into something tidy. It applies a set of simple, aggressive heuristics: keep lines that end in proper punctuation, drop pages with too few sentences, remove boilerplate, and strip out documents containing words from a blocklist. The result is a much smaller, English focused corpus of cleaned prose.
C4 is a sensible baseline precisely because its rules are easy to understand. The trade offs are equally clear: the blocklist approach removes some legitimate content along with the noise, and the cleaning is coarse by modern standards. It remains a good reference point and a reasonable choice when you want predictable, readable English text without much engineering. If you are comparing corpora for a study, C4 is a useful yardstick against which newer pipelines measure themselves.
FineWeb and FineWeb2: the modern default
FineWeb represents the current generation of thinking about web data. It processes many Common Crawl snapshots through a carefully tuned filtering and deduplication pipeline, using quality heuristics refined by ablation (systematically testing which filtering choices actually improve a trained model). The stated goal is to keep as much genuinely useful text as possible while discarding low quality pages, rather than relying on a blunt blocklist.
The corpus is large, on the order of trillions of tokens (a token is a chunk of text, roughly three quarters of a word on average), and it is released openly with its processing code documented. For most people building an English language dataset today, FineWeb is the sensible default: it is clean, well documented, and transparent about how it was made.
If your project is not English first, note FineWeb2, which extends the same approach to a broad range of languages. Multilingual coverage is a common gap in web corpora, and FineWeb2 is one of the better open options; the wider multilingual category covers alternatives.
RedPajama and SlimPajama: reproducing and slimming
RedPajama began as an effort to reproduce the data recipe behind a well known family of models as an open dataset. Its web component is Common Crawl derived, but the project is broader than web text alone: it bundles several source types together into a single large corpus intended for pretraining. That makes it useful when you want a ready assembled mixture rather than just filtered web pages.
SlimPajama is exactly what the name suggests: a cleaned and heavily deduplicated version of RedPajama. Deduplication matters more than it first appears. The web is full of near duplicate pages, and removing them improves training efficiency and reduces the chance a model simply memorises repeated text. SlimPajama trades raw size for a denser, cleaner corpus, so you get fewer tokens but each one earns its place.
RefinedWeb, Dolma and DCLM-Baseline
Three more derivatives round out the family, each with a distinct emphasis.
- RefinedWeb made the case that carefully filtered web data alone, without curated books or code mixed in, could train strong models. It is Common Crawl derived, aggressively deduplicated, and English focused. A large openly released subset is available, though the full corpus was kept private by its authors.
- Dolma is a large, openly licensed corpus that combines web text with other sources, and it is notable for how thoroughly its construction and tooling are documented. If reproducibility and transparency matter to you, Dolma is a strong choice, and its open toolkit lets you re-run or adapt the filtering.
- DCLM-Baseline comes from a project built specifically to compare data curation methods on a level playing field. Its baseline corpus reflects filtering choices that performed well in those controlled comparisons, which makes it interesting when data quality is the thing you most want to get right.
It is worth mentioning The Pile here too. It predates most of the corpora above and is a curated mixture of many sources rather than a pure web crawl, but it appears in the same discussions and remains a common reference dataset.
How they compare, in short
- Filtering and deduplication: Common Crawl does none. C4 applies simple heuristics. FineWeb, RefinedWeb, Dolma and DCLM-Baseline apply modern, ablation informed filtering with heavy deduplication. SlimPajama is defined by its aggressive deduplication of RedPajama.
- Scale: Common Crawl is the largest and rawest. FineWeb and RedPajama are very large. C4, SlimPajama and the openly released RefinedWeb subset are smaller but denser.
- Cleanliness and RAG readiness: all of these are still document level text, not chunked or embedded, so expect to do your own chunking and embedding whatever you choose. The cleaned derivatives simply save you the boilerplate and spam removal.
- Licence and openness: these corpora are released under permissive terms, but the underlying web content carries the original site owners’ rights. That distinction trips up a lot of teams, and it is worth reading the licensing trap before you rely on any web corpus in a product. Each source page states the specifics.
Which one should you use
For most RAG and fine tuning work you do not need a raw crawl. Reach for raw Common Crawl only when you want maximal coverage of the open web and you are willing to build and maintain your own cleaning, deduplication and language filtering pipeline. That is real engineering work, justified when your requirements are unusual or your coverage needs are extreme.
For everything else, a cleaned derivative saves weeks. Choose FineWeb as a modern English default, FineWeb2 when you need other languages, SlimPajama when density and deduplication matter more than raw size, and Dolma or DCLM-Baseline when transparency and curation quality are your priority.
If you are assembling a retrieval knowledge base rather than a training set, remember that a broad web corpus is often the wrong tool: you usually want a narrower, higher trust source for your domain, then good chunking on top. The choosing your first RAG dataset post and the hands on DIY RAG course both work through that decision, and the get your data lesson is a practical starting point.