C4 (Colossal Clean Crawled Corpus)

C4 is a single snapshot of Common Crawl (from April 2019) that has been put through a set of hand-written cleaning rules to leave behind readable English prose. The pipeline keeps only lines that end in a full stop, drops pages with fewer than a handful of sentences, strips navigation menus and boilerplate, removes a blocklist of offensive words, and throws out code and placeholder text like "lorem ipsum". What survives is about 750 GB of fairly clean natural language, small enough to work with on a single machine rather than a cluster.

The Allen Institute for AI (Allen AI) hosts C4 on HuggingFace as Parquet files. The quickest way to start is to stream it with the `datasets` library, pulling one shard at a time so you do not have to download the whole thing before you experiment. Alongside the default English split you will find variants: a noisier `en.noclean` version, `realnewslike` and `webtextlike` subsets, and the multilingual mC4 relative if you need languages beyond English.

For RAG, C4 is a dependable source of general-purpose English text: broad web coverage that has already had the worst rubbish filtered out, ready to chunk and embed for a knowledge base that needs wide but shallow coverage. It also remains a common baseline for pretraining experiments, so results are easy to compare against published work.

The main thing to watch is that the cleaning is heuristic, not semantic: the rules remove obvious junk but do not judge whether a page is accurate or useful, so you still get plenty of low-value filler and 2019-era facts. The licence is ODC-By 1.0, which permits commercial use as long as you attribute the dataset; the underlying page content carries no separate restriction flagged here.

If you want cleaner or larger web text, FineWeb applies a more modern filtering and deduplication pipeline, while raw Common Crawl gives you everything C4 discarded.