Common Crawl

Common Crawl has been crawling the open web since 2007 and publishes a fresh snapshot most months, each one holding billions of pages and running to 200 to 400 TB compressed. Every snapshot comes in three matching formats: WARC files hold the full archived HTTP response including headers and raw HTML, WET files hold just the extracted plain text, and WAT files hold metadata such as links and page titles. For most RAG work the WET text is the natural starting point.

The data lives on AWS as a public dataset, so the cheapest way to work with it is from inside AWS in the us-east-1 region, where there is no transfer cost. Each snapshot ships with index files that let you fetch only the pages you want by URL or domain, so you do not have to pull a whole crawl to get, say, every page from a handful of sites. Tools like cdx-toolkit and the columnar URL index make that targeted access straightforward.

Common Crawl is best when you need broad, open-domain coverage at genuine web scale: pretraining a base model, building a general knowledge corpus, or mining a specific slice of the web by domain. It is raw material, not a finished corpus, and that is the main caveat. The data is noisy, full of boilerplate, spam, duplicates, and machine-translated pages, so you will need to filter, deduplicate, and clean it heavily before it is fit for retrieval.

On licensing, Common Crawl provides the archive freely, but the underlying pages stay under their original site terms, so commercial reuse is restricted and you should treat redistribution of the content with care. If cleaning raw crawl is more than you want to take on, reach for a derived corpus instead: C4, FineWeb, RedPajama, and Dolma are all built from Common Crawl with the filtering already done.