CC-News

CC-News is a continuously updated news corpus from the Common Crawl Foundation. Separately from the main monthly web snapshots, Common Crawl crawls news sites around the world day after day and packages the results as their own stream, so instead of a single point-in-time dump you get a steady flow of articles reaching back to 2016 and running to millions of pieces across many languages, though English dominates.

The raw data is published as WARC files (Web ARChive, the standard format that stores each crawled page's full HTTP response, headers and all) and hosted on AWS as a public dataset, so the cheapest way to work with the originals is from inside AWS, where there is no transfer cost. If you do not need the raw archive, a cleaned, article-extracted version is mirrored on Hugging Face and loads in a couple of lines with the `datasets` library, which is the quickest way to get usable article text for experiments.

For RAG, CC-News is a convenient off-the-shelf news corpus for current-affairs and media use cases: building a knowledge base of reporting, fine-tuning on journalistic writing, or testing retrieval over time-stamped articles. It has fed several pretraining pipelines, including the corpus behind RoBERTa, so it is well trodden.

The caveats are those of news and of Common Crawl together. Coverage depends on which sites were crawled and when, so it is broad but not comprehensive or evenly balanced, and near-duplicate articles (wire copy republished across outlets) are common, so deduplicate before indexing. On licensing, the articles remain under their original publishers' terms even though Common Crawl provides the archive freely, so commercial reuse is restricted and the content licence differs from the packaging: fine for research and internal work, but take advice before republishing article text in a product. If you want deeper structured news metadata, GDELT is a strong companion; CC-News gives you the full article text GDELT largely points at.