The open web is closing: what it means for your training data

For most of the past decade, the working assumption behind large-scale AI data was simple: if a page was reachable, it was fair game. Scrape it, filter it, train on it, move on. That assumption is now falling apart from several directions at once. None of these developments is fatal on its own. Taken together, they mark a shift in how the raw material for Retrieval-Augmented Generation (RAG) and model training is going to be sourced, and provenance is moving from a nice-to-have to the centre of the story. (None of this is legal advice, and several of these matters are still live.)

The settlement that put provenance on the invoice

The clearest signal came from the courts. In Bartz v. Anthropic, a US federal court drew a line that the industry had been arguing about for years: training a model on books the company had lawfully acquired was treated as fair use, but downloading and retaining pirated copies to build a permanent library was not. The distinction was not about the training itself. It was about where the copies came from.

That distinction now has a price attached. In 2026 the court gave final approval to a settlement of at least US$1.5 billion covering roughly 500,000 works, reported as around US$3,000 per book and described as the largest copyright recovery on record. The settlement is not binding precedent, so it does not resolve the underlying law. What it does is make the cost of bad provenance concrete. If the same corpus had been assembled from lawfully sourced copies, the analysis would have looked very different. The lesson for anyone building a data pipeline is blunt: how you obtained a document can matter more than what the document contains.

Regulation now wants the receipts

Where the courts have set a price, regulation is starting to demand disclosure. Under the EU AI Act, providers of general-purpose AI models must publish a “sufficiently detailed summary” of the content used for training, following a template released by the European Commission under Article 53(1)(d). The obligation began applying to new models from 2 August 2025, with a transitional deadline of 2 August 2027 for models already on the market, and it reaches free and open-source providers too.

You cannot summarise what you did not track. A transparency template turns provenance from an internal engineering detail into a published artefact that copyright holders and regulators can read. Teams that logged their sources will find this tedious. Teams that hoovered up a mixed bag of web scrape and cannot say what is in it will find it genuinely hard. That asymmetry is the point, and it rewards the same discipline the courts are now pricing: keep a manifest, record the licence, and be able to show your working. The disclosure obligation lands on model providers rather than RAG builders directly, but the habit it forces is one any serious data team should already have.

The base layer is being fenced off

Meanwhile, the freely scrapable web itself is shrinking. In July 2025 Cloudflare began blocking AI crawlers by default for new sites, flipping the long-standing opt-out model to opt-in, and floated a “Pay Per Crawl” marketplace letting publishers charge bots per request (later reworked toward paying when content is actually used in an answer). One infrastructure provider does not control the whole web, but Cloudflare sits in front of a very large slice of it, and defaults are powerful. A generation of pipelines was built on the premise that access equals permission. That premise is being switched off at the edge.

The tension is sharpest around news. An investigation by The Atlantic in late 2025 alleged that Common Crawl’s archives contained large volumes of paywalled articles from major publishers, despite the organisation stating it does not go behind paywalls. Common Crawl underpins a great deal of open pretraining data, so questions about what sits inside it ripple outward into every derivative corpus. If you build on Common Crawl or on the many web corpora derived from it, this is not someone else’s problem. It is a question about your own inputs, and one worth understanding before you rely on the answer.

The Reddit dispute rounds out the picture. In October 2025 Reddit sued Perplexity and several data-scraping firms, alleging industrial-scale collection of user comments obtained via search results and resold without consent. Perplexity disputes the characterisation, and the case is ongoing. Whatever the outcome, the direction is unmistakable: platforms that once tolerated open access are now treating their content as a licensable asset and litigating to enforce that.

The throughline

Read together, these are not five separate stories. They are one story told five ways. The courts are pricing provenance. Regulators are demanding it in writing. Infrastructure providers are gating access at the source. Publishers and platforms are asserting rights over content that used to be treated as ambient. The comfortable base layer of freely scrapable web data is being fenced off and made legally riskier at the same time.

For practitioners this reframes what “good data” means. It is no longer only about scale, cleanliness or freshness. It is about whether you can say where each part came from and under what terms you may use it. A corpus that scores well on every quality metric but cannot answer that question is now a corpus with a hidden liability, and the size of the Bartz settlement is a rough indication of how large that liability can grow. We have written before about how easily this catches teams out, in the licensing trap, and about the practical choices involved when the output is commercial in choosing a licence for commercial RAG. The environment described above is exactly why those questions have stopped being academic.

Why copyright-clean corpora are suddenly the interesting ones

Here is the constructive turn. The same pressure that makes indiscriminate scraping riskier has made a different kind of release genuinely valuable: corpora built from public-domain and permissively licensed material, with provenance you can trace to the token.

A cluster of these has arrived, and they are worth knowing by name. Common Corpus, released by Pleias, gathers over two trillion tokens of public-domain and permissively licensed text across more than thirty languages, drawn from cultural-heritage collections, open code, legal and financial documents and academic journals. EleutherAI’s Common Pile assembles roughly 8TB from thirty openly licensed and public-domain sources, and the models trained on it held their own against comparable models trained on unlicensed data, which quietly undercuts the assumption that clean data means weak data. Harvard’s Institutional Books contributes hundreds of thousands of public-domain volumes, carefully OCR-corrected and documented, from library scans rather than a web crawl. And MixtureVitae takes a permissive-first approach at web scale, tiering its sources by legal risk and shipping shard-level provenance metadata so you can make risk-aware choices rather than hoping for the best.

These are not charity projects. They are a bet that traceable licensing is about to be a competitive advantage, and the developments above suggest the bet is sound. For a RAG builder the appeal is even more direct than for a pretrainer: your retrieval corpus is smaller, more visible and easier to audit than a trillion-token training set, so choosing clean inputs costs you little and buys you a defensible answer to the provenance question when it comes.

None of this means the open web is finished, or that every scrape is now a liability. It means the era of treating access as permission is closing, and the projects that took provenance seriously early are the ones worth building on. If you are starting fresh, our view on how we assess and record sources sits in our methodology, the full catalogue is at /datasets, and if the retrieval-versus-training trade-off is on your mind, our research note on long context versus RAG is a useful companion. The web is not closing so much as growing a set of doors. The interesting question now is which ones you have a key to.