Unpaywall

Unpaywall does one job extremely well: for any given research paper, it tells you whether a legal, free-to-read copy exists anywhere and, if so, exactly where it is. It maps each work's DOI (the Digital Object Identifier, a paper's permanent web address) to open-access versions it has found in university repositories, on publisher pages, and in preprint archives, spanning well over 30 million open-access articles. It does not store the papers themselves; it is the index that points you to legitimate full text.

There are three ways in. A free REST API takes a DOI and returns JSON listing every location a copy lives, complete with host type and version. A full database snapshot lets you download the whole thing and work locally for bulk jobs, and a companion change feed keeps that local copy current without hammering the API. There is also a browser extension for casual reading, though for a pipeline the snapshot-plus-feed route is the one to build on. The data comes from OurResearch, the nonprofit behind OpenAlex, and sits under CC0, so you can reuse the metadata freely with no attribution required.

In a RAG workflow, Unpaywall is the filter between discovery and ingestion. Pair it with a metadata catalogue like Crossref or OpenAlex to find the papers relevant to your domain, then let Unpaywall narrow that list to the ones you can lawfully fetch and index, handing you a direct link to each full text. That keeps your corpus both relevant and legally clean.

The one thing to keep straight is the licence split. The Unpaywall metadata is CC0 and safe for commercial use, but every paper it points to carries its own terms. A free-to-read location is legal to read, yet the article underneath may sit under a restrictive publisher agreement that limits redistribution, so respect each work's own licence when you store or serve its text.