Semantic Scholar / S2ORC

Semantic Scholar is a free literature search and discovery service from the Allen Institute for AI, and behind it sits a large academic graph connecting papers, authors, venues, and the citations between them. That graph is what makes it valuable for RAG: you can retrieve not just a paper but the web of work it cites and is cited by.

S2ORC, the Semantic Scholar Open Research Corpus, is the bulk dataset you download to work locally. It gives you cleaned full text where the licence allows, plus parsed section structure, inline citation markers, and reference lists, so you can chunk at the passage level and follow citation links between papers instead of treating each one as an opaque blob. The papers are pooled from arXiv, PubMed, Crossref, direct publisher feeds, and web crawls, which gives it far wider coverage than any single source.

There are two ways in. The S2ORC snapshot on the Hugging Face Hub is the route for bulk local use, delivered as JSON and JSONL that you can stream. The Semantic Scholar API is the route for live lookups of individual papers, authors, and citations, and it is the better fit when you want fresh metadata rather than a fixed snapshot; a free API key raises the rate limits.

On licensing, the bulk S2ORC corpus is released under ODC-By, an open data licence that lets you use the data, including commercially, as long as you credit the source. Live API access is governed by separate API terms. The usual caveat applies to the full text: it is only as complete as what publishers allow, so some records are metadata and abstract only.

If you are building RAG over science, S2ORC pairs the cleaned text of arXiv with a citation graph and the reach of PubMed and Crossref in one downloadable place.