CORE

CORE harvests open-access research from thousands of institutional repositories, preprint servers, and journals worldwide and pulls it into a single searchable collection. It holds over 300 million metadata records and more than 40 million full-text articles, which is the part that sets it apart: many scholarly aggregators give you only titles, authors, and abstracts, whereas CORE actually stores the full text of the paper where an open-access version exists. Instead of querying each repository separately and reconciling their quirks, you get one place to find and retrieve papers across every discipline.

You can reach the collection three ways. The search API is best for live lookups and building retrieval on top of CORE directly, returning JSON records with links to the full text. The bulk data dumps are the practical choice for building your own corpus offline, since pulling tens of millions of papers through an API is slow and rate-limited. There is also a dataset service aimed at research use. Register for a free API key to lift the anonymous rate limits before you start any serious ingestion.

For RAG (Retrieval-Augmented Generation), CORE is at its best when you want broad, cross-disciplinary open-access coverage without wiring up dozens of repositories by hand: a general research assistant, a literature-aware question-answering tool, or the raw material for a scientific knowledge base. The text arrives as PDFs and extracted plain text, so budget time for parsing, cleaning boilerplate, and chunking before you embed anything.

The main caveat is licensing. CORE aggregates rather than publishes, so full-text reuse terms follow each original source and vary paper by paper; the metadata is freely available, but do not assume every full text is cleared for commercial reuse. Where CORE gives you full text at scale, Crossref and OpenAlex give you cleaner metadata and citation links, so many builds use them together.