arXiv

arXiv has been the home of scientific preprints since 1991, the place researchers post work before, or instead of, formal journal publication. It holds well over 2.5 million papers across physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering, and economics, and almost all current machine learning research appears here first. Each paper carries metadata, the author's LaTeX source where available, and a PDF.

There are two bulk routes. The full text and PDFs are offered as a requester-pays dataset on Amazon S3, meaning arXiv hosts the files but you cover the download and transfer costs, which is the practical way to get everything. Metadata alone (titles, authors, abstracts, and categories) can be harvested for free through OAI-PMH, a standard protocol repositories use to expose their records for bulk collection, and there is also a lightweight query API for smaller, targeted pulls.

For RAG, the LaTeX source is worth the extra handling over the PDFs: it gives you clean section structure, headings, and equations without the layout mangling that PDF text extraction so often introduces. arXiv suits any system that answers technical or research questions, and works well when you retrieve at the section or paragraph level rather than treating a whole paper as one chunk.

Watch two things. Preprints are not peer reviewed, so a retrieved claim may be preliminary or later revised, and papers have versions (v1, v2, and so on) you should track. On licensing, terms vary by paper: most sit under arXiv's non-exclusive distribution licence, which does not clearly grant commercial redistribution, while some authors add a Creative Commons licence, so check each paper before redistributing its text.

For fuller coverage of published literature and a citation graph on top, pair arXiv with Semantic Scholar's S2ORC or, for biomedical work, PubMed Central.