A curated directory of open datasets, knowledge bases, and data repositories for AI. Ideal for Retrieval-Augmented Generation (RAG), fine-tuning, and any project that runs on high-quality data you can trust.
238
data sources
28
categories
213
open access
Large-scale web-crawled text datasets
Wikipedia, Wikidata, and structured knowledge bases
Research papers, preprints, and citation graphs
Semantic knowledge graphs and ontologies
Source code repositories and developer Q&A
Court opinions, legislation, and regulatory filings
Official government open data portals
Geographic data, maps, and place databases
News archives, event databases, and media datasets
Public domain books and literary corpora
Clinical data, drug databases, and medical literature
Platforms for discovering and trading datasets
Benchmark datasets designed for RAG evaluation
Awesome-lists and dataset directories
Large text datasets beyond English
Spoken-language and audio datasets
Paired image and text datasets
Maths, proofs, and technical reasoning
Standard benchmarks for retrieval quality
Datasets that arrive already vectorised
Patent full text and metadata
Museum, library, and archive collections
Structured scientific and chemical data
Satellite, weather, and environmental data
Official statistics and economic indicators
Open textbooks and course materials
Product, food, and company databases
Tool-calling, function-calling, and agent trajectory datasets
A good place to start, whatever you are building.
An open-access preprint server for physics, mathematics, computer science, quantitative biology, statistics, and more, holding over 2.5 million papers. You can pull the full archive in bulk from Amazon S3, or harvest the metadata through OAI-PMH, a standard protocol for sharing records between repositories.
6.9M US court decisions spanning the 1600s to 2020, digitised by Harvard Law School Library and released in a consistent machine-readable format. Covers every official, book-published state and federal case, with a free bulk API.
The November 2023 Wikipedia dump across 300+ languages, split into passages and turned into embeddings (numeric vectors that capture meaning) with the Cohere Embed V3 multilingual model. Close to 250 million paragraph embeddings in total. Drop it into a vector database for semantic search over all of Wikipedia, or use it as a knowledge source for multilingual RAG, including cross-lingual search where a query in one language returns relevant results from another.
A nonprofit that crawls the web and freely provides its archives and datasets. Petabytes of raw web data from billions of pages, with a new snapshot each month. Used in the training of GPT-3, LLaMA, T5, and many other large language models.
The largest open hub for machine learning datasets, with well over 100,000 datasets you can search, stream, and version. Many RAG-ready corpora live here, and its Python library lets you pull data straight into a pipeline in a few lines.
The Massive Text Embedding Benchmark, a common framework for comparing text embedding models (the numeric vectors that represent meaning for search and retrieval). It brings together SemEval, BEIR, and many other datasets across tasks such as retrieval, classification, and clustering. The MMTEB extension widens the scope to over 250 languages and more than 500 tasks.
A crowdsourced map of the world, built and maintained by a community of volunteers. It holds detailed geographic data on roads, buildings, land use, points of interest, and much more, which you can download as a full planet file or as smaller regional extracts.
A portfolio of free geospatial datasets covering Great Britain from the national mapping agency, available for commercial and personal reuse under the Open Government Licence. Provides mapping, boundary, and location data through the OS Data Hub in formats including GeoPackage, GML, Shapefile, and CSV.
Research and interactive visualisations on global problems, built by aggregating and harmonising data from official sources. Every chart can be downloaded as a ZIP containing a CSV file, JSON metadata, and a README, and the underlying data is available programmatically via an API. Full provenance is documented for every dataset.
An image-text dataset built only from materials marked with a Public Domain Mark or released under Creative Commons Zero (CC0). 12.4M image-caption pairs, with a 3.3M subset called PD3M, sized to match the Conceptual Captions datasets while staying copyright-clean for commercial use.
A volunteer effort to digitise and archive public domain books, with more than 70,000 free ebooks. Mostly English, but it covers many languages. One of the oldest digital library projects, so it is a clean, permissively licensed source of full-text literature for RAG.
The US National Library of Medicine's database of biomedical and life sciences literature. PubMed indexes over 36 million citations and abstracts, and PubMed Central (PMC) adds free full-text access to a growing subset. You can download the data in bulk over FTP.
Allen AI's academic graph, covering hundreds of millions of papers linked by billions of citations. S2ORC, the Semantic Scholar Open Research Corpus, is the downloadable dataset with full text and parsed references, drawn from arXiv, PubMed, Crossref, publishers, and web crawlers.
Periodic XML dumps of every site in the Stack Exchange network, including Stack Overflow. Each dump holds questions, answers, comments, votes, and user profiles. One of the richest question-and-answer datasets you can freely download, and a natural fit for building technical support and developer RAG systems.
A free, collaborative knowledge base that holds structured facts for Wikipedia and the other Wikimedia projects. More than 100M items, each one machine-readable with statements, relationships, and identifiers, so you can pull clean facts instead of parsing article prose. Released into the public domain under CC0.
Complete database dumps of Wikipedia and the other Wikimedia projects, in every language, refreshed roughly twice a month. The most widely used knowledge source for RAG systems, and an easy first corpus to build a retrieval pipeline on.
More than a directory: guides, comparisons, and research to help you choose and use data well.
Practical guides, comparisons, and roundups for building with data.
Deeper reading on where retrieval is heading, and what it means today.