Notes on RAG, data repositories, and the messy business of data licensing.
The Protein Data Bank, AlphaFold, and UniProt are extraordinary open resources, but they are 3D structures and sequences, not prose. How to actually plug structural-biology data into a RAG or AI system, and the literature layer that makes it useful.
Read Building a structural-biology RAG system: the data you actually need
A wave of settlements, regulation and access controls is reshaping where AI training and RAG data can come from. What changed, why it matters, and why copyright-clean corpora are suddenly the interesting ones.
Read The open web is closing: what it means for your training data
A curated tour of the open datasets we reach for most when building RAG and AI systems, from web-scale corpora and encyclopaedic knowledge to legal text and evaluation benchmarks.
The big open web corpora look alike from the outside. Here is how Common Crawl, FineWeb, C4, RedPajama and their cousins actually differ in filtering, size, cleanliness and licence, and how to choose between them.
Read Common Crawl vs FineWeb vs C4 vs RedPajama: choosing a web corpus
A practical guide to reading dataset licences before you ship: permissive versus share-alike versus non-commercial, attribution, and the trap where the content licence differs from the dataset licence.
Read How to choose a dataset licence for a commercial RAG product
A practical guide to sizing hardware for local RAG, from RAM and GPU for embeddings and local models to the storage your corpus and vector index will really consume.
Read Hardware for a local RAG system: what you actually need
Case law, legislation and regulatory filings for legal RAG across the UK, US and EU, with notes on licensing and how to structure legal text for retrieval.
Read Building a legal RAG system: the data sources you actually need
You cannot improve what you cannot measure. The retrieval and RAG evaluation datasets worth knowing, from BEIR and MS MARCO to MTEB and purpose-built RAG benchmarks, plus how to build your own eval set.
Read Evaluating your RAG system: the benchmarks and datasets that matter
How you split documents shapes retrieval quality more than almost anything else. A practical tour of fixed-size, recursive, structure-aware, semantic and contextual chunking, and how to choose.
Open multilingual text and speech corpora for building RAG beyond English, from CulturaX and FineWeb2 to regional and community-driven datasets, with notes on coverage, quality and licences.
The two most-used knowledge sources in RAG, done properly: which dumps to use, how the text and the knowledge graph complement each other, licensing, and pre-embedded options.
Why data licences catch RAG teams out, and how to spot share-alike, non-commercial, and not-quite-open terms before they become a problem.
Practical, friendly advice for picking your first RAG dataset: start small, check the licence and format, and match the data to what you are building.
RAG Repo is a free, curated directory of open datasets and knowledge bases for building RAG systems, plus plain-English guides to help you use them.
Prefer a feed? Subscribe via RSS.