How we review sources

RAG Repo is a curated directory, not an automatically scraped list. Every source is chosen, described, and checked by hand against a consistent set of standards. This page explains that method, so you can judge how much to trust what you read here, and hold us to it.

What earns a place in the directory

We favour accuracy over completeness. We would rather describe a smaller number of sources well than list a huge number badly. A data source earns an entry when it meets three tests:

• It is genuinely useful for retrieval or AI work. The directory is built for people assembling data for Retrieval-Augmented Generation (RAG), fine-tuning, and related AI projects. A source has to help with that, whether it is a broad web corpus, a focused domain dataset, an evaluation benchmark, or a structured knowledge base.
• It is openly accessible, or clearly worth the access it asks for.Most entries are free and openly licensed. We do list some sources that require registration, credentialing, or payment, but only when the data is important enough to justify the hurdle, and we always say what the hurdle is.
• It is documented well enough to describe honestly. If we cannot establish what a source contains, how it is licensed, and what shape the data is in, we leave it out rather than guess.

We always link to the primary source, the official site or repository, never a mirror or a third-party reseller. If a source is discontinued but still valuable as an archive, we mark it as archived rather than quietly dropping it.

How we describe each source

Every entry follows the same structure, so you can compare sources at a glance. For each one we record a plain-English description of what it is and what it is good for, its licence, its approximate size, the formats it ships in, how often it updates, who maintains it, and links to the documentation, downloads, API, and code where they exist. The descriptions are written to be neutral: we tell you what a source is and where it fits, and let you decide, rather than ranking or hyping.

A machine-readable catalogue

Everything in the directory is also published as a single JSON file at/sources.json, regenerated on every build. For each source it gives the category, formats, access patterns (how you actually fetch the data), a coarse size tier, the licence and our licence-clarity flags, and links back to the primary source and its page here. It carries a legend that documents every field. Reuse it with attribution to RAG Repo; each source keeps its own licence, recorded in the file.

How we judge access

Every source carries one of three access tiers, shown as a coloured badge, so you can see the practical barrier to entry before you click through:

• Open means the data is freely available to download or query, under an open licence, with no gate beyond perhaps a free account.
• Limited covers sources that are free but conditional: they may need registration, credentialing, an application, or have a free tier alongside paid access. Useful data, with a catch worth knowing about first.
• Commercial means access is primarily paid, or the source is a marketplace or platform where data is bought and sold.

How we assess RAG-readiness

Two sources can both be excellent and still take very different amounts of work to use. To capture that, we tag each one with how ready it is to drop into a retrieval pipeline:

• Raw: as collected, often with boilerplate, markup, or noise to strip.
• Cleaned: filtered and tidied, but not yet split for retrieval.
• Chunked: already divided into passages suitable for indexing.
• Embedded: shipped with precomputed vectors, ready to load into a vector store.
• Mixed: offered in several of these states, depending on the subset you take.

This is an editorial judgement about the data as distributed, meant to set expectations, not a precise technical grade. You can read more about these stages in our Learn section.

How we handle licences

Licensing is where directories most often mislead people, so we treat it carefully. We state the licence on every entry. Where a licence is genuinely unclear, we say so plainly rather than guess. Beyond the licence name, we record four practical flags that answer the questions teams actually ask before shipping:

• Commercial use: whether the terms permit, restrict, or prohibit commercial use, or leave it unclear.
• Share-alike: whether a copyleft clause could require you to release derived data or models under the same terms.
• Attribution required: whether you must credit the source.
• Content licence differs: whether the licence on the dataset itself differs from the licence on the underlying content, a common trap with web crawls and aggregated collections, where the compilation is open but the individual items are not.

These flags are our reading of the published terms, provided to help you shortlist. They are not legal advice. Before you build on any source commercially, confirm the terms with the licence text yourself, and take proper advice if anything material rests on it. Our guide to choosing a dataset licence goes into this in more depth.

Why money never influences the directory

RAG Repo is a community resource that also needs to be sustainable, so it does carry advertising and, in clearly labelled blog posts, may carry affiliate links. We keep a firm wall between that and the directory. Source listings, their ordering, which sources are featured, and the editorial notes are never affected by advertising or any commercial relationship. There are no sponsored listings, and there never will be. Where money appears, it is a single, plain ad unit on a page, or a disclosed affiliate link inside an opinion piece, never a thumb on the scale of what we recommend.

Keeping it current, and fixing mistakes

Data sources change: licences are revised, datasets are archived, links move. We review entries over time, but we also rely on readers. If you spot something out of date or wrong, or know a source we should include, please tell us on thecontribute page. Corrections are welcome and we act on them, because a reference is only as good as its accuracy.