Meet the RAG Repo API: the directory as open JSON

RAG Repo has always been a directory you browse. Now it is also one you can build on. The whole catalogue is published as static JSON at rag-repo.org/sources.json, free and read-only, with no API key and no sign-up, and served with cross-origin access so you can fetch it straight from a browser, a script, or a backend. This post explains what it gives you and the sort of thing you can do with it. The full reference lives on the API page.

Why an API

If you are building anything that helps people find or ingest datasets, the awkward part is not writing the code. It is knowing which sources exist, what shape each one is in, and whether you are actually allowed to use it. That is the work this directory already does by hand, source by source, and until now the only way to get at it was to read the site. A machine-readable catalogue turns that curation into something a tool can consume directly, rather than scraping HTML and guessing.

Concretely, the catalogue is the honest, structured version of every source page: not just a name and a link, but the category, the formats, how you actually get the data, a size estimate, and the full licence position.

What each source tells you

Every entry in /sources.json carries the fields you need to decide whether a source fits before you download a byte of it:

• Category, formats and a description, the same as the source page.
• Access pattern: how you actually fetch the bytes, as one or more of huggingface, cloud-bucket, bulk-download, api, git-repo, web-portal or web-crawl. This is the field that tells a tool whether it can ingest a source automatically or needs to hand the user off.
• Size tier: a coarse tiny to huge band for the whole-corpus download volume, so you can warn before someone tries to pull a petabyte onto a laptop.
• Licence clarity: the licence itself, plus our flags for commercialUse, shareAlike, attributionRequired and contentLicenceDiffers. This is the part most directories leave you to work out yourself.
• Versioning: a version label where the source has editions, a lastReviewed date, and supersedes / supersededBy links between editions.

The response also includes a legend object that documents every one of these fields, so a consumer can read the meanings rather than hard-coding them.

A couple of examples

Fetching it is a single request:

curl https://rag-repo.org/sources.json

Because everything is in one file, you filter client-side. Say you only want open sources you can use in a commercial product:

curl -s https://rag-repo.org/sources.json \
  | jq '.sources[] | select(.accessType=="open" and .commercialUse=="permitted") | .name'

Or, in JavaScript, everything small enough to pull down locally:

const { sources } = await (await fetch('https://rag-repo.org/sources.json')).json();
const laptopFriendly = sources.filter(s => s.sizeTier === 'tiny' || s.sizeTier === 'small');

The API page has more of these, including how to pick out everything on the Hugging Face Hub or narrow to a single category.

Watching for changes

The directory is not static: sources get added, and editions move. Rather than re-downloading and diffing the whole catalogue, you can watch the lean updates feed, which lists sources ordered by when they were last reviewed, newest first. Compare against the last date you saw and you have picked up everything new. If you would rather subscribe than poll, the same thing is published as an RSS feed, and there is a human-readable what’s new page.

The one rule that matters

The catalogue metadata is yours to reuse, with attribution to RAG Repo. The important caveat is about the data it points at, not the catalogue: each dataset keeps its own licence, and that is what binds you. We surface it in the licence field and the clarity flags precisely so you can filter to what you are allowed to use, but those flags are a shortlist to help you triage, not legal advice. Before you build on a source, confirm its terms at the source. We wrote more about why this matters in choosing a dataset licence for a commercial product.

Where this is going

The API is deliberately simple: static files, no auth, no rate limits beyond being reasonable with a one-hour cache. That keeps it fast and dependable, and it is enough to build a real tool on today. Fields may be added over time, but we will not remove or repurpose the existing ones without notice, and every response is dated so you can tell how fresh your copy is.

If you build something with it, or you need a field it does not expose yet, we would genuinely like to hear about it: there is an email on the contribute page. And if you just want to understand how the sources behind the data are chosen and checked, that is what the methodology page is for.