Redistributing data: linking vs bundling
When can you copy and re-host a dataset, not just point at it? The licence rules for redistribution, in plain English.
There is a difference between using data and redistributing it, and it is the difference that catches people out. Pointing your RAG system at a source, or linking to it the way this directory does, is one thing. Copying that source, re-hosting it, and handing it to other people, as a downloadable dataset or an offline bundle, is another. The second act asks a harder question of the licence, and a source that is perfectly fine to read or query can be one you are not allowed to redistribute at all.
This guide is the companion to Data licensing for RAG. It is practical, not legal advice. When money or real risk is involved, read the actual licence and, if in doubt, ask a lawyer.
Two different acts
Most licence confusion clears up once you separate these:
- Use and link. You read a source, query it, retrieve from it, and point people at the original. You are not making a copy for others to keep.
- Redistribute. You copy the content, store it, re-host it, and ship it to other people so they hold their own copy. A downloadable dataset, a corpus on a mirror, or an offline bundle are all redistribution.
RAG Repo itself only links. That is why the directory can list a source under any licence: a signpost redistributes nothing. The moment you build a dataset people download, you have crossed into redistribution, and every document in it has to clear that higher bar.
The question redistribution adds
Data licensing for RAG gives you two questions: can you use it commercially, and does using it force conditions onto your own work. Redistribution adds a third:
May I copy this, re-host it, and let the people I give it to use it too?
Run each source through all three. Here is how the common licences answer the third one.
Copy freely
- Public domain and CC0. No copyright to clear. Copy, re-host and redistribute without conditions. CC0 asks for nothing, not even attribution (a courtesy credit is still polite).
- Works of the US federal government. Under US law these carry no copyright, so federal agency material is effectively public domain. The trap: this covers work by federal employees. Material produced by contractors, grantees or university researchers under a co-operative agreement does not automatically qualify, even when it sits on a government website. Check who actually authored it before you assume you can ship it.
Copy, but attribute
- CC BY. Redistribute freely, including commercially, as long as you credit the source as the licence asks. Keep the credit with the content.
- UK Open Government Licence (OGL). Crown copyright material, such as a lot of gov.uk and NHS content, is reusable commercially with an attribution statement. Some bodies attach a refresh clause to verbatim copies; the OGL “adapted content” route (paraphrase and cite, rather than copy word for word) usually avoids that and suits a static download better.
Copy, but your bundle inherits the licence
- CC BY-SA, and the GFDL. Share-alike is sticky. If your redistributed work is a derivative, it has to carry the same share-alike licence. Fold one CC BY-SA source into a bundle and you can push the whole bundle into being share-alike, which may be the opposite of what you want. Wikipedia, Wikibooks and Appropedia are CC BY-SA.
- ODbL. The same idea for databases: a derived database must be offered under ODbL too. Open Food Facts is ODbL. If you want a permissive, commercially-reusable bundle, keep share-alike sources in a separate pack rather than mixing them in.
Do not put it in a redistributable bundle
- Non-commercial licences (CC BY-NC, NC-SA). You cannot redistribute these inside anything others may use commercially, or anything you sell. Link to them instead.
- All rights reserved, custom terms, or unverified. No permission to copy means no redistribution. If you cannot find and read the licence, treat it as off-limits for the bundle and link only.
Provenance is the other half of the job
Permission is not enough on its own; you have to be able to show it. A redistributable dataset should ship with a manifest that records, for every document: where it came from, its licence, the exact attribution to reproduce, the date you retrieved it, and a checksum. That manifest is what makes the dataset defensibly compliant, and it is a feature, not overhead: anyone building on your data can see what they are standing on.
Two practical habits go with it:
- Strip what is not yours even inside something that is. Government and agency pages almost always embed third-party photographs and reserve their logos and trademarks, so bundle the text and data, not the images. If your own documents quote a source verbatim, track the licence of each quote, not just of the document, and paraphrase or link any quote you are not allowed to redistribute.
- Pin and date your snapshot. Live sources change and some disappear. Record the exact version or date you shipped, so a reader knows what they have and you can show what the source said when you copied it.
Rules of thumb
- Linking is not redistributing. You can point at anything; you can only ship what the licence lets you copy.
- Public domain, CC0, CC BY and OGL are your friends for a commercial, reusable dataset. Your own original content is cleanest of all, because you set its terms.
- Keep share-alike (CC BY-SA, ODbL) in its own pack, or it makes the whole bundle copyleft.
- Non-commercial and all-rights-reserved sources are for linking, not bundling.
- Ship a provenance manifest and an attribution file. If you cannot say where a document came from and under what licence, it should not be in the bundle.
Get redistribution right and you can turn a scattered set of sources into something people can actually download and build on, which is where a pile of links becomes a dataset.