How to choose a dataset licence for a commercial RAG product

Before a Retrieval-Augmented Generation (RAG) system reaches paying customers, someone has to answer a plain question about every dataset in it: are we actually allowed to use this? RAG systems retrieve passages from a corpus and feed them to a language model, so the data you index is not a passive dependency. It is baked into what your product outputs, and in some cases into the model weights you fine-tune. This guide walks through the licensing questions that matter for a commercial launch, framed as four checks you can run against any source. It is practical guidance for engineers and product owners, not legal advice. When money and reputation are on the line, have a qualified lawyer read the licences that apply to you.

The four checks below map directly to the licence-clarity fields we record on every entry in our directory. If you would rather skim before reading, each source page shows the same flags: whether commercial use is permitted, whether a share-alike clause applies, whether attribution is required, and whether the content licence differs from the dataset licence.

Check 1: Can I use it commercially at all?

Start here, because a “no” ends the conversation quickly. A large number of open datasets carry permissive licences that allow commercial use as long as you meet simple conditions, usually attribution. The common ones you will meet are Creative Commons Attribution (CC BY), the Open Data Commons Attribution Licence (ODC-By), and, for code and code-adjacent corpora, MIT and Apache 2.0. All of these permit commercial products. They differ mainly in what you must preserve: a notice, a copyright line, a link back, or a list of changes.

Permissive does not mean obligation-free. CC BY and ODC-By both require attribution, and “we used open data” in a footer is not always enough. Read the specific wording for how credit must be given and whether a licence notice must travel with any redistributed copy. If your product ships the corpus itself, or a filtered subset of it, those conditions follow the copy. If you only serve retrieved snippets to end users, the obligations are lighter but not always zero. Apache 2.0 adds a patent grant and a requirement to note changes, which rarely bites for data but matters if you are redistributing tooling alongside the corpus. The safest habit is to keep the exact licence text next to the data, so the conditions are never a matter of memory.

Public-domain and CC0 material is the cleanest case: no conditions at all. Much of the books and literature catalogue falls here, and public-domain government works often do too.

Check 2: Is there a share-alike clause?

Share-alike (sometimes called copyleft for data) is the check teams most often miss, and the one most likely to cause trouble after launch. A share-alike licence says: you may build on this, including commercially, but anything you derive and distribute must be released under the same or a compatible licence. The two you will meet constantly are CC BY-SA and the Open Data Commons Open Database Licence (ODbL).

The obvious example is Wikipedia, whose text is CC BY-SA. The encyclopaedic category is full of similar structured knowledge that carries a share-alike condition. In the geospatial world, ODbL is the norm for large community-maintained map data.

The question that matters for a commercial RAG product is how far the clause reaches. Serving a retrieved passage to a user is generally fine. The risk sits in redistribution and in derived databases. If you produce a cleaned, chunked, or enriched version of a share-alike dataset and then distribute that artefact, the share-alike obligation can attach to it. ODbL in particular is written around “derivative databases”, which is close to what a RAG pipeline produces when it restructures a corpus into chunks and embeddings. Whether your vector index counts as a derivative database, and whether serving results counts as distribution, are exactly the questions to raise with a lawyer before you commit. If your business model depends on keeping your processed data proprietary, a share-alike source may not fit.

Check 3: Is it non-commercial only?

Some of the most convenient datasets are released under a non-commercial licence, most often CC BY-NC. These are free to download and easy to prototype with, which is precisely why they end up in production systems by accident. The “NC” prevents exactly the use a commercial RAG product makes of them.

The practical hazard is timing. A dataset picked during a research spike, or copied from a tutorial, can sit quietly in a pipeline until launch. By then it is entangled with everything else, and removing it means re-indexing. Catch non-commercial sources at selection time. When you browse the directory, filter them out early rather than discovering the “NC” during a pre-launch review. Our companion piece, The licensing trap, goes deeper on how these terms slip through unnoticed, and choosing your first RAG dataset covers how to weigh licence alongside quality when you are just starting out.

Check 4: Does the content licence differ from the dataset licence?

This is the subtle one, and it applies even after the first three checks pass. A dataset can carry a clean, permissive licence over its structure, packaging, or metadata while the underlying content it points to or contains is governed by something else entirely. Two licences, two sets of rules, one dataset.

Web crawls are the classic case. Common Crawl makes its archive freely available, but the crawl is billions of pages authored by other people, each page still subject to its own copyright and the terms of the site it came from. The permission to access the crawl is not a permission to reuse every page inside it however you like. Anything built on top of large web corpora inherits this ambiguity.

Aggregated metadata is a second case. The Digital Public Library of America releases its metadata under CC0, so the records describing items are free to use. The items those records point to, the digitised books, photographs, and manuscripts held by contributing institutions, are not automatically free. This pattern is common across the cultural-heritage catalogue: open metadata, mixed-rights objects.

Mixed corpora are the third. Pile of Law gathers many legal sub-collections whose terms vary by subset, so a single top-level licence line does not tell you what governs each part. By contrast, a source like legislation.gov.uk is clearer, because official UK legislation is published under a well-defined open licence. Even within a domain like mathematics, a corpus such as MegaMath draws from sources of differing provenance, and code collections like The Stack carry the individual licences of each repository they include. Where the content licence differs from the dataset licence, we flag it on the source page so it is visible before you commit.

A pre-ship checklist

Run this before a commercial launch, once per dataset in your index:

• Confirm commercial use is permitted, in writing, in the actual licence text.
• Identify any share-alike clause and decide whether your processed data or fine-tuned model could inherit it.
• Rule out non-commercial sources entirely; do not let a prototype dependency reach production.
• Check whether the content licence differs from the dataset or metadata licence.
• Record the attribution you owe and build it into the product, not a post-launch to-do.
• Keep a short provenance note per source: where it came from, which licence, which version, and the date you checked.

Licences change, and datasets get re-released under new terms, so treat this as a recurring review rather than a one-off. If a term is genuinely unclear, we mark it as such rather than guessing, and so should you. The glossary defines the licences and terms named here, and the wider learn section covers how the data choices you make early shape the system you end up with. When you are ready to choose sources, the directory’s commercial-use filter is the fastest way to narrow the field to what you can actually ship.