Wikipedia and Wikidata for RAG: the practical guide

Almost every RAG (Retrieval-Augmented Generation) system built for general knowledge starts in the same place: Wikipedia and its structured sibling, Wikidata. They are free, broad, well-maintained, and reasonably clean, which makes them the default answer to “what should I put in my retrieval corpus?” This guide covers how to get them, how they complement each other, what their licences ask of you, and the pre-built options that let you skip the tedious parts. You will find both, along with their neighbours, in our encyclopaedic and knowledge graphs categories.

Wikipedia: the passage corpus

Wikipedia is the workhorse text corpus for RAG. It gives you hundreds of millions of words of encyclopaedic prose across every major topic, in over three hundred languages, written in a consistent house style and reviewed by a large editing community. When your system needs a passage it can quote to ground an answer, Wikipedia is usually the first corpus to reach for.

There are three practical ways to get it, in rough order of how much work they save you.

Official dumps. The Wikimedia Foundation publishes complete database dumps of every project, refreshed roughly twice a month, one per language edition. These are the canonical source and the most current, but they arrive as compressed XML full of MediaWiki markup.
Cleaned distributions. Several groups republish parsed, plain-text versions with the markup already stripped. These save you the parsing work and are usually the right choice unless you need something bespoke.
Pre-chunked and pre-embedded sets. Some distributions go further and split articles into passages, and a few include vector embeddings. More on those below.

If you take the official dumps, budget real time for cleaning. Raw Wikipedia is written in wikitext, a markup language with templates, infoboxes, reference tags, tables, and category links that you almost never want in a retrieval passage. Stripping it well is fiddly: a naive regex pass leaves behind template residue and mangles nested structures. The cleaned distributions exist precisely because this step is annoying to get right, and reusing someone else’s careful parse is often the sensible call. Whichever route you take, you will still want to decide how to split articles into retrieval units. Our lesson on chunking your documents covers the trade-offs; a common approach is to chunk by section or paragraph rather than treating a whole article as one passage.

Wikipedia text is released under Creative Commons Attribution-ShareAlike (CC BY-SA), with some material also under the GNU Free Documentation License. The two obligations that matter for RAG are in the name.

Attribution. You must credit Wikipedia and its contributors.
Share-alike. If you create and distribute an adapted version of the text, you must release that adaptation under the same or a compatible licence.

For most RAG use, where you retrieve passages and quote or summarise them in generated answers, this is straightforward: attribute the source. The share-alike clause becomes something to think carefully about if you build and redistribute a derived corpus, for example a cleaned or reformatted dump you publish for others. Verbatim quotation in an answer is different from redistributing a modified dataset, and the obligations attach to the latter. Licence terms are the single most common thing teams get wrong when they move from a prototype to a product; our post on the licensing trap walks through how to avoid it.

Wikidata: the structured facts

Wikidata is Wikipedia’s structured sibling. Where Wikipedia stores prose, Wikidata stores facts as machine-readable statements. Every entity, a person, place, book, chemical, or event, has a stable identifier, and facts about it are expressed as triples: a subject, a property, and a value. “Ada Lovelace, date of birth, 1815” is one such statement. Because the data is structured, you can query it precisely with SPARQL, the query language for graph data, and get back exact answers rather than passages you then have to parse.

This is the crucial difference in practice. Free text is good at nuance and context but poor at precise, list-shaped, or relational questions. Ask a text corpus “which novels did this author publish, and in what year?” and you are relying on a passage happening to contain all of it in one place. Ask Wikidata the same question as a graph query and you get a clean, complete list. Entity linking, the task of resolving a name in a user’s question to a specific real-world thing, is also far easier when you have stable identifiers to link to.

Wikidata is released under CC0, a public-domain dedication that waives essentially all rights. There are no attribution or share-alike obligations. That licence contrast is worth holding in your head: the Wikipedia text you retrieve carries CC BY-SA obligations, while the Wikidata facts you query carry none. If licence cleanliness matters to your product, the structured layer is the easier one to build on.

Use them together

The key insight is not to choose. The two sources answer different kinds of question, and a good general-knowledge RAG system uses both:

Wikipedia text for passages to quote, summarise, and ground generated prose in.
Wikidata facts for precise factual lookups, entity disambiguation, filtering, and anything list-shaped or relational.

A common pattern is to use Wikidata to resolve and disambiguate the entities in a question, then retrieve Wikipedia passages about those specific entities to feed the language model. The graph gives you precision and clean facts; the text gives you the readable, quotable context. If you are new to combining a retrieval corpus with structured lookups, the DIY RAG course builds the pieces up in order, and the glossary defines the terms as they come up.

Pre-built options that save work

You do not have to build everything from raw dumps. Two distributions in particular remove whole stages of work.

Pre-embedded Wikipedia. Cohere’s Wikipedia embeddings provide Wikipedia already chunked into passages and converted into vectors, across many languages. If you are happy to use the embedding model it was built with, you can load it straight into a vector database and skip the embedding step entirely, which is often the slowest and most compute-hungry part of a first build. Our lesson on creating embeddings explains what you would otherwise be doing yourself.
A tiny test set. rag-mini-wikipedia is a small question-and-passage set drawn from Wikipedia, meant for wiring up and sanity-checking a pipeline before you point it at a full corpus. It is not a corpus to ship, but it is ideal for confirming that retrieval, ranking, and generation all connect before you pay to embed millions of passages.

Neighbours worth knowing

Three related sources sit close by and are worth knowing about.

DBpedia extracts structured data from Wikipedia, chiefly from infoboxes, and turns it into a knowledge graph. It predates much of Wikidata’s growth and remains a useful structured view of Wikipedia content.
ConceptNet is a graph of commonsense relations between everyday concepts, things like “a dog is a kind of animal” or “coffee is used for staying awake”. It captures the sort of background knowledge that neither Wikipedia prose nor Wikidata’s factual statements express directly.
Wikimedia Commons holds the images, audio, and video used across Wikimedia projects, which matters if your system is multimodal rather than text-only.

Where to start

For a first general-knowledge build, a reasonable path is: prove the pipeline with rag-mini-wikipedia, load a cleaned or pre-embedded Wikipedia distribution for the text layer, and add Wikidata lookups when you need precise facts or entity linking. Attribute Wikipedia, note that Wikidata asks nothing of you, and keep the two licences straight in your head as you go. When you are ready to widen the corpus beyond encyclopaedic knowledge, browse the full directory, and if you spot a source we have missed, we would welcome a note on the contribute page.