Skip to content
RAG Repo

Wikimedia publishes complete database dumps of Wikipedia and its sister projects (Wiktionary, Wikiquote, Wikisource, and the rest) at dumps.wikimedia.org, refreshed roughly twice a month. You can take a single language edition or every one of the nearly 300 languages, and choose raw wikitext, parsed HTML, or the SQL tables behind the site. The compressed English article text is around 22 GB.

The quickest route for RAG is to skip the dump parsing entirely and pull a pre-processed, one-article-per-record version from the Hugging Face Hub, where the wikitext markup has already been stripped back to clean prose. If you do work from the raw XML, budget time for a parser: wikitext is fiddly, with templates, infoboxes, and citation markup that all need handling before you get usable text.

Wikipedia is the natural first corpus for almost any general-knowledge RAG system. It is broad, consistently structured, cleanly licensed, and its article sections chunk neatly into passages. It is an excellent way to stand up and test a retrieval pipeline before you move on to messier, domain-specific data.

Two things to watch. First, currency: a dump is a snapshot, so anything edited after it was taken will be missing, which matters for fast-moving topics. Second, the licence. The text is Creative Commons Attribution-ShareAlike (CC BY-SA 4.0), which means commercial use is fine but you must credit Wikipedia and share any derived text under the same terms. Some embedded media and older content carry different or additional licences, so check before redistributing.

For structured facts rather than prose, pair Wikipedia with Wikidata, its sister project, which stores the same knowledge as machine-readable statements and stable identifiers you can use to ground and disambiguate the entities your documents mention.

encyclopaediamultilingualgeneral-knowledgenonprofitbulk-download

Related sources