MixtureVitae

MixtureVitae is an open web-scale pretraining dataset built by Ontocord.AI to give model builders a large corpus with a clearer legal footing than raw web scrapes. It totals about 422 billion tokens across roughly 60 million rows, and is deliberately assembled from permissive-first sources: explicitly open-licensed and public-domain material, curated permissive repositories, and civic or government works, alongside a substantial body of synthetic instruction, maths, and reasoning data.

The sources are catalogued into three provenance tiers. Tier 1 (around 352 billion tokens) is open-licensed and public-domain text. Tier 2 (around 52 billion tokens) is curated permissive repositories, including code collections such as The Stack. Tier 3 (around 18 billion tokens) is civic and government material. This tiering, together with per-shard metadata, lets you include or exclude parts of the mixture according to your own risk appetite.

Access is straightforward: the data is published on Hugging Face as JSON (auto-converted to Parquet), so you can stream it or bulk-download shards, then chunk and embed the passages you need. It suits pretraining and continued pretraining, and the large instruction and reasoning portion makes it useful for building RAG knowledge bases or fine-tuning sets where provenance matters more than sheer web coverage.

Watch-outs are important. There is no single compilation licence: each constituent carries its own terms, so attribution and other obligations vary by source and you must check the tier and shard you actually use. The maintainers are candid that it is not free of legal risk, that licence heuristics can miss edge cases, and that cross-dataset deduplication is incomplete. If you want comparably permissive alternatives already common in directories, look at Common Pile and FineWeb.