Zyda-2

Zyda-2 is a five trillion token English pretraining mixture built by Zyphra, the AI research company behind the Zamba2 model series. It aggregates four existing open corpora: DCLM-baseline, FineWeb-Edu (the score-2 slice), Zyda-1 and the Common Crawl portion of Dolma v1.7. Rather than simply concatenating them, Zyphra cross-deduplicated the four sources against one another (so a document appearing in more than one corpus is not stored and trained on twice) and applied model-based quality filtering, using Nvidia's NeMo quality classifier on the Zyda-1 and Dolma portions. The result is published on HuggingFace as Parquet, roughly 13 TB in size, and streams directly with the datasets library so you can pull individual shards without downloading the whole thing.

For RAG or fine-tuning, Zyda-2 is best treated as a ready high-quality base rather than raw material. Because the deduplication and filtering are already done, you can move straight to chunking and embedding passages, or use it as pretraining and continued-pretraining data for a domain model. Zyphra reports that models trained on Zyda-2 outperform identical models trained on the Pile, RefinedWeb, FineWeb, FineWeb-Edu and DCLM, and an early version was the primary phase-one corpus for Zamba2. It is English only and web-derived, so it suits broad-coverage knowledge bases rather than specialist or multilingual needs.

The dataset is released under ODC-BY 1.0, which permits commercial use provided you attribute Zyphra. The important caveat is that Zyda-2 is a compilation: you remain bound by the licences and terms of use of every upstream source, and the web text carries Common Crawl's terms, so treat it as filtered web content rather than material cleared for any purpose. Compared with the sibling sources we list, it overlaps heavily with FineWeb-Edu and DCLM by design; its value is the cross-deduplication and reweighting that combine them into one cleaner, denser mixture.