MegaMath
MegaMath is an open pretraining corpus built specifically for mathematics. It draws on a range of maths-focused sources, including filtered web text, code, and synthetically generated material, and totals more than 300 billion tokens.
That scale puts it among the largest open mathematics corpora, several times bigger than earlier collections such as OpenWebMath and Proof-Pile-2. It is aimed at training and continued pretraining of maths-capable language models, but the same breadth makes it a rich knowledge base for maths and reasoning RAG.
The dataset is released on Hugging Face by the LLM360 project under ODC-By 1.0, a permissive open-data licence that allows commercial use with attribution.
Related sources
AMPS
A dataset of informal mathematics introduced alongside the MATH benchmark. It includes more than 100,000 Khan Academy problems with step-by-step solutions in LaTeX and over 5 million problems generated with Mathematica scripts, totalling around 23 GB.
AutoMathText
Around 200 GB of mathematical text compiled from websites, arXiv, and GitHub, drawing on OpenWebMath, RedPajama, and AlgebraicStack. Every piece of content carries a score from 0 to 1 for relevance, quality, and educational value, labelled automatically by the Qwen-72B model.
Awesome AI Math Datasets
A community-curated list of open-source mathematics datasets for training and evaluating maths-capable language models. A useful index for finding newer additions in this space.
NaturalProofs
A dataset of 32,000 theorem statements and proofs, 14,000 definitions, and 2,000 other pages including axioms and corollaries, drawn from ProofWiki, the Stacks Project, and mathematics textbooks.