Proof-Pile-2
Proof-Pile-2 pulls together three complementary sources into a single maths pretraining corpus: arXiv papers, the web text of OpenWebMath, and AlgebraicStack, a code-heavy collection focused on mathematical computation. Together they total roughly 55 billion tokens.
AlgebraicStack is the distinctive part. It contributes about 11 billion tokens covering numerical computing, computer algebra systems, and formal mathematics (machine-checkable proofs), which makes the corpus useful for models that need to reason about symbolic maths and code, not just prose.
The dataset is hosted on Hugging Face by EleutherAI and was assembled for training the Llemma family of maths language models. Licensing varies by subset, so check the terms of each component before building on it commercially.
Related sources
AMPS
A dataset of informal mathematics introduced alongside the MATH benchmark. It includes more than 100,000 Khan Academy problems with step-by-step solutions in LaTeX and over 5 million problems generated with Mathematica scripts, totalling around 23 GB.
AutoMathText
Around 200 GB of mathematical text compiled from websites, arXiv, and GitHub, drawing on OpenWebMath, RedPajama, and AlgebraicStack. Every piece of content carries a score from 0 to 1 for relevance, quality, and educational value, labelled automatically by the Qwen-72B model.
Awesome AI Math Datasets
A community-curated list of open-source mathematics datasets for training and evaluating maths-capable language models. A useful index for finding newer additions in this space.
MegaMath
An open mathematics pretraining dataset curated from diverse, maths-focused sources, with over 300 billion tokens. Among the largest open maths corpora available.