AMPS
AMPS (Auxiliary Mathematics Problems and Solutions) was released alongside the MATH benchmark as pretraining material for maths problem solving. It pairs two kinds of content: more than 100,000 problems from Khan Academy with worked, step-by-step solutions written in LaTeX, and over 5 million problems generated programmatically with Mathematica scripts.
Together they add up to roughly 23 GB. The Khan Academy portion gives natural, human-written explanations, while the Mathematica portion provides scale and coverage across many problem types, from algebra to calculus.
The dataset ships from the MATH benchmark repository on GitHub under the MIT licence. The step-by-step LaTeX solutions make it a good source for RAG systems that need to retrieve worked examples rather than bare answers.
Related sources
AutoMathText
Around 200 GB of mathematical text compiled from websites, arXiv, and GitHub, drawing on OpenWebMath, RedPajama, and AlgebraicStack. Every piece of content carries a score from 0 to 1 for relevance, quality, and educational value, labelled automatically by the Qwen-72B model.
Awesome AI Math Datasets
A community-curated list of open-source mathematics datasets for training and evaluating maths-capable language models. A useful index for finding newer additions in this space.
MegaMath
An open mathematics pretraining dataset curated from diverse, maths-focused sources, with over 300 billion tokens. Among the largest open maths corpora available.
NaturalProofs
A dataset of 32,000 theorem statements and proofs, 14,000 definitions, and 2,000 other pages including axioms and corollaries, drawn from ProofWiki, the Stacks Project, and mathematics textbooks.