OpenWebMath
OpenWebMath sets out to capture the bulk of the high-quality mathematical text on the open web. The team started from more than 200 billion HTML files in the Common Crawl archive and applied maths-aware extraction and filtering to keep the pages that carry real mathematical content, preserving equations and notation that generic web extractors tend to mangle.
The result is 6.3 million documents worth roughly 14.7 billion tokens. While the majority of the content is mathematics, a large share covers related technical subjects, including physics, computer science, and statistics, so it works well as a broad reasoning corpus rather than a narrow one.
The dataset is published on Hugging Face and is a common ingredient in larger maths corpora such as Proof-Pile-2 and AutoMathText.
Related sources
AMPS
A dataset of informal mathematics introduced alongside the MATH benchmark. It includes more than 100,000 Khan Academy problems with step-by-step solutions in LaTeX and over 5 million problems generated with Mathematica scripts, totalling around 23 GB.
AutoMathText
Around 200 GB of mathematical text compiled from websites, arXiv, and GitHub, drawing on OpenWebMath, RedPajama, and AlgebraicStack. Every piece of content carries a score from 0 to 1 for relevance, quality, and educational value, labelled automatically by the Qwen-72B model.
Awesome AI Math Datasets
A community-curated list of open-source mathematics datasets for training and evaluating maths-capable language models. A useful index for finding newer additions in this space.
MegaMath
An open mathematics pretraining dataset curated from diverse, maths-focused sources, with over 300 billion tokens. Among the largest open maths corpora available.