AutoMathText
AutoMathText gathers roughly 200 GB of mathematical text from three kinds of source: general websites, arXiv papers, and GitHub code, building on the earlier OpenWebMath, RedPajama, and AlgebraicStack collections.
What sets it apart is the scoring. Each document is assigned a value between 0 and 1 reflecting its relevance, quality, and educational value, labelled autonomously by the Qwen-72B language model rather than by hand. That lets you filter the corpus to whatever quality threshold suits your use case, keeping only the highest-scoring material or casting a wider net.
The dataset is published on Hugging Face under CC BY-SA 4.0, so any derivative database you share must carry the same licence.
Related sources
AMPS
A dataset of informal mathematics introduced alongside the MATH benchmark. It includes more than 100,000 Khan Academy problems with step-by-step solutions in LaTeX and over 5 million problems generated with Mathematica scripts, totalling around 23 GB.
Awesome AI Math Datasets
A community-curated list of open-source mathematics datasets for training and evaluating maths-capable language models. A useful index for finding newer additions in this space.
MegaMath
An open mathematics pretraining dataset curated from diverse, maths-focused sources, with over 300 billion tokens. Among the largest open maths corpora available.
NaturalProofs
A dataset of 32,000 theorem statements and proofs, 14,000 definitions, and 2,000 other pages including axioms and corollaries, drawn from ProofWiki, the Stacks Project, and mathematics textbooks.