Specialist corpora of mathematical and technical text, including proofs, problem sets, and worked solutions, for RAG systems that handle quantitative reasoning.
9sources
Mathematical and reasoning corpora are specialist collections of proofs, problem sets, and worked solutions, and they matter because general web text teaches a model very little about how to reason quantitatively. If your RAG system needs to explain a derivation, retrieve a relevant theorem, or check the steps in a solution, generic knowledge bases will let you down. This kind of data carries the structured, step-by-step thinking that maths and technical work depend on, which is exactly what most corpora strip away as noise.
The main choice is between naturally occurring maths and synthetic maths. Sources like OpenWebMath and Proof-Pile-2 gather real writing from the web, textbooks, and formal proof libraries, so the language is authentic but uneven in difficulty. Sets like AMPS and AutoMathText lean on generated or auto-labelled problems, which gives you clean, plentiful examples at the cost of some variety and the occasional machine-made mistake. Think about whether you want breadth of style or predictable, well-labelled structure.
Watch out for how the maths is encoded. Some datasets keep formulae as LaTeX (a typesetting language for equations), others as plain text or images, and a formula that renders correctly in one format can turn to gibberish in another, so check that your chunking and embedding steps preserve the symbols. Correctness is the other worry: not every solution in a scraped set is right, and formal proof libraries, though rigorous, cover only the theorems someone has bothered to formalise.
The sources below span raw web maths, formal proof collections, and curated problem-and-solution sets.
A dataset of informal mathematics introduced alongside the MATH benchmark. It includes more than 100,000 Khan Academy problems with step-by-step solutions in LaTeX and over 5 million problems generated with Mathematica scripts, totalling around 23 GB.
Around 200 GB of mathematical text compiled from websites, arXiv, and GitHub, drawing on OpenWebMath, RedPajama, and AlgebraicStack. Every piece of content carries a score from 0 to 1 for relevance, quality, and educational value, labelled automatically by the Qwen-72B model.
A community-curated list of open-source mathematics datasets for training and evaluating maths-capable language models. A useful index for finding newer additions in this space.
An open mathematics pretraining dataset curated from diverse, maths-focused sources, with over 300 billion tokens. Among the largest open maths corpora available.
A dataset of 32,000 theorem statements and proofs, 14,000 definitions, and 2,000 other pages including axioms and corollaries, drawn from ProofWiki, the Stacks Project, and mathematics textbooks.
An open reasoning dataset of 1.2 million rows (roughly 850k maths, 250k code and 100k science questions) each paired with a chain-of-thought answer. Released by the Open Thoughts team in June 2025, it is the training set behind OpenThinker3 and is licensed Apache 2.0.
A corpus of high-quality mathematical text extracted from the web. Filtered from over 200 billion HTML pages on Common Crawl down to 6.3 million documents totalling 14.7 billion tokens. Mostly mathematics, with substantial coverage of physics, computer science, and statistics.
A 55 billion token dataset of mathematical and scientific documents drawn from arXiv, OpenWebMath, and AlgebraicStack. The AlgebraicStack subset alone contributes 11 billion tokens covering numerical computing, computer algebra, and formal mathematics.
A curated collection of 2 million mathematical questions and answers sourced from various Stack Exchange sites. The question-and-answer structure maps naturally onto retrieval, which makes it well suited to Q&A-style RAG.