MegaMath

MegaMath is a pretraining corpus built specifically for mathematics, and at more than 300 billion tokens it is among the largest open maths collections you can get. It draws on three kinds of source: maths-heavy web pages filtered out of Common Crawl, mathematics-related code, and synthetically generated problems and explanations. Together they cover a wide span, from school arithmetic through to research-level notation and proof.

The corpus is published on Hugging Face by the LLM360 project in Parquet, the columnar format the datasets library reads efficiently. At this size streaming is the sensible default: you can iterate over the data without downloading the whole thing, which matters because the full set needs serious storage. The web, code, and synthetic subsets are kept separate, so you can take just the slice you need rather than the lot.

It is designed for training and continued pretraining of maths-capable models, but that same breadth makes it a rich base for mathematics and reasoning RAG, where you want wide coverage of worked examples, definitions, and notation to retrieve against.

The trade-offs are the ones you would expect from raw pretraining material. It is not a tidy set of question-answer pairs, so it is noisy and repetitive in places and rewards further filtering when you only need a focused slice. The synthetic portion is machine-generated, so spot-check it before you trust it for factual answers. MegaMath is released under ODC-By 1.0, a permissive open-data licence that allows commercial use provided you attribute the source.

If MegaMath is more than you can handle, OpenWebMath and Proof-Pile-2 are smaller and easier to work with. Reach for MegaMath when sheer coverage matters more than convenience.