OpenWebMath

OpenWebMath is an effort to gather most of the good mathematical writing on the open web into a single dataset. Starting from more than 200 billion HTML pages in Common Crawl, the team applied maths-aware extraction and filtering to keep the pages carrying genuine mathematical content, leaving 6.3 million documents worth roughly 14.7 billion tokens. Its standout feature is that it preserves equations and LaTeX notation, which ordinary web extractors tend to mangle into gibberish.

It lives on Hugging Face under open-web-math/open-web-math as Parquet files, so you can stream it with the datasets library or download the whole thing. The key handling tip for RAG is to keep the mathematical notation intact when you chunk and embed: split on document or section boundaries rather than through the middle of an equation, and bear in mind that many general-purpose embedding models handle LaTeX poorly, so a maths-aware or code-aware embedding model will retrieve noticeably better.

It is a foundational choice for maths and reasoning work, whether you are continuing the pretraining of a model or building retrieval over worked explanations, proofs, and problem discussions. The substantial physics, computer science, and statistics content means it doubles as a broad technical-reasoning corpus rather than a narrow maths-only one.

Because it comes from the web, quality varies: forum threads, homework pages, and half-finished explanations sit next to polished material, and nothing here is proof-checked or verified. It is English-centric, and it is large enough to need real storage. On licensing it is ODC-By 1.0, which permits commercial use but requires attribution; the text was drawn from Common Crawl, so the usual caution about the underlying page terms still applies.

It is a common ingredient in larger maths corpora such as Proof-Pile-2 and AutoMathText, and it pairs well with arXiv for formal papers. Start here, then layer those on for coverage.