Essential-Web v1.0

Essential-Web v1.0 is a 24-trillion-token corpus of roughly 23.6 billion documents, built by Essential AI and released in 2025 from 101 Common Crawl snapshots (CC-MAIN-2013-20 through CC-MAIN-2024-38). What sets it apart from a plain crawl rebuild is the metadata: every document carries a twelve-category taxonomy spanning topic, page format, content complexity and several quality dimensions (including reasoning depth, technical correctness and education level), alongside a Free Decimal Correspondence subject code and Bloom's taxonomy labels. These annotations were produced by EAI-Distill-0.5b, a small fine-tuned classifier the team reports as agreeing with a much larger reference model to within a few per cent.

The data lives on HuggingFace as Parquet (about 75.3 TB in total) and is also mirrored on the AWS Registry of Open Data, so you can stream shards with the datasets library or pull from S3 rather than downloading everything. Because the labels sit in columns beside the text, you filter with ordinary SQL-style predicates: select the medical pages above a quality threshold, or the code and STEM documents, and build a focused subset in minutes. Essential AI reports that such filtered slices are competitive with purpose-built curated sets in maths, web code, STEM and medical domains.

For RAG and fine-tuning this is most useful when you want a domain or quality-controlled corpus but do not want to run your own classification pipeline over trillions of tokens. Compared with FineWeb or C4, which ship a single pre-filtered view, Essential-Web hands you the labels and lets you decide the cut, which is the main reason to choose it.

The compilation is released under ODC-BY 1.0, so commercial use is permitted with attribution to the dataset. Two caveats apply: the Common Crawl Terms of Use also govern the data, and the underlying web pages retain their own copyright, so treat the text as filtered web content rather than material cleared for any purpose.