TxT360

TxT360 (Trillion eXtracted Text) is a large-scale pre-training dataset from the LLM360 open-model initiative. It combines 99 Common Crawl snapshots with 14 curated, high-quality domains, including legal text from FreeLaw, books from PG-19, Wikipedia, and scientific papers. Its distinguishing feature is global deduplication: rather than deduplicating each snapshot in isolation, the pipeline removes exact duplicates with a Bloom filter (dropping around 17% of input documents) and near-duplicates with MinHash-based fuzzy matching across the whole corpus. The deduplicated base is roughly 5 trillion tokens, which the maintainers upsample with a documented weighting recipe to reach the headline 15 trillion-plus tokens.

Access is through the Hugging Face dataset card, where the data is distributed as gzip-compressed JSONL shards totalling around 23.9 TB. The companion GitHub repository publishes the full processing code and methodology, so the corpus can be audited, re-weighted, or regenerated rather than treated as an opaque blob. This is a raw-to-cleaned pre-training resource, not a retrieval-ready index.

For AI work it is best suited to pre-training or continued pre-training of language models, and to research on data curation, deduplication, and mixing strategies. For RAG specifically it is more of a raw ingredient than a finished knowledge base: you would filter to a domain, then chunk and embed the subset you need. Watch the licence carefully. The compilation is released under ODC-By 1.0, which permits commercial use with attribution, but the underlying sources keep their own terms, so Common Crawl content and curated domains carry provenance obligations of their own. If you want a comparably documented open alternative, FineWeb and RedPajama cover similar ground.