The Pile

The Pile was one of the first serious efforts to build a diverse open pretraining corpus, and its design still repays study. Instead of relying on web crawl alone, EleutherAI stitched together 22 separate sources into a single 825 GB dataset: academic papers from PubMed Central and arXiv, GitHub code, Stack Exchange question-and-answer threads, books, subtitles, patents, Wikipedia, and filtered web text. The bet was that a deliberate mix of domains produces more capable, better-rounded models than web data on its own.

It ships as JSON Lines, where each line is one document with a metadata field marking which sub-dataset it came from. That tagging is the practical key to using it: you can keep the academic and code portions for a technical assistant, drop the ones you do not want, or weight domains differently, all with a simple filter rather than reprocessing raw data.

The Pile trained the early open GPT-Neo, GPT-J, and GPT-NeoX models and remains a common research baseline, which makes it useful when you want a mixed-domain retrieval corpus whose composition is well documented and easy to compare against published work.

The licensing needs care. MIT covers EleutherAI's compilation, but each subset keeps its own terms, so commercial reuse is genuinely unclear across the whole set. The most-cited example is the Books3 subset, which was pulled after copyright complaints, so the official hosting is no longer complete and you may need community mirrors to reassemble parts of it. Check the terms of any subset before you redistribute or build a product on it.

If you want a similar domain mix with cleaner licensing, Dolma and RedPajama cover comparable ground with more modern filtering, but The Pile's clear sub-dataset boundaries still make it one of the easiest corpora to slice by domain.