Institutional Books 1.0
Institutional Books 1.0 is a large text corpus drawn from books digitised through Harvard Library's participation in the Google Books project, which began in 2006. The release covers about 983,004 volumes identified as public domain, totalling roughly 242 billion tokens across nearly 400 million pages, with 254 distinct volume-level languages detected. Subject matter is broad, taking in literature, law, philosophy, history and science, which makes it useful well beyond any single domain.
Access is through Hugging Face, where the data is published as Parquet files. Each record carries the raw OCR text (available for every volume) alongside post-processed, cleaned text for roughly 850,000 volumes in English, German, French, Italian and Spanish, plus bibliographic, source and generated metadata. You can stream it, pull specific language subsets, or bulk download for local processing. The paired raw and cleaned layers are handy for comparing OCR quality or choosing how much noise you tolerate.
For RAG and AI work it suits long-horizon, historical and multilingual retrieval, plus pretraining and fine-tuning where documented provenance matters. The cleaned layer chunks more cleanly into passages; the raw layer preserves page structure.
Watch-outs are real. This is an early-access release under noncommercial terms: commercial use, including AI model training by or for commercial actors, requires you to contact the Institutional Data Initiative first. The underlying works are public domain, but the compilation's terms are more restrictive, and copyright determinations can occasionally be wrong, so independent legal assessment sits with you. OCR quality also varies by language and era.
If you need broadly similar public-domain book text, Project Gutenberg offers cleaner but smaller English-first coverage, while HathiTrust provides a far larger catalogue with its own access constraints.
Related sources
Google Books Ngrams
Word and phrase frequency data drawn from Google's digitised book collection, spanning centuries of published text. It counts how often words and short phrases appear by year, which is useful for historical language analysis and time-aware RAG features.
LibriVox
A volunteer project providing free public domain audiobooks. Volunteers record themselves reading works whose copyright has expired, which makes it a handy source of speech data for training speech-to-text models and building multimodal RAG.
Open Library
An Internet Archive project building a web page for every book ever published. It holds more than 20 million catalogue records and lends many titles digitally, making it a rich source of book metadata for RAG systems.
Project Gutenberg
A volunteer effort to digitise and archive public domain books, with more than 70,000 free ebooks. Mostly English, but it covers many languages. One of the oldest digital library projects, so it is a clean, permissively licensed source of full-text literature for RAG.