Google Books Ngrams

Google Books Ngrams takes one of the largest digitised book collections ever scanned and reduces it to something far smaller and more focused: frequency counts. For every n-gram, a run of one to five consecutive words, it records how many times that string appeared in print in a given year, going back several centuries across English and a number of other languages. The full set of tab-separated (`TSV`) files comes to roughly 2.2 TB.

There are two practical ways in. The viewer at books.google.com/ngrams is ideal for quick exploration and the familiar frequency-over-time curves. For real work you download the bulk files, which are split by n-gram length and opening letters, so you can take only the slices you need instead of the whole 2.2 TB. Every row carries the n-gram, the year, a match count, and a volume count, which loads neatly into a columnar engine like DuckDB or a plain database.

The key thing to grasp is that this is aggregate statistics, not book text. You cannot pull a passage out of it, so it is a signal layer rather than a retrieval corpus. Use it to date when a term entered common use, flag anachronistic or archaic vocabulary, weight how common a phrase is, or add time-aware features to a system answering questions about how language and ideas have shifted.

Licensing is friendly: the data is released under CC BY 3.0, so commercial use is fine as long as you credit Google Books Ngrams. Keep the well-known caveats in view, though. Optical character recognition mistakes inflate counts in older scans, the mix of books changes across decades (scientific titles surge through the twentieth century), and pre-1800 coverage is thin and noisy.

If you need the actual text of old books to retrieve and quote, reach for Project Gutenberg or HathiTrust; keep Ngrams for the quantitative, linguistic side of a project.