GitHub Public Repositories (GH Archive / GHTorrent)

These two projects capture the activity around public GitHub rather than the code itself. GH Archive records every public event on GitHub (pushes, forks, stars, issue comments, pull request activity, and more) and packages them into hourly files you can download as gzipped JSON. The record goes back to 2011, and because the whole timeline is also published as a public BigQuery dataset, you can run large queries across years of activity without downloading and parsing the archives yourself.

GHTorrent (https://ghtorrent.org) took a different angle, offering a relational mirror of GitHub's metadata (users, repositories, commits, issues, and the links between them) so you could query relationships rather than replay a stream of events. It is worth knowing that GHTorrent is no longer actively updated, so treat it as a historical snapshot; for current data, GH Archive plus the GitHub API is the more reliable pairing.

For RAG, the value here is the natural language wrapped around software: issue threads, pull request discussions, commit messages, and review comments. That context pairs well with a code corpus, letting a code assistant retrieve not just an implementation but the discussion of a bug, a design decision, or how an interface is meant to be used.

Two things to watch. The raw event stream is huge and repetitive, so you will want to filter to the event types and repositories you care about before extracting text; BigQuery makes that far cheaper than local processing. On licensing, the data sits under GitHub's terms of service and each repository's own licence, and commercial reuse of user-generated content is genuinely unclear, so this is safer for research, analysis, and internal tooling than for a redistributed dataset.

Alongside The Stack, which gives you the source files under known licences, GH Archive supplies the human conversation that explains why that code looks the way it does.