The Stack v1 / v2

The Stack is BigCode's open dataset of source code. Version 1 collects files scraped from GitHub and keeps only those under permissive licences such as MIT, Apache, and BSD, with duplicate and near-duplicate files removed so you are not storing the same snippet many times over. Version 2 is a large step up in scale: it draws from Software Heritage, a long-term public archive of code, and spans more than 600 programming languages.

Both versions live on HuggingFace as Parquet files. A practical difference to plan for: v1 includes the file contents directly, whereas v2's main tables hold metadata and pointers, and you fetch the actual code blobs from Software Heritage's S3 bucket. So for a quick start v1 is simpler to stream with the `datasets` library, while v2 rewards a bit more pipeline setup in exchange for far broader coverage.

For RAG, The Stack is a strong retrieval corpus for code assistants, letting you index real-world implementations across many languages, and it is also the pretraining data behind code models such as StarCoder. Because every file is tracked back to its repository licence, it is a much safer starting point than a raw GitHub scrape when you care about the terms attached to the code you index and ship.

The caveats are mostly about provenance and consent. Filtering is by repository licence, so the collection permits commercial use in principle, but individual files keep their own permissive terms, several of which still require you to preserve attribution notices in any redistributed code. BigCode also honours an opt-out: developers can ask for their code to be excluded, so a given snapshot is not a complete mirror of public code.

If you want the discussion around code rather than the code itself, GH Archive covers issues and pull requests, while Stack Exchange covers question-and-answer threads; The Stack is the piece that gives you the source files.