RefineCode (OpenCoder)

RefineCode is the code pretraining corpus assembled for OpenCoder, an open effort to publish not just model weights but the full recipe behind a competitive code language model. The corpus reported in the OpenCoder paper covers roughly 960 billion tokens across around 607 programming languages, drawn from raw source code plus a further body of code-related tokens recalled from web data. Its selling point is reproducibility: the pipeline applies over 130 language-specific filtering rules with tuned weightings, and the authors document each step so others can rebuild or adapt the corpus rather than trust an opaque blob.

Access has an important catch. The Hugging Face release (RefineCode-code-corpus-meta) currently publishes metadata only, chiefly the repository name and file path for each item, covering roughly the half of the corpus that overlaps The Stack v2. The maintainers cite legal considerations for not shipping the complete source code directly. In practice you download the metadata as Parquet and then fetch the referenced files yourself to reconstruct the training set, following the OpenCoder cookbook.

For RAG or AI work this is best treated as a recipe and an index rather than a ready-to-embed dataset. It suits teams pretraining or fine-tuning code models, or anyone who wants a documented, filtered alternative to raw crawls for building a code knowledge base. The main watch-outs are the assembly step (you must gather the actual files) and licensing: the metadata dataset is released under MIT, but the underlying source files carry their own, heterogeneous open-source licences that you are responsible for honouring. If you need the code contents in one download, The Stack v2 (BigCode) is the obvious companion, and for developer Q and A the Stack Exchange data dumps complement code corpora well.