SWE-bench Verified

SWE-bench Verified is the de facto standard benchmark for coding agents. It was assembled by the SWE-bench team at Princeton together with OpenAI, who screened the larger SWE-bench test set and kept a human-validated subset of 500 tasks that reviewers judged to be well specified and solvable. Each task is a real GitHub issue drawn from one of twelve popular open-source Python projects, paired with the base commit, a gold patch, the project test suite, and the specific tests that must move from failing to passing. A model under evaluation reads only the issue and the repository, proposes a code patch, and is scored by whether that patch makes the hidden tests pass inside a Docker harness.

You access the task data from Hugging Face as a single Parquet file of around 2.1 megabytes, with thirteen columns covering the repository, the issue text, the patches, the test identifiers, and a difficulty label. The evaluation harness that actually runs the patches lives in a separate GitHub repository and installs as a Python package. The data itself is small, so this is an evaluation set, not training material.

For anyone building a coding agent or a retrieval-augmented development tool, SWE-bench Verified is the reference gate, measuring end-to-end issue resolution rather than isolated code completion, and most agent releases quote a Verified score. Two caveats matter. The tasks are Python-only and skew towards familiar open-source repositories, and Epoch AI rates contamination risk as high, with independent work suggesting models may have seen many of these tasks in training; OpenAI has since flagged flawed tests in part of the set. The harness and code carry an MIT licence, but each task embeds code and text from its upstream GitHub repository, so content licences differ per task and commercial reuse should be checked against each source project. It sits alongside our BFCL and ToolBench entries, which test tool calling rather than full issue resolution.