1. Why This Matters
Historically, evaluating language models on coding tasks relied on benchmarks like HumanEval A widely used benchmark by OpenAI featuring 164 hand-written Python programming problems testing basic algorithmic tasks. . While useful, these benchmarks are highly contrived: they focus on writing isolated functions from scratch based on docstrings, typically requiring fewer than 10 lines of code.
In the real world, software engineering is vastly different. Engineers spend most of their time reading existing code, understanding complex system architectures, locating bugs across thousands of files, and writing precise, minimal adjustments (patches) that preserve existing functionality.
"Fixing a bug might involve navigating a large repository, understanding the interplay between functions in different files, or spotting a small error in convoluted code."
2. The Big Idea
SWE-bench bridges this gap by turning real-world open-source software maintenance into an automated benchmark. It leverages GitHub pull requests (PRs) from 12 popular Python repositories (such as django, scikit-learn, sympy, and matplotlib).
For each task, the model is given a GitHub issue description and the entire codebase at the base commit before the issue was resolved. The model must generate a codebase-wide edit in the form of a unified patch file A file containing instructions on how to modify existing text/code files line-by-line, generated by tools like diff. . This patch is applied and validated by running the repository's actual unit tests.
3. How It Works
Creating a robust, reproducible execution-based benchmark from raw GitHub data is extremely challenging. The authors designed a rigorous 3-stage pipeline to convert raw pull requests into high-quality, verified evaluation tasks:
Interactive Widget 1: The SWE-bench Construction Pipeline
Click on each stage to see how 90,000 raw pull requests are filtered down to 2,294 clean, verified software engineering tasks.
Scrape & Select
Collect PRs from 12 popular open-source Python repositories.
Attribute Filter
Filter for merged PRs that resolve issues and contribute new tests.
Execution Validation
Ensure environment installs and test status changes from Fail → Pass.
Stage I: Repo Selection and Data Scraping
The authors started by selecting 12 massive open-source repositories written in Python (e.g., Django, Matplotlib, Scikit-learn). They scraped all pull requests, yielding a raw pool of approximately 90,000 merged PRs.
4. Interactive Exploration
Interactive Widget 2: Context Length & Retrieval Strategy Simulator
LMs cannot fit entire 400k-line codebases into context, so they rely on document retrieval. Adjust context limits and retrieval modes to see the performance trade-offs.
Oracle context includes *only* the files modified by the developer. BM25 searches the codebase using the issue text.
The "Lost in the Middle" Effect
Notice how under BM25 retrieval, **increasing the context window actually decreases model performance**. Adding irrelevant files distracts the model, making it harder to localize the bug.
At 13k context using BM25, Claude 2 achieves its peak BM25 performance of 1.96%. However, BM25 only successfully retrieves the correct files (recall) 29.58% of the time.
Interactive Widget 3: Patch Generation & Test Runner Simulation
See a real task instance from sphinx-doc/sphinx. Toggle between the model-generated patch (which missed a config check) and the gold patch, then run the tests.
# Model's Attempted Fix (Fails Config Check Validation) def _parse_other_parameters_section(self, section): - fields = self._consume_fields() - return self._format_fields('Other Parameters', fields) + # Model hardcoded formatting as if napoleon_use_param is always True + return self._format_docutils_params(self._consume_fields())
The Bug
The Sphinx extension napoleon should format "Other Parameters" according to the napoleon_use_param config. The model incorrectly assumed it should always format it as True, failing when the config is set to False.
$ pytest tests/test_napoleon.py
Select a patch and click "Run Test Suite" to execute...
5. Benchmark Results
The overall performance of state-of-the-art models on SWE-bench is shockingly low. Even when using the best sparse retrieval methods, models struggle to resolve more than a tiny fraction of real-world issues.
| Model | Retrieval Setting | % Applied Successfully | % Resolved (Full) |
|---|---|---|---|
| Claude 3 Opus | BM25 (Sparse) | 46.56% | 3.79% |
| Claude 2 | BM25 (Sparse) | 43.07% | 1.97% |
| GPT-4 Turbo | BM25 (Sparse) | 26.90% | 1.31% |
| SWE-Llama 13b | BM25 (Sparse) | 53.62% | 0.70% |
| Claude 2 | Oracle (Cheat) | 62.82% | 4.80% |
6. Limitations & Open Questions
- Language Constraint: Currently, SWE-bench only supports Python codebases. Expanding to statically typed languages (like Rust, Go, or Java) is an open and crucial next step.
- Execution & Environment Flakiness: Building execution environments for older commits is difficult. Dependency changes, deprecated Python versions, and network flakiness can cause valid patches to fail execution.
- The "Oracle" Gap: The massive gap between BM25 and Oracle performance shows that *localization* (finding which files to edit) is almost as hard as writing the code itself.
7. Glossary
Gold Patch
The exact patch file created by the original human developer that successfully resolved the issue and was merged into the master repository.
BM25 Retrieval
A classic bag-of-words ranking function used by search engines to estimate the relevance of documents to a given search query (in this case, matching codebase files to the issue description).
Fail-to-Pass (F2P) Tests
Unit tests that fail on the original codebase but pass after applying the patch. These verify that the specific bug reported in the issue has been successfully resolved.
8. Citation
@inproceedings{jimenez2024swebench,
title={SWE-bench: Can Language Models Resolve Real-World GitHub Issues?},
author={Carlos E. Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik Narasimhan},
booktitle={International Conference on Learning Representations (ICLR)},
year={2024}
}