Headline finding
State-of-the-art proprietary language models and fine-tuned SWE-Llama models can resolve only the simplest issues on SWE-bench, with Claude 2 solving 1.96% of the issues using a BM25 retriever.
The experimental results in Tables 2 and 5 support the overarching finding that all evaluated models resolve only a tiny fraction of SWE-bench issues (under 2% for Claude 2, and 3.79% for Claude 3 Opus in the updated Table 5).
“Our evaluations show that both state-of-the-art proprietary models and our fine-tuned model SWE-Llama can resolve only the simplest issues. The best-performing model, Claude 2, is able to solve a mere 1.96% of the issues.” · Show in paper