By Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik NarasimhanSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2310-06770.git
Cite
Version link
https://openversions.com/index/arxiv-2310-06770/commit/ee859535b02d7cfc9c99660edfda21e88e39bf1c
Version identifier (commit hash)
ee859535b02d7cfc9c99660edfda21e88e39bf1c

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Claims

Findings

Headline finding

State-of-the-art proprietary language models and fine-tuned SWE-Llama models can resolve only the simplest issues on SWE-bench, with Claude 2 solving 1.96% of the issues using a BM25 retriever.

SWE-bench is a continually updatable evaluation framework of 2,294 real-world software engineering problems drawn from GitHub issues and pull requests across 12 popular Python repositories.

In some settings, SWE-Llama 13b is competitive with Claude 2 and is capable of processing contexts exceeding 100,000 tokens.

Model performance drops considerably as total context length increases, because models struggle to localize problematic code needing to be updated.

Collapsing unedited lines in the 'oracle' retrieval context (leaving +/-15 lines of buffer) increases model performance, with GPT-4 jumping from 1.3% to 3.4% and Claude 2 from 4.8% to 5.9%.

Difficulty does not correlate with issue resolution date for most models in the 'oracle' retrieval setting, suggesting models are unlikely to 'cheat' by generating a more recent version of the repository.

Generating patch files is easier for language models than regenerating whole files with their proposed changes.

Language models tend to generate shorter, simpler edits than gold patches, with correctly applied model patches being less than half the total length of all gold patches and rarely editing more than a single file.

Finetuned models SWE-Llama 7b and 13b perform poorly with BM25 retrieved context because they are sensitive to context distribution shifts from their 'oracle' fine-tuning setup.

Methods

Three-stage benchmark construction pipeline: scraping ~90,000 pull requests from 12 popular Python repositories, filtering for merged PRs that resolve an issue and modify test files, and execution-based filtering requiring at least one fail-to-pass test and no installation or runtime errors.

Source excerpts

“To find high-quality task instances at scale, we use a 3-stage pipeline as follows. Stage I: Repo selection and data scraping. We start by collecting pull requests (PRs) from 12 popular open-source Python repositories on GitHub, producing about ~ 90,000 PRs in total. ... Stage II: Attribute-based filtering. We create candidate tasks by selecting the merged PRs that (1) resolve a GitHub issue and (2) make changes to the test files of the repository... Stage III: Execution-based filtering. For each candidate task, we apply the PR’s test content, and log the associated test results before and after the PR’s other content is applied. We filter out task instances without at least one test where its status changes from a fail to pass (henceforth referred to as fail-to-pass test). We also filter out instances that result in installation or runtime errors.” · Show in paper

Execution-based evaluation procedure: applying the model-generated patch to the base codebase using unix's patch program (with automated patch repair if needed) and running unit and system tests; a task is resolved if all FAIL_TO_PASS and PASS_TO_PASS tests pass.

Source excerpts

“To evaluate a proposed solution, we apply the generated patch, using unix’s patch program, to the codebase and then execute the unit and system tests associated with the task instance. If the patch applies successfully and all of these tests pass we consider the proposed solution to have successfully resolved the issue. The metric for our benchmark is the percentage of task instances that are resolved.” · Show in paper

“As defined and used in the main paper, a task is considered solved if all tests across FAIL TO PASS and PASS TO PASS pass.” · Show in paper

Supervised fine-tuning of SWE-Llama 7b and 13b from CodeLlama-Python using LoRA (r=16, alpha=16, dropout=0.05) on attention sublayers over 10,000 training instances (sequences <= 30,000 tokens) derived from 19,000 issue-PR pairs across 37 disjoint Python repositories.

Source excerpts

“We follow our data collection procedure and collect 19,000 issue-PR pairs from an additional 37 popular Python package repositories. In contrast to Section 2.1, we do not require that pull requests contribute test changes. ... To eliminate the risk of data contamination, the set of repositories in the training data is disjoint from those included in the evaluation benchmark. ... For memory efficiency, we fine-tune only the weights of the attention sublayer using LoRA Hu et al. (2022), and exclude training sequences with more than 30,000 tokens, reducing the effective size of the training corpus to 10,000 instances.” · Show in paper

“We finetune using LoRA (Hu et al., 2022) with r = 16, α = 16, dropout = 0.05, on the query, key, value, and output projection matrices of every attention sublayer.” · Show in paper

Context retrieval baselines using BM25 sparse retrieval (with file paths prepended to documents) under 13k, 27k, and 50k token limits, and 'oracle' retrieval using the files edited by the reference patch.

Source excerpts

“Therefore, we choose to use BM25 retrieval (Robertson et al., 2009) to retrieve relevant files to provide as context for each task instance. We experiment with three different maximum context limits, and simply retrieve as many files as fits within the specified limit. ... For analysis purposes we also consider a setting where we “retrieve” the files edited by the reference patch that solved the issue on GitHub.” · Show in paper

“During retrieval we make a slight augmentation to the documents by pre-pended files’ contents with their file paths to better enable retrieval based on filenames that may be mentioned directly in the issue.” · Show in paper