Headline finding
The authors introduce a novel, automated, and scalable pipeline to continuously extract real-world interactive SWE tasks from diverse GitHub repositories, constructing SWE-rebench, a public dataset of 21,336 verifiable Python-based SWE tasks from 3,468 distinct repositories, as well as a continuously updated, decontaminated benchmark of 294 tasks from 169 repositories.
The manuscript thoroughly documents the four-stage automated pipeline, provides concrete funnel metrics, and reports the exact counts of 21,336 tasks (3,468 repos) for the dataset and 294 tasks (169 repos) for the benchmark subset.
“To address these limitations, we introduce a novel, automated, and scalable pipeline to continuously extract real-world interactive SWE tasks from diverse GitHub repositories. Using this pipeline, we construct SWE-rebench, a public dataset comprising over 21,000 interactive Python-based SWE tasks, suitable for reinforcement learning of SWE agents at scale.” · Show in paper