Headline finding
SWE-BENCH PRO is a contamination-resistant benchmark of 1,865 human-verified, enterprise-level software engineering problems across 41 repositories, on which widely used coding models under a unified scaffold achieve below 45% Pass@1 performance.
The dataset counts (1,865 problems across 41 repositories) and the <45% Pass@1 performance ceiling (max 43.6% in Table 1 and 17.8% in Table 2) are directly supported by the reported tables and dataset breakdown.
“SWE-BENCH PRO contains 1,865 problems sourced from a diverse set of 41 actively maintained repositories spanning business applications, B2B services, and developer tools. ... In our evaluation of widely used coding models, under a unified scaffold, we observe that their performance on SWE-BENCH PRO remains below 45% (Pass@1).” · Show in paper