Headline finding
Current LLM coding agents exhibit systematic generalization failures in environment specification, producing dependency specifications that are inconsistent, redundant, or incomplete in ways that functional tests do not detect, and this failure is not resolved by scale or recency.
The empirical results across 1,000 runs substantiate that all three tested agents frequently over-specify, under-specify, and inconsistently declare dependencies even when they ultimately pass execution.
“Our results show that current coding agents exhibit systematic generalization failures along this dimension, producing dependency specifications that are inconsistent, redundant, or incomplete in ways that functional tests do not detect. Across agents, dependency set agreement is as low as 7% for identical tasks, and newer agents show no meaningful improvement, suggesting the failure is not resolved by scale or recency.” · Show in paper