By Bhanu Prakash Vangala, Tanu MalikSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2610-00425.git
Cite
Version link
https://openversions.com/index/arxiv-2610-00425/commit/37cf431a81e43c00ff2aa91f6f5842966f2271a9
Version identifier (commit hash)
37cf431a81e43c00ff2aa91f6f5842966f2271a9

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Latest version

Checks complete

Archive from arXiv Source: https://arxiv.org/abs/2610.00425v1 OpenVersions: 3ee9dc0f-cc53-432f-baf8-13f86bb0fd0d

Posted

Claims

Findings

Headline finding

Current LLM coding agents exhibit systematic generalization failures in environment specification, producing dependency specifications that are inconsistent, redundant, or incomplete in ways that functional tests do not detect, and this failure is not resolved by scale or recency.

First-attempt environment specification success varies widely across agents and languages, whereas iterative repair up to k=10 cycles enables nearly every configuration to converge to functional execution.

Environment specification quality and transitive inflation vary substantially by language ecosystem, with Python achieving the highest initial-manifest accuracy (macro F1 = 0.765) and JavaScript exhibiting the worst misspecification and largest transitive inflation (macro F1 = 0.219, mean rho = 11.40).

Coding agents solving identical tasks in the same ecosystem declare substantially different dependency sets, with mean pairwise Jaccard similarity below 0.5 in all ecosystems and as low as 0.073 between Codex and Gemini in JavaScript.

Dependency selection within a single coding agent is non-deterministic across independent runs, yielding a 0% unanimous consensus rate across all four languages for Claude Code.

In C++, first-attempt failures for Claude (SLAR = 87.5%) and Codex (SLAR = 17.9%) are driven by assuming pre-installed system development libraries absent from clean containers, and 100% of these failures are repaired once feedback is provided.

On tasks solvable entirely with the standard library, coding agents still frequently declare unnecessary external packages, with Claude introducing external packages in 91.8% of such tasks on macro average.

Larger declared dependency sets are negatively associated with first-attempt execution success in Python, Java, and C++.

Methods

Two-phase Agent Evaluation Protocol separating initial generation (C^(0), M^(0)) = Gen(A, \pi_T) from iterative error-driven repair (C^(i), M^(i)) = Repair(A, I, [C^(i-1), M^(i-1), e^(i-1)]) up to k=10 cycles in an isolated environment I.

Source excerpts

“Our agent evaluation protocol divides this agent behavior into two skills, a generation skill, in which it produces code and a dependency manifest for a given task, and a repair skill, in which it iteratively revises both specified code and dependency manifest in response to runtime failures.” · Show in paper

“All agents receive the same prompt \pi_T for each task, with the iteration budget k = 10 repair cycles specified in each agent's configuration file as described in Section 2, yielding a maximum of 11 agent interactions per instance -- one Gen call followed by up to 10 Repair calls.” · Show in paper

Three-Layer Dependency Extraction Framework decoupling dependencies into declared D^(1) = ParseManifest_l(M^(0)), installed D^(2) = Resolve_l(M^(k), I), and runtime-traced D^(3) = Trace(C^(k), I) using ptrace-based Sciunit inside clean ubuntu:22.04 Docker containers.

Source excerpts

“From these artifacts, obtained iteratively, we derive three dependency sets: The declared set D^(1) := ParseManifest_l(M^(0)) captures the agent's initial claims about the environment the code needs given the programming task. The installed set D^(2) := Resolve_l(M^(k), I) captures the environment installed in I at the end of the iterative cycle. ... The traced set D^(3) := Trace(C^(k), I) is defined only when e^(k) = \emptyset, i.e., when the program has been successfully installed and executes without error.” · Show in paper

“We realize Trace(C^(k), I) using Sciunit [13], compiled from source and installed in every Docker image. Sciunit attaches to the target process via the Linux ptrace API and intercepts all open(), openat(), read(), and execve() system calls, writing the absolute path of each accessed file to a provenance log.” · Show in paper

Controlled empirical study spanning 3 production coding agents (Claude Code, Codex, Gemini Code Assist), 4 language ecosystems (Python, Java, JavaScript, C++), and 50 platform-independent, self-contained programming tasks, totaling 600 primary instances and 400 stochastic trials (1,000 manual evaluation runs).

Source excerpts

“Every agent a \in \mathcal{A} receives every prompt \pi_T for every task-language pair (T, l) \in \mathcal{T} \times \mathcal{L}, yielding |\mathcal{T}| \times |\mathcal{L}| \times |\mathcal{A}| = 50 \times 4 \times 3 = 600 primary instances. To assess the stochastic variability of the Gen and Repair skills, a_1 is additionally executed for two further independent trials per task-language pair, each initialized from a fresh conversation with no memory of prior attempts, contributing 50 \times 4 \times 2 = 400 stochastic trials. ... The total study scale is thus 1000 evaluation runs (Table 2).” · Show in paper

“All 1000 instances were conducted manually, with each instance comprising a multi-turn interaction: the initial Gen prompt followed by up to ten Repair turns, for a maximum of eleven agent interactions per instance (Table 2).” · Show in paper

Quantitative environment specification metrics including mismatch rates (phantom \phi, hidden \eta, bloat \beta), inflation ratio \rho = |D^(2)|/|D^(1)|, retrieval metrics (Precision, Recall, F1 against D^(3)), Jaccard similarity J, complete disjointness rate (CDR), unanimous consensus rate (UCR), unnecessary dependency rate (UDR), and C++ system library assumption/recovery rates (SLAR, EGAR).

Source excerpts

“The three mismatch rates are \phi = |D^(1) \ D^(3)| / n_1, \eta = |D^(3) \ D^(1)| / n_3, \beta = |D^(2) \ D^(3)| / n_2, (4) with each rate set to zero when its denominator is zero. ... Treating D^(1) as a retrieval system predicting D^(3), per-project precision, recall, and F1 are P = |D^(1) \cap D^(3)| / n_1, R = |D^(1) \cap D^(3)| / n_3, F1 = 2PR / (P+R)” · Show in paper

“For both assessments, we define the pairwise declared-dependency agreement between agents a_i and a_j on task p in language l as J(a_i, a_j, p, l) = |D^(1)_{a_i} \cap D^(1)_{a_j}| / |D^(1)_{a_i} \cup D^(1)_{a_j}|” · Show in paper