Keith Tyser
@keithtyser
Publications
By @keithtyser
0 starsUpdated
Claims 8
Programming a Coding Agent's Weights by Hand
Show 7 major claimsHide 7 major claims
- The compiled LoRA adapter closely reproduces the intended inference-time neuron-addition hook, achieving residual-stream cosine 0.972–0.996 across layers 18–41 and completion-token KL(hook || adapter) of and , which is 99 and 110 times smaller than the edit's own effect.Verified
- Always-on weight edits move their targeted agent decision monotonically with edit strength on held-out decisions, while matched random-direction controls of the same size do not.Verified
- On the 170 development tasks, low-dose always-on explore-more edits (pooled strengths 0.125–0.5) exceed the unedited model by 2.7 tasks over 12 batches (p = 0.030) and exceed matched random controls by 3.6 tasks over 7 shared batches (p < 0.001), while strength 1.0 solves 6 tasks fewer than the unedited model.Verified
- Three frontier LLM authors (Claude, GPT, Gemini) all produced working edits under one brief, with Claude's before-first-edit gate solving 30.3 against 25.3 development tasks (+5.0, p = 0.014 uncorrected), though no edit survives Holm–Bonferroni correction and the gate did not separate from its ungated 0.125-gap control (+3.75, p = 0.11).Verified
- Neither of two pre-registered held-out tests on 130 repository-disjoint tasks confirmed that the development-set solve-rate gain transfers out-of-sample (always-on explore-more 0.25: +0.33 tasks-equivalent, p = 0.92; before-first-edit gate: +3.0 tasks-equivalent, p = 0.50).Verified
- Control of an agent decision is not enough to solve more tasks for any tested direction other than explore-more, including submit-sooner, think-briefly, test-before-submitting, failed-edit neurons, compiled prompt rules, and one-shot gradient weight deletion.Verified
- The unedited greedy Gemma 4 E4B agent fails by inconsistency rather than incapacity (solving 22–30 of 170 tasks per run, 51 at least once across 13 runs), and failed file edits (within-task OR 0.42, 95% CI 0.23–0.74) and long exploration of 8+ calls before the first edit (OR 0.12, 95% CI 0.04–0.38) are associated with lost episodes.Verified