When a coding agent works for a long time, its transcript can look like a research notebook. It forms hypotheses, edits code, runs tests, and responds to failures. Those activities matter. So does a different object: the program left on disk when the agent stops.
In Compiled Agency, with Joey Xiao, we study the relationship between that process and its delivered result.
Develop, freeze, evaluate
An agent starts with a game interface and develops a controller. We then freeze the program and evaluate it separately. During play, the frozen controller makes no language-model calls.
This design separates two capabilities: using a model to research and build a controller, and the resulting controller’s ability to act. The environments include a roguelike, StarCraft II, and Civilization.
Holding the artifact fixed makes later evaluation easier to interpret. It also lets us inspect intermediate versions rather than treating an entire development session as a single success or failure.
Local rigor can leave a gap at delivery
The study finds substantial progress across model generations. It also reveals a less flattering pattern: detailed investigation of local problems can coexist with sparse validation of the program that actually ships.
A successful test earlier in a session does not establish that later changes preserve the behavior. Nor does a well-reasoned explanation establish how the final artifact performs on fresh cases. Our intervention with refreshed validation panels tests whether changing the evidence available during development can improve the delivered result.
The implication is practical: an agent’s choice of experiments belongs inside the evaluation. We need to examine what it tests, which evidence it trusts, and why it stops.
The same question in scientific computing
I encounter a related issue in autonomous computational physics. A cited reference provides a target, but the citation alone does not show that a calculation reproduces it. Numerical confrontation—running the calculation and checking the comparison—provides a stronger constraint.
Game controllers and first-principles calculations are different systems. In both, however, the artifact can be tested against evidence beyond the agent’s explanation of its own work.
That is the research direction I want to push further: agents whose experimental practice can be inspected, evaluated, and improved alongside the outputs they produce.