Reasoning effort, not tool access, buys first-try reliability in agentic code generation: an observational study
Source: arXiv:2607.02436 · Published 2026-07-02 · By Achint Mehta
TL;DR
This study investigates what drives first-try reliability in agentic code generation by evaluating 90 independent runs of agents building the same real-time retrospective board application from a detailed specification. It tests common assumptions that extra capabilities given to agents, such as browser-based testing tools and design-oriented system prompts, improve software reliability and quality. The runs spanned multiple modern model generations, reasoning effort levels, harnesses, testing tool usage, and design prompt variants. The key finding is that reasoning effort and model capability tier dominate functional outcomes and first-try success, while adding the testing tool significantly raises cost without improving reliability or function. Design prompts raise visual quality but come with a functional reliability cost. Notably, raising reasoning effort from High to xHigh increased first-try perfect runs from 28% to 89%, cutting corrective prompts roughly five-fold, for a modest cost increase (9-29%). This highlights that failures mainly stem from weak agent reasoning, which greater deliberation or stronger models reduce, rather than from flaws detectable by interface-level testing tools. A short paraphrase of the design prompt matches the full prompt's visual improvement, showing that the effect depends only on its content.
Key findings
- Frontier models (Claude Opus 4.7, 4.6, Sonnet 4.6) cluster near ceiling functional score (mean ~41/42), while low-cost local model Qwen scores 24-37, about 10 points lower.
- Docker container deployment fails on first attempt in 44% of runs, dominating first-try failures and driving subtle shifts unseen in aggregate run totals.
- Adding a browser-based testing tool (Playwright) raises session cost by 42-68% without improving functional score or first-try reliability, even on interface visible criteria.
- Raising reasoning effort from High to xHigh in Opus 4.7 raised first-try perfect runs from 28% (5/18) to 89% (16/18) and cut mean corrective prompts per run about five-fold (1.0 down to 0.17), at a modest cost increase of 9-29%.
- A design-oriented prompt improves visual quality scores from 3.0 to 4.5-4.7 on a 5-point scale but slightly decreases first-try reliability, mainly due to failures in drag-and-drop interactions.
- A one paragraph abridged design prompt reproduces the full prompt’s visual quality gains with similar reliability costs and token usage, showing length and tone are unnecessary.
- Most run-to-run variability within identical configurations is concentrated in environment failures (Docker, local dev environment), highlighting sensitivity to reasoning and model generation.
- Despite identical specifications and repeated runs, output variability is real but significantly reduced at higher reasoning effort levels.
Threat model
Not a security-focused paper; adversary considered implicitly as software generation failures from weaker models or insufficient agent reasoning causing code defects and first-run failures rather than intentional attacks.
Methodology — deep read
The study employed an observational experimental design varying one factor at a time across 90 independent runs of agentic code generation agents tasked with building the same real-time retrospective board app from a fixed OpenSpec specification. The specification required features including board creation, guest sign-in, drag-and-drop card movements, nested comments, WebSocket real-time sync, single container Docker deployment, CSV export, and developer documentation.
Five model families were tested: Claude Opus 4.7, Opus 4.6, Sonnet 4.6, Gemini 3.1, and a local low-cost Qwen model, with 2-48 runs per family. Two agent harnesses were used (Claude Code and Antigravity). Key variables were reasoning effort (High vs xHigh), testing tool availability (Playwright enabled or not), and use of a design-oriented system prompt (none, full original prompt from Antigravity, or a one-paragraph abridged paraphrase).
Each independent run received the same detailed specification and identical configurations were repeated multiple times per condition for statistical comparison. The testing tool allowed agents browser-based interaction with the live interface, but the agent’s use of it was logged and confirmed.
Runs produced their code and exercised the app in a sandboxed environment on Windows with a Unix-like shell or WSL, and human evaluators scored each run post-hoc against a 14-criterion rubric with 3-point scale (3=success first try, 2=fail first try but fixed after a single human corrective prompt describing only the fault symptom, 1=unfixed). Corrective prompts were used to measure repair burden. Visual quality was rated separately from screenshots on a 5 point scale by a large language model with human verification.
Metrics included functional score (max 42), first-try success per criterion, number of corrective prompts, session cost (estimated dollar cost from token usage and model pricing), output tokens, cache read tokens, lines of code added, and model API time. Costs were right-skewed so medians are reported.
Analysis focused on contrasts varying one factor at a time with matched replicates. Fisher's exact test was used descriptively for run counts with first-try success. The study reports the token-level cause of cost differences, such as context re-reading by models due to testing tool usage rather than new output tokens. Visual prompt effects were analyzed with ablations comparing full vs abridged prompts.
Example run: An Opus 4.7 agent configured with xHigh effort, no testing tool, and full design prompt produced a highly polished UI with visual quality rating ~4.8, functional score near ceiling (41-42), requiring few or no corrective prompts, but sometimes failed interaction element criteria first try due to richer UI complexity. This contrasted with the same model at High effort securing fewer first-try perfect runs and more repair burden.
Code, rubric files, and scoring data are openly available, enabling reproducibility. However, the models and environment depend on proprietary hosted APIs and internal tooling not fully public.
Technical innovations
- Demonstration that increasing model reasoning effort from High to xHigh dramatically boosts first-try code generation reliability, independently of model family or additional tooling.
- Empirical quantification that browser-based testing tools increase operational cost due to repeated context re-reads without measurable gains in code correctness or interface reliability.
- Use of a detailed multi-criteria functional rubric combined with corrective prompt counting to distinguish first-try reliability from post-repair scores, exposing subtle but critical reliability differences.
- Design prompt ablation showing a short paraphrase alone suffices to produce a significant visual quality lift, challenging assumptions about prompt length or emphasis importance.
Datasets
- Agentic code generation runs dataset — 90 runs — publicly available rubric and scoring data but proprietary model APIs
Baselines vs proposed
- Opus 4.7 High effort, base prompt: First-try perfect runs = 28% (5/18) vs Opus 4.7 xHigh effort, base prompt: 89% (16/18)
- Opus 4.7 High effort, base prompt: Median cost = $3.06 vs with Playwright tool: $4.34 (+42%) with no score increase
- Full design prompt vs no design prompt: Visual quality rating 4.5 vs 3.0 with no functional score gain
- Abridged design prompt vs full prompt: Visual quality 4.7 vs 4.5, functional scores similar, cost nearly same
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.02436.
Fig 1 (page 1).
Fig 2 (page 1).
Fig 3 (page 1).
Fig 4 (page 1).
Fig 5 (page 1).
Limitations
- Observational study without randomized assignment limits causal claims despite matched comparisons.
- Limited sample size in some experimental cells, e.g., only 2 Qwen runs and small numbers for Gemini and abridged prompt sets.
- Human evaluation introduces subjective bias in rubric scoring and visual quality ratings despite LLM verification.
- Tool usage limited to one testing harness (Playwright), results may not generalize to other test frameworks or agent integrations.
- Runs executed in a Windows host environment with Unix-like shells may introduce environment-specific failures not generalizable.
- No adversarial fault injection to simulate malicious or adversarial input conditions.
- Dataset and code accessibility partial—while scoring data is open, models and agent environment configurations are proprietary.
Open questions / follow-ons
- Can agentic coding assistants incorporate novel reasoning strategies that further reduce first-try environment deployment failures beyond xHigh effort settings?
- How do different types of testing tools or integration points with agents (beyond browser-level) impact first-try reliability and repair costs?
- What are the tradeoffs in agent prompt design between visual polish and functional reliability in more complex, multi-component applications?
- How might these findings extend to collaborative multi-agent systems or human-in-the-loop debugging workflows?
Why it matters for bot defense
For bot-defense and CAPTCHA engineers concerned with automated code generation or agentic automation, this study highlights that adding more interface-level checking tools may not improve the reliability of code generated by agents, especially if core reasoning is insufficient. Instead, improving the agent’s underlying reasoning effort or deploying stronger model generations substantially enhances first-try correctness, reducing costly human interventions.
The insights about differing failure modes—environment setup vs user interface interaction—inform how to prioritize agent capabilities and tooling investments. For CAPTCHA applications relying on agentic components, focusing on improving agent reasoning processes and controlling for model generation will yield better robustness than layering on external interface checking tools. Additionally, the nuanced effect of design prompts on visual quality versus function informs design decisions about UI polish relative to reliability tradeoffs when agents generate interface code autonomously.
Cite
@article{arxiv2607_02436,
title={ Reasoning effort, not tool access, buys first-try reliability in agentic code generation: an observational study },
author={ Achint Mehta },
journal={arXiv preprint arXiv:2607.02436},
year={ 2026 },
url={https://arxiv.org/abs/2607.02436}
}