Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agents

Source: arXiv:2606.31270 · Published 2026-06-30 · By Xueqiao Sun, Xiaohan Wang, Ludwig Schmidt, Serena Yeung-Levy, Yuhui Zhang

TL;DR

This work addresses a key inefficiency in current computer-use agent training pipelines, which discard failed execution trajectories despite their rich diagnostic information. Standard practice fine-tunes agents only on successful trajectories collected in verifiable simulated environments, leaving failure data unused. The authors propose a complementary failure-driven self-improvement loop that operates at inference time. By leveraging a large language model (LLM) as a meta-controller, failed trajectories are analyzed to identify structured failure modes, and targeted code-level patches implementing strategies like visual search, terminal command execution, knowledge retrieval, and repetition detection are synthesized with light human verification. Incorporating these patches upgrades the agent's inference-time behavior without additional training. When applied to the state-of-the-art OpenCUA-72B agent on the OSWorld benchmark, this approach improves success rate by 6.6 percentage points from 42.3% to 48.9%, a 15.6% relative increase, with only modest inference overhead. The improvements generalize across architectures and benchmarks. This work thus demonstrates that failed trajectories are a valuable supervisory signal for computer-use agents and that an LLM-guided failure-case loop offers a practical, complementary means to improve agent capabilities efficiently and continuously.

Key findings

• Failure-driven inference-time self-improvement improves OpenCUA-72B success on OSWorld from 42.3% to 48.9% (+6.6 points, +15.6% relative) without additional model training.
• Visual search strategy alone increases performance from 41.67% to 47.22% on OSWorld small set (a 5.55-point gain).
• Terminal execution strategy improves OSWorld small set performance by 5.52 points (41.67% to 47.19%).
• Knowledge support (via software manuals and external search) adds 2.77-point gain (41.67% to 44.44%).
• Repetition detection and warning contributes 2.73-point improvement (41.67% to 44.40%).
• Combining all four inference-time strategies boosts OSWorld small set success to 52.74%, a total gain of 11.07 points over baseline.
• Failure mode distribution shifts after applying the framework: reductions in grounding errors and action loops, increase in higher-level failures like task misinterpretation and multi-step planning challenges.
• Framework generalizes with consistent performance improvements across models (OpenCUA-32B, GUI-Owl-32B) and diverse benchmarks (OmniACT, Android-Control, ScreenSpotPro, WebVoyager) without retraining.

Threat model

The adversary is the complexity and variability of GUI environments causing execution failures during agent task operations. The framework assumes no malicious actors manipulating the agent or meta-controller. The main challenge is intrinsic task difficulty and agent limitations in perception, knowledge, and reasoning. The adversary cannot prevent the agent from receiving human-verified code patch updates or block access to external knowledge sources used for inference-time improvement.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is essentially the environment challenges and task complexities that cause agent failures. The framework assumes access to a multimodal large language model (MLLM)-based computer-use agent, a verifiable environment that can label trajectory success, and an LLM meta-controller capable of analyzing failure trajectories and proposing inference-time code patches. The adversary cannot manipulate the LLM or verifiable environment but exposes realistic failures during agent rollouts.

2. Data: Failed and successful trajectories are collected during agent rollouts in the OSWorld verifiable environment. Trajectories contain state-action pairs, instructions, thought processes, and screenshots. The OSWorld benchmark includes diverse GUI tasks with ground-truth success labels. The authors use a fully open-source, publicly available dataset, split into a full set and a lightweight small set for rapid iteration.

3. Architecture/Algorithm: The base agent is OpenCUA-72B, an MLLM trained to predict GUI actions given screenshot states and instructions. The novel component is the failure-case loop meta-controller implemented by Claude 4.5 Sonnet, an LLM that processes failed trajectories, diagnoses failure modes (grounding errors, competency gaps, knowledge deficiencies, redundant loops), and generates inference-time behavioral patches. Key strategies include:

• Visual Search: Multi-round zoomed-in grounding verification on click/move actions using cropped screenshot regions with visual hints.
• Terminal Execution: Bypass complex GUI interaction by invoking terminal commands through hotkeys.
• Knowledge Support: External retrieval from software manuals and web search engines to fill domain-specific knowledge gaps.
• Repetition Warning: Sliding window detection of repetitive thoughts, actions, and constant screen states triggering recovery interventions.

These patches modify the agent’s inference-time action generation without retraining the model weights.

4. Training regime: No additional training is performed on the base agent during self-improvement. Instead, the process iterates multiple rounds (four rounds in experiments) of failure collection, LLM diagnosis and patch synthesis, human verification (minimal edits in >97% cases), and agent evaluation.

5. Evaluation protocol: Performance is measured by task success rate on OSWorld, with both a 30-step small set for ablation and a 100-step full benchmark for primary results. Ablations isolate the effect of each strategy individually and combined. The meta-controller models are compared across Claude 4.5 Sonnet, GPT-5.2, Gemini 3 Flash, and Qwen3-VL-32B-Instruct for best overall performance. Generalization tests run failure-case patches on different agent models and four distinct GUI benchmarks with the same evaluation metrics and no retraining to demonstrate transferability.

6. Reproducibility: Code for the failure-case loop is released at https://github.com/snow10072740/Learning_from_Failure. The OSWorld dataset and OpenCUA agents are publicly available. The meta-controller model is Claude 4.5 Sonnet, which is a commercial LLM, limiting full replication. Detailed appendices describe the prompting, patch formats, and implementation.

Concrete Example: The visual search strategy corrects an agent’s erroneous click on an up-arrow in GIMP by zooming in around the action coordinate, showing a red circle prompt, and refocusing to the correct down-arrow to lower brightness, turning a failure into success at inference time without retraining.

Technical innovations

• Introduction of a failure-case loop that utilizes failed trajectories at inference time for agent self-improvement without additional training.
• Use of an LLM meta-controller to diagnose systematic failure modes from multimodal trajectory data and synthesize code-level behavioral patches.
• Implementation of inference-time correction strategies (visual search, terminal execution, knowledge support, repetition warning) that modify agent actions dynamically.
• Integration of light human verification to validate and minimally adjust automatically generated code patches for reliable inference-time agent upgrades.

Datasets

• OSWorld — medium-scale multimodal GUI task benchmark with verifiable success labels — public

Baselines vs proposed

• OpenCUA-72B baseline: OSWorld success rate = 42.3%
• OpenCUA-72B + failure-case loop: OSWorld success rate = 48.9%
• Visual Search alone: OSWorld small set = 47.22% vs baseline 41.67%
• Terminal Execution alone: OSWorld small set = 47.19% vs baseline 41.67%
• Knowledge Support alone: OSWorld small set = 44.44% vs baseline 41.67%
• Repetition Detection alone: OSWorld small set = 44.40% vs baseline 41.67%
• All strategies combined: OSWorld small set = 52.74% vs baseline 41.67%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.31270.

Fig 2: Overview of our failure-case loop, a self-evolving framework. In each round,

Fig 3: Case studies of the visual search and terminal execution strategies. In both

Fig 3 (page 5).

Fig 4 (page 5).

Fig 5 (page 5).

Fig 6 (page 5).

Fig 7 (page 5).

Fig 8 (page 5).

Limitations

• Heavy reliance on access to commercial LLM Claude 4.5 Sonnet as meta-controller limits full reproducibility and applicability.
• Human verification, though minimal, is still required to review and adjust generated patches, adding operational overhead.
• Evaluation focuses on success rate improvements without detailed analysis of computational cost trade-offs during inference.
• Framework addresses low- and mid-level failures but does not fully resolve high-level cognitive challenges like task misinterpretation or multi-step planning.
• No explicit adversarial or robustness testing against intentional failure modes or environment perturbations.
• Tests on heterogeneous environments show promise but may require further adaptation to more diverse or unseen GUIs.

Open questions / follow-ons

• How to fully automate patch verification to eliminate human-in-the-loop steps for scalability?
• Can the failure-case loop be adapted to handle adversarial GUI perturbations or malicious environment manipulations?
• What is the long-term interaction effect of repeated inference-time improvements on agent behavior stability and safety?
• How to better integrate high-level cognitive improvements (e.g., multi-step planning, semantic understanding) into the failure-driven framework?

Why it matters for bot defense

This paper provides valuable insights for bot-defense and CAPTCHA practitioners by demonstrating a practical framework to enhance interactive agents’ robustness and accuracy through inference-time self-improvement using LLM-guided failure analysis. The identification and automated correction of systematic failure modes such as grounding errors and repetitive actions have direct parallels to CAPTCHA-breaking bot behaviors, where mis-recognition or repetitive failed inputs degrade success rates. The failure-driven loop suggests a new dimension of adaptive bot or agent hardening by dynamically patching behavioral logic without retraining, which can inform designs for more resilient CAPTCHA defenses and adaptive challenge formulations.

Moreover, the inference-time code patch paradigm highlights a potential vector for real-time agent behavioral modification by attackers or defenders, providing a cautionary perspective for CAPTCHA systems reliant on fixed interaction logic. Understanding failure modes like redundant loops and grounding precision problems can guide CAPTCHA challenge selection to exploit agent weaknesses, while insights into adaptive recovery mechanisms inform how bots might evolve to overcome defenses. Overall, this work enriches the toolkit for both building stronger interactive agents and designing CAPTCHA systems robust against continuous agent self-improvement.

Cite

@article{arxiv2606_31270,
  title={ Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agents },
  author={ Xueqiao Sun and Xiaohan Wang and Ludwig Schmidt and Serena Yeung-Levy and Yuhui Zhang },
  journal={arXiv preprint arXiv:2606.31270},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.31270}
}

Read the full paper

• arXiv:2606.31270 (abstract)
• arXiv:2606.31270 (PDF)