EvoCUA-1.5: Online Reinforcement Learning for Multi-turn Computer-Use Agents

Source: arXiv:2607.09773 · Published 2026-07-07 · By Mianqiu Huang, Taofeng Xue, Chong Peng, Jinrui Ding, Sicheng Fan, Jiale Hong et al.

TL;DR

EvoCUA-1.5 addresses the challenge of training multi-turn computer-use agents that operate in partially observable, multimodal desktop environments. Unlike static offline imitation learning approaches, EvoCUA-1.5 employs online reinforcement learning (RL) to leverage live interactions with sandbox environments and learn from sparse, verifiable terminal rewards. The paper identifies critical difficulties in this setting including variable-length trajectories, delayed feedback, context-managed observations, and slow environment step times. To overcome these, EvoCUA-1.5 introduces Step-Level Policy Optimization (STEPO), which correctly distributes trajectory-level advantages to step-level samples to avoid overweighting long trajectories. It also applies policy-aware data filtering and pass-rate calibration to maintain a high signal-to-noise ratio training set, and uses a novel Dynamic Tri-Adaptive Curriculum (DTAC) to balance learnable, hard positive replay, and infeasible tasks. A fully asynchronous RL infrastructure with staleness control and mini-group batching supports stable scalable training on expensive interactive desktop tasks. Experiments demonstrate that EvoCUA-1.5 achieves 63.2% success on OSWorld-Verified, significantly surpassing previous 32 billion parameter baselines and nearing much larger models. Ablations confirm the importance of STEPO, curriculum learning, and data filtering. The framework constitutes a practical approach to scaling online RL for generalist multi-turn computer-use agents.

Key findings

• EvoCUA-1.5 achieves 63.2% success rate on OSWorld-Verified benchmark using 32B parameters and 100-step interaction budget, improving over predecessor EvoCUA-32B at 56.7%
• Step-Level Policy Optimization (STEPO) corrects bias from naive advantage reuse by dividing trajectory-level reward across step samples, restoring group-level advantage normalization
• Dynamic Tri-Adaptive Curriculum (DTAC) combining Variance-Adaptive Sampling, Difficulty-Adaptive Positive Replay, and Infeasibility-Controlled Sampling improves training stability and sample efficiency under sparse rewards
• Policy-aware filtering and pass-rate calibration prevent noisy or trivial tasks from dominating the training buffer, maintaining a high signal-to-noise ratio dataset
• Asynchronous rollout–buffer–training architecture with staleness control and mini-group batching reduces idle GPU time and prevents biased advantage normalization
• EvoCUA-1.5 outperforms comparable open-weight 32B/35B models like OpenCUA-72B and UI-TARS-1.5-7B, and approaches performance of significantly larger parameter count models
• Cross-platform evaluation on WindowsAgentArena and MacOSArena shows transfer of improved policy beyond OSWorld environment
• Ablation study shows STEPO, policy-aware curriculum, and asynchronous infrastructure each contribute significant gains

Threat model

N/A — this paper addresses reinforcement learning methodology for interactive multi-turn computer-use agents rather than adversarial threat scenarios. The implicit threat is environment complexity and partial observability rather than a malicious adversary.

Methodology — deep read

The authors focus on online reinforcement learning for computer-use agents interacting with graphical user interfaces through multi-turn trajectories. The threat model assumes no adversary but highlights the complexity of partial observability, multimodal observations (screenshots, textual reasoning traces), and sparse terminal rewards verified by executable validators. Data originates from a synthesized task pool produced by a prior EvoCUA pipeline. These tasks cover atomic computer-use abilities and are paired with sandbox environments and executable validators. Tasks are filtered through sandbox feasibility checks and validator inspection to remove noisy or infeasible tasks. The training split is implicit, with online RL continually sampling from and updating over this synthesized but filtered pool. Observations at each step are managed by a context-management policy using a sliding window: older turns are compressed into action-history summaries, while recent turns retain full modalities to fit within model context windows. Training samples correspond to these step-level context-managed inputs paired with reasoning traces and executable actions. The core algorithmic novelty is Step-Level Policy Optimization (STEPO), which computes trajectory-level advantages normalized over a rollout group and distributes these evenly across all steps in the trajectory. This solves a bias inherent in naive application of Group Relative Policy Optimization (GRPO) — which otherwise overweighted longer trajectories by assigning the entire trajectory advantage to each step, destabilizing training. STEPO thus preserves group-level zero mean advantage structure at the step sample level. The policy gradient objective is clipped similarly to PPO. To improve learning stability and sample effectiveness, they introduce policy-aware data filtering and pass-rate calibration: tasks whose current policy pass rates are near 0 or 1 are downweighted, focusing training on tasks with intermediate success rates that produce informative success/failure contrasts. To dynamically balance the curriculum as the policy evolves, the Dynamic Tri-Adaptive Curriculum (DTAC) samples each minibatch from three complementary channels—Variance-Adaptive Sampling (VAS) targeting medium-difficulty tasks; Difficulty-Adaptive Positive Replay (AdaPR) sampling rare positive trajectories from hard-but-learnable tasks; and Infeasibility-Controlled Sampling (ICS) maintaining some samples from infeasible tasks for failure recognition. The entire online RL system uses a fully asynchronous rollout-worker, data-buffer, and training-worker setup. Rollouts are executed continuously by parallel workers generating trajectories with policy versions tracked for staleness control. The buffer retains only recent samples from policies within a staleness window to mitigate off-policy drift. Training minibatches are constructed by grouping all step samples from complete rollout groups (mini-group batching) to maintain the advantage normalization intact. Adaptive GPU reallocation allocates resources dynamically between rollout inference and training to avoid stalls when rollout speed varies. Evaluation encompasses the OSWorld-Verified benchmark for general computer-use tasks with 100-step interaction budgets, and cross-platform tests on WindowsAgentArena and MacOSArena. Ablations isolate the impact of STEPO, curriculum components, asynchronous infrastructure, and reward design. While the code and datasets are internal or partially closed—the synthesized tasks and sandbox environments are from the EvoCUA line—model backbones are open Qwen3-VL variants. The paper provides detailed algorithmic pseudocode, empirical ablation tables, and figure visualizations of curriculum distributions and asynchronous system structure. A complete end-to-end example parsing a multi-turn trajectory through context management, advantage redistribution by STEPO, and subsequent policy optimization is described conceptually but exact code-level detail is not included.

Technical innovations

• Step-Level Policy Optimization (STEPO) distributes trajectory-level advantages evenly over decomposed step-level samples, preserving group-level normalization and correcting biases from naive GRPO application.
• Policy-aware online data filtering and pass-rate calibration dynamically adjust task sampling weights based on sandbox feasibility, validator reliability, and model-dependent pass rates.
• Dynamic Tri-Adaptive Curriculum (DTAC) combines Variance-Adaptive Sampling, Difficulty-Adaptive Positive Replay, and Infeasibility-Controlled Sampling to adaptively focus training on learnable, rare positive, and infeasible tasks, respectively.
• Fully asynchronous rollout–buffer–training architecture with staleness control and mini-group batching maintains training throughput and preserves intra-group advantage structure despite variable length multi-turn trajectories.

Datasets

• Synthesized Task Pool — size unspecified — generated with EvoCUA verifiable task synthesis pipeline and sandbox environments
• OSWorld-Verified — benchmark suite of automated computer-use tasks — sandbox-based execution and executable validators
• WindowsAgentArena — cross-platform benchmark on Windows environment — source from Bonatti et al., 2024
• MacOSArena — cross-platform benchmark for macOS GUI tasks — part of MMBench-GUI collected by Wang et al., 2025c

Baselines vs proposed

• EvoCUA-32B (offline synthesis + RL baseline): Pass@1 = 56.7% vs EvoCUA-1.5-32B: 63.2%
• OpenCUA-72B: lower than EvoCUA-1.5-32B at 32B scale (exact numbers not stated)
• UI-TARS-1.5-7B: underperforms compared to EvoCUA-1.5-32B
• Qwen3-VL-32B-Thinking: baseline pass rate lower on WindowsAgentArena and MacOSArena compared to EvoCUA-1.5-32B (exact improvements not quantified)
• Ablation of STEPO vs naive GRPO shows degraded stability and lower success rates without STEPO (deltas not precisely quantified)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.09773.

Fig 1: Performance comparison on OSWorld-Verified. EvoCUA-1.5 achieves 63.2% success rate, outperforming

Fig 2 (page 1).

Fig 3 (page 1).

Fig 4 (page 1).

Fig 2: Context management in multi-turn computer-use agent training. The full trajectory history is transformed

Fig 3: Atomic-ability coverage in the synthesized task pool. EvoCUA-1.5 builds on EvoCUA’s verifiable task

Fig 4: Overview of Dynamic Tri-Adaptive Curriculum (DTAC). Each training batch combines tasks selected by

Fig 5: Asynchronous online RL infrastructure. Rollout workers continuously generate computer-use trajectories,

Limitations

• Synthesized task pool is internal and not publicly released, limiting reproducibility
• Exact computational budgets, wall-clock runtimes, and training time are not fully detailed
• No explicit adversarial evaluation or robustness testing against manipulated environments or reward hacking beyond reward hacking note
• Sparse terminal rewards may limit fine-grained credit assignment despite STEPO improvements
• Potential residual length biases introduced by STEPO require careful monitoring to avoid exploitation
• Evaluation focuses on sandbox environments; performance in fully real-world desktop settings remains unverified

Open questions / follow-ons

• How well does STEPO and DTAC scale to even longer horizon tasks with multiple parallel GUI applications or concurrent processes?
• Could reward models or other auxiliary signals complement sparse terminal rewards to improve credit assignment?
• How generalizable is this framework when applied to real user desktops beyond sandbox simulators?
• What defenses or detection mechanisms can mitigate potential reward hacking via repeated or cyclic GUI actions in multi-turn RL?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, EvoCUA-1.5 offers a compelling approach to training agents that interact robustly and adaptively with complex, multi-step GUI environments through online reinforcement learning. The Step-Level Policy Optimization (STEPO) technique is particularly relevant because it addresses the challenge of credit assignment in variable-length, multi-turn interaction sequences—a setting analogous to interaction with CAPTCHA challenges requiring multiple user actions over time. The Dynamic Tri-Adaptive Curriculum (DTAC) provides an adaptive method to continuously focus training on tasks of appropriate difficulty, analogous to evolving bot challenges and user response distributions. The asynchronous infrastructure details highlight engineering best practices needed to efficiently train such agents given slow, expensive environment feedback, a scenario common in interaction with CAPTCHA or fraud detection systems that throttle or delay responses. While EvoCUA-1.5 targets autonomy in desktop environments, the core insights into context management, curriculum sampling under sparse terminal rewards, and policy-aware data filtering could inform future bot-defense AI, especially for sequential automated interaction detection or resilient multi-turn CAPTCHA solving. However, evaluation in real adversarial settings, robustness to reward manipulation, and transfer to live unstable environments remain open challenges for immediate direct application.

Cite

@article{arxiv2607_09773,
  title={ EvoCUA-1.5: Online Reinforcement Learning for Multi-turn Computer-Use Agents },
  author={ Mianqiu Huang and Taofeng Xue and Chong Peng and Jinrui Ding and Sicheng Fan and Jiale Hong and Yufei Gao and Xiaocheng Zhang and Linsen Guo and Xin Yang and Dengchang Zhao and Yuchen Xie and Peng Pei and Xunliang Xie and Xipeng Qiu },
  journal={arXiv preprint arXiv:2607.09773},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.09773}
}

Read the full paper

• arXiv:2607.09773 (abstract)
• arXiv:2607.09773 (PDF)