ProCUA-SFT Technical Report

Source: arXiv:2606.17321 · Published 2026-06-15 · By Jaehun Jung, Ximing Lu, Brandon Cui, Muhammad Khalifa, Shaokun Zhang, Hao Zhang et al.

TL;DR

This work addresses the significant bottleneck in training computer-use agents (CUAs), models that interact with full graphical desktop environments via screenshots and keyboard/mouse actions. The core challenge is collecting large-scale, rich, and diverse training trajectories covering complex workflows in realistic desktop setups. The largest existing public dataset, AgentNet, contains 22.5K human trajectories but suffers from negative transfer when used for supervised fine-tuning (SFT), reducing task success rates substantially on standard benchmarks like OSWorld. To overcome this, the authors present ProCUA-SFT, a synthetic dataset comprising 3.1 million step-level training samples distilled from 93K trajectories spanning 2,484 application combinations. These trajectories are generated by an automated pipeline using a single vision-language model (VLM) that synthesizes grounded, feasible goals with binary precondition checks on live desktops seeded with authentic real-world content, including 912 large spreadsheets and ~10K permissively-licensed presentations. The joint goal generation and execution model eliminates planner–actor mismatches common in prior work.

Fine-tuning the UI-TARS 7B agent model for just one epoch on ProCUA-SFT yields a 45.0% success rate on OSWorld, an 18.7 percentage-point absolute improvement over the base UI-TARS model and more than 35 percentage points better than AgentNet fine-tuning. The authors further analyze trajectory structure, showing ProCUA’s longer horizons, richer multi-application workflows, and higher reliance on robust keyboard actions contribute to its superior performance. Large-scale data diversification, grounded task feasibility verification, and context-aware step-prefix training sample expansion are key enablers. The dataset and methodology represent a new state-of-the-art foundation for training CUAs and have already helped improve subsequent models such as Nemotron 3 Nano Omni.

Key findings

• Fine-tuning UI-TARS 7B on ProCUA-SFT for one epoch achieves 45.0% success rate on OSWorld, improving 18.7 percentage points over the base 26.3% model and 35 percentage points over AgentNet-trained models which plateaued at 8-10%.
• AgentNet’s 22.5K human trajectories cause negative transfer when fine-tuned, dropping OSWorld success rate from 26.3% to 8-10%.
• ProCUA-SFT contains 3.1 million step-level supervised samples distilled from 93K synthetic trajectories across 2,484 distinct application combinations.
• Trajectory length in ProCUA is roughly 1.8X longer than AgentNet (∼30 vs ∼17 steps), with a heavier long tail extending to more complex workflows.
• Nearly half of ProCUA trajectories target LibreOffice apps (Impress 24%, Calc 17%, Writer 6%) and 20% involve multi-app workflows, matching OSWorld evaluation distribution better than AgentNet.
• ProCUA’s action distribution is less click-dominated (∼41% clicks) and has a higher proportion of keyboard actions compared to AgentNet (∼63% clicks), resulting in more deterministic and robust interactions.
• Round-robin sampling by application combination during SFT produces 30.9% success at fixed training budget, outperforming base (26.3%) and non-diversified sampling (27.3%), while sampling by action set or nested methods underperform.
• The single-VLM pipeline with in-loop binary precondition verification ensures synthesized tasks are feasible before rollout, reducing wasted computation and negative training signals.

Threat model

n/a — This work focuses on dataset generation and supervised fine-tuning methodology for desktop computer-use agents. There is no explicit adversarial threat model or security evaluation; the assumptions are of reliable and correct environment emulation, accurate screenshot capture, and deterministic VM states for training trajectory collection.

Methodology — deep read

1. Threat Model & Assumptions: The work assumes a training setup for CUAs where the agent perceives the desktop state only through screenshots and issues mouse/keyboard commands. The adversary here is not explicitly defined as this is a dataset and training methodology paper, but assumptions include accurate desktop state observability through screenshots and a fixed known environment setup allowing precondition verification. There is no assumption of malicious adversary attacking the model.

2. Data: ProCUA-SFT synthesizes 93,000 trajectories on fully instantiated Linux desktop VMs booted with multi-app configurations seeded with real documents. Seed content comes from three sources: (i) OSWorld snapshots with multi-application states, (ii) SpreadsheetBench with 912 real-world large spreadsheets uploaded to the desktop, and (iii) Zenodo10K with ~10,000 permissively-licensed presentation (.pptx) files grouped by parent record for multi-file states. The trajectories span 2,484 distinct application combinations involving applications like LibreOffice Calc, Impress, Writer, Chrome, File Manager, GIMP, Thunderbird, etc. Each trajectory averages ~30 steps. The data is split implicitly, with some trajectories used for ground-truth evaluation (OSWorld benchmark) and a large set for training.

3. Architecture/Algorithm: The pipeline revolves around a single vision-language model called Kimi-K2.5 for synthesis and rollout. The model has three roles, each via different prompts: (a) Goal Generator, which proposes a textual high-level task goal for the given desktop state along with accompanying binary preconditions; (b) Judge, which verifies whether each precondition holds in the current desktop state using input screenshot and OS config; and (c) Trajectory Executor, which acts on the goal via chain-of-thought reasoning to generate mouse/keyboard actions stepwise. The single-model approach closes the common planner–actor capability gap by ensuring the generator only proposes goals the executor can feasibly complete.

4. Training Regime: UI-TARS 7B is fine-tuned for one epoch on the 3.1M step-level supervised samples from ProCUA-SFT. Training uses batch size 512, learning rate 2e-5, cosine decayed scheduler, weight decay 0.1, sequence length up to 32k tokens. Hardware specifics: multiple GPU nodes with Slurm orchestrating local KVM VM environments (Singularity container) and optional remote NVIDIA Cloud Functions (NVCF) to scale rollout data collection asynchronously. Step-prefix expansion is applied: Each trajectory of T steps yields T training samples, each consisting of the context of all screenshots and actions up to that step to mimic the inference scenario.

5. Evaluation Protocol: Evaluation is on OSWorld, a benchmark with realistic Linux/Windows/macOS desktop tasks with unrestricted keyboard and mouse. The metric is task success rate. Baselines include UI-TARS 7B pretrained without fine-tuning and UI-TARS fine-tuned on AgentNet. Ablation studies investigate sampling strategies for diversity (round-robin by app combo, action set, nested, no diversification) on a matched training budget of ~700 steps. Metrics are aggregated over multiple runs, smoothed over training steps, but no note of statistical significance tests are mentioned.

6. Reproducibility: The ProCUA-SFT dataset is released publicly on Hugging Face. The precise model weights for Kimi-K2.5 and UI-TARS 7B checkpoints are not explicitly mentioned. The environment setup leverages open datasets (SpreadsheetBench, Zenodo10K) under permissive licenses, but SpreadsheetBench files are not redistributed due to licensing. The data collection pipeline is described in detail but code release status is unclear from the excerpt.

End-to-end example: Given a live Linux desktop with a real spreadsheet open (from SpreadsheetBench), the single VLM first generates a goal such as "open quarterly report Q3.xlsx and add a revenue chart" along with preconditions like file existence, app availability. If Judge verifies preconditions pass on the current desktop state, rollout proceeds with the VLM generating a chain-of-thought trajectory stepwise controlling mouse/keyboard (click, typewrite, hotkeys). Each intermediate step yields a supervised training sample that replicates the exact desktop screenshot and prior history context the agent would see at inference. Failed or infeasible goals trigger iterative retries until a feasible one is found. The entire trajectory is checkpointed and aggregated into the dataset.

Technical innovations

• Use of a single vision-language model (Kimi-K2.5) for joint goal synthesis, binary precondition verification, and trajectory rollout to eliminate planner–actor capability mismatches present in prior multi-model pipelines.
• Grounded task synthesis with explicit in-loop binary precondition checks that verify task feasibility against desktop state before rollout, reducing wasted compute and false supervision signals.
• Seeding synthetic task generation from large-scale, real-world, permissively-licensed complex desktop content (912 spreadsheets from SpreadsheetBench and ~10K presentations from Zenodo10K) to enable difficult multi-app workflows.
• Step-prefix expansion of trajectories into step-level supervised training samples that exactly reproduce the context layout seen at inference time, improving alignment between training and deployment context windows.

Datasets

• ProCUA-SFT — 3.1 million step-level samples distilled from 93,000 synthetic trajectories across 2,484 application combinations — publicly released on Hugging Face
• AgentNet — 22.5K human trajectories covering cross-platform desktop workflows — public but limited due to annotation noise and limited diversity
• SpreadsheetBench — 912 real-world large spreadsheets (100+ columns, 20K+ rows) harvested from online forums — research license, not redistributed
• Zenodo10K — ~10,000 permissively-licensed presentations (.pptx) mined from Zenodo open-research repository — CC-BY 4.0 reuse license

Baselines vs proposed

• UI-TARS 7B base model: OSWorld success rate = 26.3% vs ProCUA-SFT fine-tuned: 45.0%
• UI-TARS 7B fine-tuned on AgentNet: OSWorld success rate ~8-10% vs ProCUA-SFT fine-tuned: 45.0%
• Sampling strategy - Round-robin by app combination: 30.9% vs non-diversified sampling: 27.3% vs base model: 26.3%
• Sampling strategy - Round-robin by action set: 24.9% vs base model: 26.3%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.17321.

Fig 1 (page 1).

Fig 1: | ProCUA-SFT pipeline. A fresh desktop is seeded from one of three task-config sources (OSWorld,

Fig 2: | (a) OSWorld success rate during SFT on UI-TARS 7B. ProCUA (blue) rises steadily from 11.7%

Fig 4: | Six representative GUI-agent trajectories illustrating the spectrum of structural complexity

Fig 5: | Comparison of the full ProCUA training set (16 480 trajectories) and the ProCUA + Rare-

Limitations

• No adversarial robustness analysis or evaluation under distribution shifts beyond OSWorld benchmark domains.
• Synthetic trajectories rely on a single VLM (Kimi-K2.5), limiting transferability if the model changes or improves.
• SpreadsheetBench files used under research license and not redistributed, limiting dataset completeness and replication fidelity.
• Evaluation focused on one standard benchmark (OSWorld); cross-benchmark generalization or web/mobile desktop environments not explored.
• No detailed ablations on model capacity scaling, hyperparameter sensitivity, or long-term learning dynamics beyond one epoch.
• The single-model rollout depends heavily on the VLM’s competence; failures in task complexity not explicitly broken down.

Open questions / follow-ons

• How does the ProCUA-SFT fine-tuned model perform on adversarially constructed or out-of-distribution desktop tasks requiring novel applications or unseen UI elements?
• Can the single VLM approach scale to even larger models or ensembles without planner–actor gaps reemerging, especially for complex multi-modal goals?
• What is the impact of expanding ProCUA-SFT with more varied OS configurations (Windows, macOS) instead of primarily Linux snapshots?
• How would online or reinforcement learning methods complement ProCUA-SFT samples to improve recovery from error states or long-horizon dependencies?

Why it matters for bot defense

For bot-defense engineers developing CAPTCHA or bot detection systems, this paper introduces a crucial new method to synthesize large-scale, diverse, and realistic training data for complex GUI interaction agents. The grounding mechanism ensures that synthetic tasks correspond to achievable real desktop states, which is vital to prevent model overfitting on unrealistic or infeasible tasks. The step-prefix expansion technique effectively mimics the incremental visual and interaction context present in live environments, helping the model learn robust, stateful action policies.

Practitioners working on bot detection using desktop automation context can leverage the ProCUA-SFT dataset or its principles to train stronger supervised policies without reliance on noisy or limited human data. The single VLM approach closing the planner–actor gap is relevant to ensuring synthetic trajectories represent executable workflows, reducing false positives in bot behavior modeling. Understanding the detailed action distribution shifts and application usage in ProCUA-SFT can help reverse-engineer bot patterns or design CAPTCHAs that resist automated multi-application coordination. Additionally, the modular synthetic pipeline architecture suggests scalable ways to continually generate adversarial or diverse agent data for robustness evaluations.

Cite

@article{arxiv2606_17321,
  title={ ProCUA-SFT Technical Report },
  author={ Jaehun Jung and Ximing Lu and Brandon Cui and Muhammad Khalifa and Shaokun Zhang and Hao Zhang and Jin Xu and Amala Sanjay Deshmukh and Karan Sapra and Andrew Tao and Yejin Choi and Jan Kautz and Mingjie Liu and Yi Dong },
  journal={arXiv preprint arXiv:2606.17321},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.17321}
}

Read the full paper

• arXiv:2606.17321 (abstract)
• arXiv:2606.17321 (PDF)