Fara-1.5: Scalable Learning Environments for Computer Use Agents

Source: arXiv:2606.20785 · Published 2026-06-18 · By Ahmed Awadallah, Sahil Gupta, Yash Lara, Yadong Lu, Hussein Mozannar, Akshay Nambi et al.

TL;DR

This paper addresses the critical bottleneck in training computer use agents (CUAs) for web browsers: the scarcity and high cost of collecting high-quality human demonstration data in realistic environments. To overcome this, the authors introduce FaraGen1.5, a comprehensive, scalable data generation pipeline that integrates two types of environments—live websites and fully sandboxed synthetic web replicas—and a single-agent GPT-5.4-powered solver with a multi-turn user simulator. The pipeline also integrates three orthogonal verifiers to filter trajectories for task correctness, efficiency, and correct user interaction at critical points, ensuring safe and reliable training data. Using datasets produced by this pipeline, they train the Fara1.5 family of native CUAs at multiple scales (4B, 9B, and 27B parameters based on Qwen3.5) using a carefully balanced supervised finetuning approach.

The results demonstrate that Fara1.5 models establish new state-of-the-art performance for their size classes on prominent benchmarks like Online-Mind2Web and WebVoyager. Notably, Fara1.5-9B achieves 63.4% task success on Online-Mind2Web (a +29.3 point gain over the prior Fara-7B and +14.8 over the previous best GUI-Owl-1.5-8B), while Fara1.5-27B reaches 72.3%, competitive with much larger proprietary agents. The use of synthetic environments expanded the range of tasks to include authenticated, irreversible, and stateful interactions impossible or unsafe on live web. Improvements in the solver design and verifier robustness further improved the quality and utility of generated data, enabling more natural, multi-turn, user-interactive trajectories for training. Overall, FaraGen1.5 combined with iterative finetuning yields practical, strong native CUAs operating in a pixel-and-screenshot-only observe-think-act paradigm.

Key findings

• Fara1.5-9B achieves 63.4% task success on Online-Mind2Web, a +29.3 point absolute increase over Fara-7B and +14.8 over GUI-Owl-1.5-8B.
• Fara1.5-9B reaches 86.6% task success on WebVoyager, setting a new state of the art for 8–9B parameter CUAs.
• Fara1.5-27B attains 72.3% task success on Online-Mind2Web, competitive with much larger commercial systems like Gemini 2.5 and Yutori n1.
• Replacing the multi-agent solver with a single GPT-5.4-powered solver raises end-to-end success on Online-Mind2Web from 67% to 83%, a +16 point absolute gain.
• The triple-verifier pipeline (task correctness ≥ 0.8 rubric score, efficiency ≥ 4/5 rating, and critical-point adherence) ensures high-quality verified trajectories.
• Synthetic environments constitute 12.8% of the final training data and enable tasks involving authentication and irreversible side effects that live websites cannot support.
• FaraGen1.5 dataset includes 1.57M trajectory steps collected by May 2026, with 60.0% from live web, 12.8% synthetic environments, and 12.5% form-filling/user interaction tasks.
• Task efficiency verifier removes inefficiencies, reducing redundant or looping actions in training data, encouraging faster inference behavior.

Threat model

The adversary is considered an untrusted internet environment that may have inaccessible or changing backends and policies restricting agent actions. The system assumes the agent cannot safely perform irreversible or credentialed actions on live websites without explicit safeguards. Adversaries cannot access internal synthetic environments which are fully sandboxed. The threat model excludes active adversaries that deliberately manipulate UI or supply adversarial inputs to mislead the agent or verifiers.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is not explicitly defined as a malicious attacker; rather the focus is on building robust CUAs that act safely and correctly in web environments. The system assumes control over synthetic environments and limited/no control over live websites. It prevents unsafe or irreversible actions on live webs by operator oversight and input gating. Adversarial inputs or manipulations of the environment are not directly studied.

2. Data: FaraGen1.5 pipeline generates training data by creating tasks in two environment types: live websites seeded from public URL indexes and synthetic sandboxed environments replicating complex apps with real backend state. Task proposals are generated by LLMs conditioned on website summaries or synthetic environment state. The task solver (GPT-5.4) acts on the environments in a multi-turn tool-calling loop cooperating with a user simulator that answers clarifications and grants permission at critical points. The pipeline collected approximately 1.57 million trajectory steps by May 2026.

3. Architecture/Algorithm: The solver is a single GPT-5.4 agent running a multi-turn tool-calling policy emulating the final model’s action space (pixel-based clicks, typing, scrolling, meta-actions like ask_user_question and pause_and_memorize_fact). The solver is restricted from actions hard for student replication and disallowed from unsafe actions on live webs. The user simulator interacts with the solver to resolve missing info or confirm critical points. Three verifiers evaluate trajectories for task correctness (LLM rubric on live web, backend state diffs on synthetic), efficiency (detect redundant or looping actions with a 1-5 rating LLM judge), and critical point adherence (ensuring user permission before irreversible steps).

4. Training Regime: Using only verified trajectories, the authors train Fara1.5 models at 4B, 9B, and 27B parameter scales based on Qwen3.5 backbone. Training uses supervised finetuning, balancing FaraGen1.5 data for broad domain coverage, specialized high-value tasks, and iterative correction of model deficiencies. Input is raw screenshots (up to 3 recent) plus trimmed URL prefix, no DOM or accessibility tree. The model predicts chain-of-thought reasoning and one atomic action per step. Training details such as batch size, epochs, optimizer are not fully specified in the paper.

5. Evaluation Protocol: CUA performance measured on widely used benchmarks Online-Mind2Web (300 tasks, 136 sites) and WebVoyager. Metrics are task success rates. The new solver system is benchmarked under WebJudge automated evaluation. Comparisons include prior Fara-7B, GUI-Owl-1.5-8B, MolmoWeb-8B, and Holo2-8B. Ablations include removing the multi-agent system in favor of single policy, using synthetic envs vs online only, and verifier filtering impact. User studies also measure smoothness and user control.

6. Reproducibility: Code for the Fara pipeline and environments is available on GitHub (https://github.com/microsoft/fara). The Fara1.5 models and data mixes are partly available, though some datasets like Online-Mind2Web are cited papers external to this one. Exact hyperparameters and random seeds are not fully disclosed. The pipeline emphasizes transparency via data filtering logs and gating for safety auditing.

Concrete Example: For a task like booking a flight on a live website, the solver uses the environment summary and task prompt to navigate the UI by pixel clicks and typing, asking the user simulator for missing PII or missing parameters when needed. Before submitting an irreversible purchase, the critical-point verifier checks whether explicit permission was granted. Only if all verifiers pass, the trajectory is included for training. The native Fara1.5 student then learns from multiple such diverse, verified trajectories across live and synthetic tasks.

Technical innovations

• FaraGen1.5 introduces synthetic, fully sandboxed web replica environments with full backend observability enabling safe irreversible action tasks and ground-truth execution verification.
• A single-agent GPT-5.4-based multi-turn tool-calling solver replaces prior multi-agent orchestration, improving end-to-end task success by +16 points and reducing distribution shift with students.
• A triple-verification system combines task correctness (rubric or backend state), task efficiency (action redundancy detection), and critical-point user interaction adherence for robust filtering of training trajectories.
• User simulator co-plays multi-turn dialogues and clarifications with the solver, providing richer, interactive training trajectories beyond single-turn task demonstrations.
• Training native CUAs that operate solely on screenshots plus short URL context for observation, supporting pixel-level action prediction without DOM or accessibility trees.

Datasets

• Online-Mind2Web — 300 tasks across 136 popular live web domains
• FaraGen1.5 generated trajectories — 1.57 million steps by May 2026 (60% live web, 12.8% synthetic, 12.5% form-fill/user interaction)
• Grounding datasets: Jedi, GroundCUA, Click100k (filtered for quality with GPT-5.2)
• Visual QA datasets: RICO-SCA, InfographicsVQA

Baselines vs proposed

• Fara-7B (previous generation): Online-Mind2Web task success = 34.1% vs Fara1.5-9B = 63.4% (+29.3 points)
• GUI-Owl-1.5-8B: Online-Mind2Web = 48.6% vs Fara1.5-9B = 63.4% (+14.8 points)
• MolmoWeb-8B: Online-Mind2Web = 35.3% vs Fara1.5-9B = 63.4%
• Multi-agent solver (FaraGen): Online-Mind2Web = 67% vs GPT-5.4 single-agent solver = 83% (+16 points)
• Fara1.5-27B: Online-Mind2Web = 72.3% vs Gemini 2.5 Computer Use (proprietary, larger models) = comparable performance

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.20785.

Fig 1: Task success rate on Online-Mind2Web and WebVoyager for similarly sized CUA models. Fara1.5-9B reaches

Fig 2: FaraGen1.5 pipeline. Phase 1 instantiates tasks in two kinds of environments: online, seeded by URLs from

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• Limited details on training hyperparameters, optimizer, and exact dataset splits reduce reproducibility clarity.
• Evaluation focuses primarily on Online-Mind2Web and WebVoyager benchmarks; generalization to unseen domains or adversarial tasks not measured.
• Safety gating currently restricts irreversible actions on live web but no adversarial robustness testing against malicious sites or inputs.
• Training data still heavily weighted to web environments with 60% live web; synthetic environments cover only 12.8%, limiting domain diversity.
• Verification relies heavily on LLM judges which may still hallucinate or misclassify rare edge cases despite multiperspective checks.
• While pixel-based observation enables broad applicability, it abstracts from semantic web structure and may limit interpretability.

Open questions / follow-ons

• How well do Fara1.5 agents generalize to previously unseen websites with novel UI layouts or interaction paradigms?
• Can the verifier system be made fully automated and adversarially robust without reliance on LLM judges or human review?
• What is the impact of incorporating richer environment metadata (e.g., DOM, accessibility trees) in combination with pixel input?
• How does multi-turn interaction with real users beyond simulation shape agent performance and safety in deployment?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, Fara-1.5’s advancements illustrate that state-of-the-art web navigation agents are becoming increasingly capable of interacting robustly on real and synthetic web platforms. The use of sandboxed synthetic environments for controlled, verifiable interaction and the integration of multi-turn user simulation highlight evolving methodologies for modeling complex user-agent interactions that can mimic human behavior closely. The triple-verifier approach to trajectory validation addresses common failure modes such as inefficiencies or unauthorized irreversible actions, providing a blueprint for filtering and assessing legitimate versus anomalous behavior. Practitioners should be aware that modern CUAs may be trained on sophisticated pipelines that combine live web and synthetic data to improve generalization, necessitating more nuanced bot-detection and CAPTCHA challenges that consider both behavioral signals and permission gating. The pixel-observation-only model approach indicates that agents need not rely on structural web metadata, complicating detection techniques relying on DOM analysis. Overall, understanding the capabilities and safeguards in such native CUAs informs the design of resilient hands-off bot detection and interaction integrity systems.

Cite

@article{arxiv2606_20785,
  title={ Fara-1.5: Scalable Learning Environments for Computer Use Agents },
  author={ Ahmed Awadallah and Sahil Gupta and Yash Lara and Yadong Lu and Hussein Mozannar and Akshay Nambi and Zach Nussbaum and Yash Pandya and Aravind Rajeswaran and Corby Rosset and Alexey Taymanov and Luiz do Valle and Vibhav Vineet and Spencer Whitehead and Andrew Zhao },
  journal={arXiv preprint arXiv:2606.20785},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.20785}
}

Read the full paper

• arXiv:2606.20785 (abstract)
• arXiv:2606.20785 (PDF)