Echoverse: Deep, Evolving Environments for Training Computer-Use Agents at Scale

Source: arXiv:2607.28074 · Published 2026-07-30 · By Yash Pandya, Sahil Gupta, Sarthak Harne, Archana Yadav, Kavyansh Chourasia, Hussein Mozannar et al.

TL;DR

Echoverse tackles the challenge of training computer-use agents that must interact with complex, stateful, and login-gated applications. Direct training on live systems is infeasible due to data privacy, statefulness, and reset difficulties. Prior work generated many synthetic environments, but Echoverse argues that environment quality — specifically "depth," capability targeting, and continuous co-evolution — is far more important than raw count. Echoverse builds deep, fully functional synthetic applications from specifications linked to real seed data, enabling graded tasks grounded in database state rather than visual inspection. A co-evolution loop repairs environment bugs informed by model failures, improving environment fidelity and training signal jointly. Trained on twelve deep environments, a 9B model nearly doubles accuracy on held-out splits (36.5% to 67.1%). Shallow environments were shown to degrade performance. Echoverse simultaneously supports supervised and reinforcement learning, achieving an RL policy that raises held-out scores from 58.8% to 68.0%. The authors release four runnable environments with seed data and verifiers as a benchmark to foster further research.

Key findings

• Training on twelve Echoverse deep environments improved a 9B model's accuracy from 36.5% to 67.1% across fourteen evaluation splits.
• On the same task distribution, shallow environments reduce live-site accuracy from 80.0% to 75.0%, whereas deep environments increase it to 85.0% and 65.0% in different domains.
• Repairing a single environment increased the trained model's accuracy on that environment's tasks from 16.2% to 38.5%.
• A reinforcement learning run on five Echoverse environments increased held-out task scores from 58.8% to 68.0% using a reward combining grounded verification with per-step judgment.
• Failures in graded rollouts were used to co-evolve the environment, its tasks, and verifiers — repairing defects before using the data for model training improved signal quality.
• The grounded verifier grades outcomes against database state and reference SQL queries, avoiding noisy, appearance-based judgments common in live web benchmarks.
• Depth is operationalized as an environment’s ability to fully support its target workflows end-to-end, including coherent state management and permission logic, rather than feature count or visual realism.
• Capability-targeted, narrow environments isolating specific failed interactions accelerated model improvements, which also generalized to held-out widget families and open-web tasks.

Threat model

The threat model assumes an offline adversary who attempts to train agents against synthetic environments that faithfully reproduce real applications’ backend logic and state. Adversaries may include model failures that arise from environment or verifier defects, potentially causing noisy or misleading supervision. The adversary does not have access to real login-gated backends or ground-truth state, and cannot tamper with the verification process itself. The system aims to preempt noisy or useless feedback by repairing environment defects before using failed rollouts for training, making the model robust only against realistic capability gaps rather than spurious bugs.

Methodology — deep read

Echoverse approaches training computer-use agents by constructing synthetic, stateful environments that replicate real applications’ backend logic, user interface, and database state. The threat model assumes that agents are trained offline without direct access to live, login-gated applications, which cannot be reset or graded reliably. Adversaries include models failing tasks due to environment defects but do not have access to the real backend or internal state.

Data provenance involves seed data from real datasets where available (e.g., InsideAirbnb for bookings, a large public forum corpus) or generated under constraints to produce dense, internally consistent synthetic data for domains like mail, calendar, banking, and health records. Environment construction begins with hand-written seed scenarios describing workflows, which are compiled into structured specifications and machine-checkable claims about routes, states, and behaviors.

The Echoverse factory pipeline consists of two phases. Phase 1 generates the synthetic environment application (React frontend, FastAPI backend, SQLite database) from the specification, then rigorously verifies and repairs the environment until >95% of machine-checkable claims pass. Phase 2 generates a task corpus regrounded on live database state, producing tasks with natural-language goals and reference outcomes from SQL queries. A panel of verifiers checks task feasibility and correctness by driving the running interface and querying the source and database directly.

Verification failures are triaged into issue groups attributed to the database, backend, frontend, task text, or verifier layers. Fixers then repair these defects and re-verify, iterating until >95% of tasks are verifiable and solvable. This produces a high-fidelity, grounded training corpus where task success is a property of application state changes, not just appearance.

Training uses a co-evolution loop where every graded rollout is analyzed twice. One pass uses model failures on clean tasks to generate new training signals; the other pass identifies environment/task/verifier defects to repair before using the data. This prioritizes environment repair before model-learning and keeps the training signal trustworthy.

Supervised fine-tuning uses a large teacher model (GPT-5.4) to generate demonstrations on the environments. Only trajectories verified by the grounded verifier are kept (verifier-filtered, rejection-sampled distillation). Reinforcement learning is applied without environment modification, using a composite reward combining the grounded verifier’s trajectory-level judgement with a dense per-step judge to overcome imitation learning ceilings. RL training on five environments raises held-out accuracy from 58.8% to 68.0%.

Evaluation occurs on fourteen held-out splits across multiple domains, including transfer tests to live WebVoyager benchmarks, showing gains with deep environments and the loop. The entire pipeline is agent-driven using GitHub Copilot SDK agents assigned builder, verifier, triager, and fixer roles, supported by tools like Playwright for UI automation and Model Context Protocol for state inspection. The authors release four runnable benchmark environments including code, seed data and grounded verifiers to support reproducibility and comparative research.

Technical innovations

• Operational definition of environment depth based on machine-checkable claims ensuring full workflow support and state coherence, rather than visual or feature-count proxies.
• Co-evolution loop reading every graded rollout twice: repairing environment, tasks, and verifier defects before using failures as model training signals.
• Construction pipeline generating fully functional, stateful web application environments from seed scenarios and specifications, with automated verification and repair at database, backend, frontend layers.
• Grounded verification grading task success against SQL-derived application state differences rather than textual or pixel-based judgements.
• Use of the same synthetic environments unmodified as reinforcement learning environments with a composite trajectory and per-step reward for improved policy training.

Datasets

• ECHOSTAY — size not explicitly stated but seeded from InsideAirbnb public dataset
• ECHOFORUM — 2.55 million public forum comments
• Synthetic datasets for mail, calendar, banking, health records generated under internal constraints

Baselines vs proposed

• Base model accuracy on live sites: 80.0% versus shallow environments: 75.0% and deep environments: 85.0% and 65.0%
• Model trained on single environment before repair: 16.2% accuracy versus after repair: 38.5%
• Reinforcement learning initial held-out score: 58.8% versus after training: 68.0%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.28074.

Fig 1: The learning loop. Every graded run is read twice. Failures that survive triage become model training

Fig 2: The environment factory. Phase 1 expands a handful of seeds into an application, then repairs the

Fig 3: The suite as an agent meets it. The first ten panels are the full-domain worlds tabulated above, each

Fig 4: Thematic spread of the two capability worlds. The same control is re-themed so that the skill cannot

Fig 5: Deep versus shallow worlds on two live WebVoyager domains, with the same domain coverage and

Fig 6 (page 10).

Fig 7 (page 10).

Fig 8 (page 10).

Limitations

• The model training and evaluation are conducted solely on synthetic environments; live-web evaluation is limited to transfer and may not capture all real-world complexities.
• While the methodology rigorously verifies and repairs environments, residual defects may still exist that impact training signals.
• The co-evolution loop relies on automated agent roles plus human supervision, which may limit scalability or introduce overhead not quantified in the paper.
• Reinforcement learning experiments are reported on only five of the environments; generalization to the full suite remains to be assessed.
• The teacher model sets an upper bound on supervised fine-tuning; improvements beyond that ceiling require more advanced RL or environment innovations.
• The released benchmark includes evaluation tasks only (not training), which limits direct comparability and full retraining experiments by external researchers.

Open questions / follow-ons

• How well do Echoverse-trained agents generalize to entirely new login-gated applications or workflows not covered by the seed specifications?
• Can the co-evolution loop be fully automated at scale without human supervision for environment repair and triage?
• What are the trade-offs in environment depth versus variety once model competency reaches higher levels—does adding many niche environments still help?
• How does reinforcement learning performance scale with more environments, more complex tasks, or use of multi-modal inputs beyond application state?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, Echoverse provides a compelling methodology for creating high-fidelity synthetic environments that simulate complex stateful applications behind authentication barriers, which are typically inaccessible for direct agent training. The emphasis on environment depth, coherence across workflows, and grounded verification minimizes noisy or adversarial signals that often plague live web benchmarks.

Practitioners building or evaluating CAPTCHA-breaking or bot-detection evasion agents can use Echoverse-like environments to train models more reliably on authentic interaction sequences involving permissions, multi-user state, and backend confirmation rather than superficial pixel-level signals. The co-evolution loop concept also provides a blueprint for continuously improving synthetic training worlds alongside model capabilities, allowing defender and attacker simulations to co-adapt. However, it is important to recognize the challenges in fully automating environment repair and scaling to diverse real-world domains. Echoverse’s benchmark offers concrete artifacts practitioners can adopt for rigorous evaluation of agent robustness on login-gated multi-step workflows.

Cite

@article{arxiv2607_28074,
  title={ Echoverse: Deep, Evolving Environments for Training Computer-Use Agents at Scale },
  author={ Yash Pandya and Sahil Gupta and Sarthak Harne and Archana Yadav and Kavyansh Chourasia and Hussein Mozannar and Vibhav Vineet and Sara Abdali and Corby Rosset and Yash Lara and Ahmed Awadallah and Ece Kamar and Akshay Nambi },
  journal={arXiv preprint arXiv:2607.28074},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.28074}
}

Read the full paper

• arXiv:2607.28074 (abstract)
• arXiv:2607.28074 (PDF)