TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents

Source: arXiv:2606.28480 · Published 2026-06-26 · By Shoufa Chen, Luyuan Wang, Xuan Yang, Zhiheng Liu, Yuren Cong, Yuanfeng Ji et al.

TL;DR

TUA-Bench is a newly introduced benchmark designed to evaluate the capabilities of large language model-powered agents that operate in terminal (command-line) environments across a broad range of real-world computer-use tasks. Unlike existing benchmarks that focus either on graphical user interfaces (GUIs) or on highly technical shell-related workflows, TUA-Bench covers 120 tasks spanning everyday productivity (document editing, email, spreadsheets), web information seeking, system/software management, multimedia, and complex scientific and engineering workflows developed with domain experts. Each task runs in a deterministic Linux container and is scored via execution-based verification to assess actual task completion rather than proxy metrics. Evaluation of top models and agent frameworks, such as Claude Code with Claude Opus 4.8, reveals a best overall success rate of 65.8%, demonstrating large remaining gaps in long-horizon planning, tool use, and error recovery in terminal environments. TUA-Bench thus fills a critical benchmark gap by providing a broad, realistic, and execution-grounded measure of general terminal-use agent capability across heterogeneous digital workflows.

Key findings

• TUA-Bench introduces 120 diverse terminal tasks covering 5 top-level categories: Office & Productivity (38.3%), Web & Information (18.3%), System & Software Operations (15.8%), Scientific & Engineering (14.2%), and Multimedia & Design (13.3%).
• The strongest agent, Claude Code with Claude Opus 4.8 max reasoning effort, achieves 65.8% mean success rate across all tasks, with 51.7% success on all 5 repeated runs, outperforming Codex GPT-5.5 at 64.7% overall.
• Within an agent framework (Terminus-2), the top models (GPT-5.5, Claude Opus 4.8, Claude Opus 4.7) achieve success rates around 58-60%, with Claude Opus 4.8 showing more consistent performance across trials (% all 5 runs success: 42.5% vs 31.7% for GPT-5.5).
• Increasing per-task runtime limit from 150s to 2400s improves success rate drastically from 33.0% to 60.1%, indicating many failures at low time budgets result from premature task termination rather than model inability.
• Scaling agent reasoning effort steadily improves success from 36.5% (no thinking) to 60.1% (xhigh), but marginal gains diminish at higher effort levels with a substantial increase in tokens used.
• Cost-performance tradeoff analysis shows prices range from ~$12 to >$170 per run, with diminishing returns near the top end; scaffold design impacts cost-efficiency significantly.
• TUA-Bench tasks were rigorously curated through conversion from GUI tasks (OSWorld) and collaboration with PhD experts for professional/scientific workflows, including difficulty-aware filtering to maintain sustained challenge against strong models.
• Agent performance varies substantially across task families and individual tasks, reflecting diverse reasoning, tool use, and domain knowledge challenges.

Threat model

N/A — The paper focuses on assessing autonomous terminal-use agent capabilities under controlled, benign settings. There is no explicit adversarial threat or malicious actor considered; the model assumes agents operate in isolated Linux containers under reproducible conditions without hostile interference.

Methodology — deep read

The research first defines the threat model as general AI agents aiming to autonomously perform real-world terminal tasks, without adversarial manipulation; the adversary model is not malicious but rather to measure agent competence and reliability across realistic digital workflows. TUA-Bench uses a curated set of 120 tasks derived from two streams: (1) everyday digital activities converted from the GUI-based OSWorld benchmark into terminal-only workflows, and (2) professional scientific and engineering workflows co-designed with domain experts across biology, medical physics, architectural and mechanical engineering. The datasets represent real task goals, inputs, and expected outputs, standardized in Linux container environments for deterministic execution. Each task includes a Dockerfile setup, task-specific input files, natural language instructions rewritten for terminal interaction, and automated verifiers for correctness (checking output file presence, format, value accuracy). The execution environment builds on the Harbor orchestration framework, enabling isolated, resettable Linux containers run with Docker or Podman. This infrastructure manages environment setup, control, logging, and verification, ensuring reproducibility and safety. A difficulty-aware curation pipeline removes ambiguous or inconsistent tasks by manual human inspection, removes overly simple tasks already solved by strong models, and filters to retain only those with low solvability scores based on evaluations from three frontier models. For evaluation, multiple state-of-the-art agent frameworks are tested including Claude Code, Terminus-2, Codex, OpenHands, and Mini-SWE-Agent, with a variety of underlying LLMs (GPT-5.5, Claude Opus 4.x, Gemini 3.1 Pro, etc.) to characterize model and agent-level performance. Agents autonomously execute tasks from start to finish, interacting solely via CLI commands. Five independent trials per task are conducted to capture stochastic variations and measure metrics such as mean success rate, Pass@1 (single run), Pass@5 (best of 5 runs), and All-5 (consistent success across all 5 runs). Ablation studies vary reasoning effort parameters, per-task time budgets, and cost-performance tradeoffs. The benchmark, task definitions, and evaluation code are open-sourced on GitHub, facilitating reproducibility. Overall, the benchmark provides rigorous, execution-grounded, multi-domain assessment of agent planning, reasoning, tool use, execution monitoring, and error recovery capabilities in complex terminal workflows. One concrete example is a document editing task where agents receive a textual CLI instruction, manipulate input files using terminal commands or scripts within a containerized environment, and success is scored by programmatically comparing final file outputs against ground truth to verify correctness.

Technical innovations

• TUA-Bench introduces the first general-purpose benchmark for terminal-use agents covering both everyday digital tasks and professional scientific workflows across multiple domains.
• Conversion of GUI-based real-world tasks into command-line-only workflows preserving user intent and broadening terminal interface applicability.
• Use of deterministic containerized Linux environments coupled with executable verification scripts for reproducible and robust execution-based scoring of agent performance.
• Difficulty-aware task curation leveraging aggregated solvability metrics from multiple strong frontier models to maintain benchmark challenge and discriminative power.
• Comprehensive evaluation framework integrating multiple agent scaffolds and a wide range of LLMs while analyzing cost-performance and reasoning-effort tradeoffs explicitly.

Datasets

• TUA-Bench task set — 120 tasks — curated and reproducible containerized environments combining everyday digital tasks and professional domain-specific workflows
• OSWorld — 369 GUI-based tasks — source for conversion of everyday digital workflow tasks

Baselines vs proposed

• Terminus-2 + GPT-5.5: success rate = 60.1% vs Terminus-2 + Claude Opus 4.8 = 59.7%
• Claude Code + Claude Opus 4.8 max: success rate = 65.8% vs Codex + GPT-5.5 xhigh = 64.7%
• Claude Opus 4.8 All-5 consistency = 42.5% vs GPT-5.5 All-5 = 31.7% within Terminus-2 agent
• Gemini 3.1 Pro Preview: 49.3% success vs GLM-5.1: 48.1% vs MiniMax-M3: 47.0%
• Top mid-tier models range 44.9%-49.3%, clearly separated from frontier models (58-60%)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.28480.

Fig 1: Overview of TUA-Bench. TUA-Bench evaluates terminal-use agents on realistic, application-grounded tasks

Fig 2: TUA-Bench task distribution. The 120

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• The benchmark relies on deterministic containerized execution which may not capture all variability present in real-world user systems or asynchronous interfaces.
• Current tasks exclude adversarial or hostile environments, so robustness under adversarial attack or deceptive inputs is not evaluated.
• Some professional scientific tasks require specialized domain knowledge that may bias results towards models pretrained or fine-tuned on related data.
• The time budgets and reasoning effort settings, while extensive, may not reflect real-time constraints in production terminal automation.
• The heuristic difficulty-aware task selection depends on current models and may need updating as agent capabilities improve.
• Internet access during tasks introduces possible nondeterminism and external dependency effects that complicate reproducibility.

Open questions / follow-ons

• How can terminal-use agents better handle long-horizon planning and error recovery in environments requiring interleaved reasoning and execution control?
• What strategies enable improved generalization across vastly different task domains, especially between everyday digital and specialized scientific workflows?
• How does integration of limited visual or multimodal input (e.g., screenshots, plots) affect terminal agent generality and robustness?
• Can reinforcement learning or online adaptation improve cost-efficiency or reduce runtime without compromising task success under real-time constraints?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, TUA-Bench highlights the growing capability of AI agents to perform complex, general-purpose terminal-based workflows autonomously, beyond narrow coding or shell scripting tasks. This broad competence across command-line interfaces suggests adversarial bots or automated adversaries could leverage terminal-agent technologies to interact with backend systems, manipulate files, or perform command-driven exploits more reliably than before. Understanding TUA-Bench’s methodology and evaluation metrics helps CAPTCHA engineers appreciate performance bottlenecks—in long-horizon planning, error handling, and execution monitoring—where defense strategies could focus to detect or thwart automation attempts. Moreover, the benchmark’s emphasis on execution-grounded verification and containerized isolation outlines a rigorous framework for evaluating agent authenticity and behavior that bot-defense systems might adapt to discern genuine human usage patterns versus advanced automated terminals. Finally, the open sharing of TUA-Bench’s tasks and evaluation protocols enables security researchers to develop tailored challenge-response workflows simulated at the terminal level, potentially augmenting CAPTCHA mechanisms that traditionally focus on GUIs or web-based bot detection.

Cite

@article{arxiv2606_28480,
  title={ TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents },
  author={ Shoufa Chen and Luyuan Wang and Xuan Yang and Zhiheng Liu and Yuren Cong and Yuanfeng Ji and Feiyan Zhou and Xiaohui Zhang and Fanny Yang and Belinda Zeng },
  journal={arXiv preprint arXiv:2606.28480},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.28480}
}

Read the full paper

• arXiv:2606.28480 (abstract)
• arXiv:2606.28480 (PDF)