OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios

Source: arXiv:2607.14989 · Published 2026-07-16 · By Chengyu Shen, Yujie Fu, Gangtao Xin, Yanheng Hou, Wenlong Fei, Guojie Zhu et al.

TL;DR

OmniaBench addresses a critical evaluation gap for general AI agents by providing a large-scale, scenario-rich benchmark covering a wide spectrum of real-world application domains and multi-turn interactive settings. Existing agent benchmarks typically focus on narrow tool ecosystems or limited task formats, making it challenging to characterize agent capabilities across heterogeneous tasks encountered in consumer (ToC), business (ToB), and employee (ToE) environments. OmniaBench derives a hierarchical taxonomy spanning 90 level-1 and 354 level-2 domains from app stores, product docs, and web resources, instantiated as executable environments with stateful tools and entities. Tasks are synthesized via four complementary routes (DAG, DAG-S, Solver, Program) covering single- and multi-turn interactions and verified with rubric-based and programmatic criteria. A ten-dimensional capability taxonomy and eight atomic difficulty factors enable fine-grained diagnostic evaluation.

Key findings

• OmniaBench dataset contains 1,431 tasks across 90 level-1 and 354 level-2 domains, with a challenging subset of 644 tasks designed to control evaluation cost and contamination.
• Even top-performing models Claude-Sonnet-5 and GPT-5.6-Sol achieve Overall Pass@1 scores of only 58.54% and 57.14%, respectively, indicating substantial evaluation difficulty.
• Task complexity involves an average of 59 tools, 13.7 entity tables, and 125 entities per environment, with tool dependency graphs supporting multi-turn stateful interaction.
• OmniaBench uses a ten-dimensional capability taxonomy (Task Understanding, Info Gathering, Planning & Decision Making, State Management, Tool Use, Code & Prog Ops, Data Analysis, Office & Doc Handling, Interactive Collaboration, Reliability & Safety) to dissect model strengths and weaknesses.
• Performance varies significantly across capabilities and domains, with planning, constraint maintenance, and adaptive correction consistently challenging.
• Models require sustained reasoning over dozens of tool calls per task (average ToolSteps ~48-64), reflecting long-horizon interactive task complexity.
• The benchmark integrates multi-turn user simulators for dialog interactions and controlled sandboxes for safe file-system/code execution.
• Open-source and closed-source models show a wide performance range with stable leaderboard rankings and reproducible evaluation protocols.

Threat model

OmniaBench targets evaluation of general AI agents interacting with complex environments; the implicit adversary is a challenging task environment designed to test model capabilities including partial observability, multi-step planning, and error handling. The benchmark assumes agents lack perfect state knowledge and must operate under uncertainty. It does not directly consider active adversarial agents attempting to deceive or subvert the system.

Methodology — deep read

OmniaBench frames general agent evaluation as a partially observable Markov decision process (POMDP) with latent states representing full task configurations and observations representing partial views (user messages, tool responses, file states). Agents act through heterogeneous actions including language outputs, tool calls, and task finalization steps. Environmental transitions and observations evolve based on agent actions, incorporating multi-turn user feedback and dialog for iterative task completion.

The authors construct a comprehensive scenario taxonomy from multiple real-world sources such as app stores, industry workflows, web retrieval, and human annotation, covering 90 level-1 and 354 level-2 domains across consumer, business, and employee settings. They synthesize executable environments by extracting entities, attributes, tools/APIs, and domain constraints, implemented as Python environment classes with tooling functions and sandboxed programming interfaces for safe evaluation.

Task synthesis uses four complementary routes: DAG (tool dependency graphs for multi-turn interaction), DAG-S (single-turn tasks derived from DAG with query refinement), Solver (optimization and scheduling scenarios using solver-guided synthesis), and Program (procedural reasoning, branching, iteration with verifyable execution). Tasks are rigorously validated via syntax checks, tool-call rollouts, and semantic consistency. Human annotation refines environment/task quality, solvability, and evaluation rubric design.

Evaluation employs a unified trajectory-based framework using multi-item rubrics for task completion criteria or VerifyCode for programmatic pass/fail signals. Multi-turn tests use persona-grounded user simulators controlling interaction complexity. Models are compared via Pass@1 success on a challenging 644-task subset, with detailed breakdowns by capability dimension, domain taxonomy, and difficulty factors. Leaderboard stability is assessed by rank correlation and coefficient of variation across user simulators.

Experiments involve a broad suite of closed-source (e.g., Claude-Sonnet-5, GPT-5.6-Sol) and open-source models evaluated via provider APIs with standardized temperature and reasoning-effort settings. Trajectories support up to 200 tool calls with retries for robustness. Consistent OpenAI function-calling schemas ensure evaluation comparability.

End-to-end example: For a DAG-route multi-turn task, an agent receives partial observations including user requests and tool states, chooses actions invoking a sequence of domain-specific tools according to a sampled dependency graph, updates latent state based on tool outputs, queries user feedback for clarifications, and iteratively refines the plan until task completion or maximum step limit, with final evaluation against rubric criteria or VerifyCode.

Technical innovations

• Integration of a large-scale, hierarchical scenario taxonomy spanning 90 level-1 and 354 level-2 domains from heterogeneous real-world sources for broad agent evaluation coverage.
• Construction of multi-turn executable environments with explicit state spaces combining tool APIs, entities, and user simulators to support complex interaction and state management.
• Introduction of a ten-dimensional capability taxonomy for diagnostic agent evaluation, capturing diverse skills from planning to collaboration and safety.
• Development of a compositional atomic difficulty framework modeling eight fundamental challenge factors driving task complexity and interaction difficulty.
• Synthesis of complementary task generation routes (DAG, DAG-S, Solver, Program) with different execution paradigms, enabling diverse assessment scenarios within one benchmark.

Datasets

• OmniaBench full set — 1,431 tasks — constructed from app stores, product docs, web retrieval, and human refinement
• OmniaBench challenging subset — 644 tasks — cost-controlled subset selected for leaderboard evaluation

Baselines vs proposed

• Claude-Sonnet-5: Overall Pass@1 = 58.54% vs GPT-5.6-Sol: 57.14%
• GLM-5.2: Overall Pass@1 = 56.83% vs GPT-5.5: 56.52%
• Qwen3.7-Max: Overall Pass@1 = 49.69% vs DeepSeek-V4-Pro: 54.50%
• Doubao-Seed-2.0-Pro: Overall Pass@1 = 23.91% vs GPT-5.4-Mini: 17.24%
• Open-source GLM-5.2 performs comparably to some closed-source models on OmniaBench
• Pass@1 scores vary substantially across task routes: DAG range ~50-57%, Solver range ~7-10%, Program 36-65%, DAG-S 30-63%
• Domain splits show ToC tasks scoring around 59-64%, ToB 35-60%, ToE 38-60% with variation per model

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.14989.

Fig 1: OmniaBench leaderboard. Overall performances of evaluated models on the challenging set.

Fig 2: Conceptual overview of OmniaBench. The figure summarizes the key components of the benchmark,

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• Evaluation relies on synthetic tasks constructed from domain knowledge with limited real-world user interaction validation.
• User simulators may fail to fully model realistic human behavior diversity or adversarial user strategies.
• No reported adversarial robustness evaluation under active exploitation attempts by hostile agents.
• Distribution shift and generalization to novel or unseen domains outside the constructed taxonomy remain untested.
• Rubric-based scoring may introduce subjectivity despite attempts for stability and consistency.
• Large-scale multi-turn evaluations may be computationally expensive, limiting accessibility for smaller-scale research.

Open questions / follow-ons

• How do general agents perform under adversarial or noisy user interactions beyond the persona-grounded simulators?
• What are effective architectural or training approaches to address OmniaBench-identified weaknesses in planning, constraint maintenance, and adaptive correction?
• Can the atomic difficulty framework guide automatic curriculum learning to improve agent robustness incrementally?
• How well do current agents generalize to domains or interaction patterns not covered by the OmniaBench taxonomy?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, OmniaBench offers a comprehensive evaluation framework for assessing the limits of general AI agents in realistic interactive settings involving multiple tools, multi-turn dialogs, and evolving stateful tasks. Understanding model capabilities along ten detailed dimensions helps diagnose vulnerabilities such as poor planning or constraint handling that bots might exploit or fail to circumvent. The explicit stateful environments and multi-modal actions provide ideas for designing CAPTCHA-like challenges that require coordinated tool use or multi-step reasoning beyond straightforward pattern recognition. Additionally, the compositional difficulty factors could inspire adaptive challenge generation to increase robustness against evolving automated attacks. Although not focused on direct adversarial resistance, the benchmark's insights on agent limitations can inform bot-defense strategies to detect or mitigate sophisticated agentic behaviors attempting to mimic human workflows.

Cite

@article{arxiv2607_14989,
  title={ OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios },
  author={ Chengyu Shen and Yujie Fu and Gangtao Xin and Yanheng Hou and Wenlong Fei and Guojie Zhu and Jiawei Li and Hongcheng Gao and Runming He and Zhen Hao Wong and Meiyi Qiang and Hao Liang and Zhao Cao and Hao Jiang and Chong Chen and Wentao Zhang },
  journal={arXiv preprint arXiv:2607.14989},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.14989}
}

Read the full paper

• arXiv:2607.14989 (abstract)
• arXiv:2607.14989 (PDF)