Omni-Decision: A Progressive Evidence-State Agent System for Omni-Modal QA

Source: arXiv:2607.11433 · Published 2026-07-13 · By Ming Ma, Yi Zhu, Yiran Zhong, Feida Zhu, Weigao Sun, Junhan Shi et al.

TL;DR

Omni-Decision addresses the challenge of omni-modal evidence-seeking question answering (QA), where relevant evidence is sparsely distributed across heterogeneous sources such as videos, audio, images, web pages, and computational results. Existing agentic systems typically use unstructured memories like scratchpads or free-form tool call histories, which makes it difficult to track what evidence has been grounded, what remains missing, and when enough evidence has been collected to answer a query. Omni-Decision introduces a training-free, query-scoped, structured evidence state that explicitly tracks confirmed evidence, unresolved conflicts, fact and computation dependencies, and open evidence needs. This shared state conditions the agent's planning, evidence acquisition, verification, repair, and stopping decisions, enabling deterministic state updates from heterogeneous observations. The design provides an inspectable and controllable loop for multi-step omni-modal evidence collection, overcoming limitations of implicit or free-form memory strategies.

Empirically, Omni-Decision improves substantially over strong baselines on two challenging benchmarks: it achieves 45.6% accuracy on OmniGAIA and 58.3% on WorldSense—improvements of +27.3 and +30.2 percentage points respectively over prior agents. Ablations removing the evidence state reveal a significant accuracy drop (-9 to -10 points), supporting the key role of explicit evidence-state control. Audits of evidence-chain progress show the approach meaningfully advances partial evidence collection even on incorrect cases, highlighting the system’s interpretability and diagnostic value. Overall, the work reframes omni-modal QA as a progressive evidence-closure process controlled by a shared, structured state, which improves planning and integration of multimodal evidence for long-horizon question answering.

Key findings

• Omni-Decision achieves 45.6% accuracy on the OmniGAIA benchmark, improving over the baseline agent by +27.23 percentage points overall (18.33% to 45.56%).
• Accuracy gains on OmniGAIA persist across difficulty splits with +30.33 points on Easy, +26.25 points on Medium, and +24.36 points on Hard questions.
• Removing the evidence state in ablation experiments drops accuracy by 9.17, 10.83, and 3.33 points for default, Qwen perception, and Qwen planner+perception settings respectively on the same fixed 120-case subset.
• Omni-Decision reaches 58.3% accuracy on WorldSense, a multiple-choice audio-video QA benchmark, improving over prior agent baselines by +30.2 points.
• On a 40-case audit of medium/hard OmniGAIA questions with human subgoal annotations, 14 (35%) runs achieved full evidence-chain completion with validated answers; 19 (47.5%) achieved partial progress; 1 (2.5%) was misvalidated; and 6 (15%) made zero measurable progress.
• The evidence state explicitly separates confirmed evidence, unresolved conflicts, fact/computation dependencies, and open needs, enabling deterministic state updates that preserve conflicting evidence rather than overwriting.
• The state-conditioned planner uses a bounded feasibility test to decide if available actions can plausibly advance evidence state, allowing it to stop with an 'insufficient' response when no action can help.
• Omni-Decision acts as a centralized shared control object consumed by planner, critic, and finalizer, with a single reducer component as the only writer updating the evidence state.

Threat model

Not applicable; the paper does not explicitly consider adversarial threat models. The challenges addressed are inherent to multi-modal evidence sparsity, partial observability, and multi-step reasoning in agentic QA systems rather than attacks or malicious adversaries.

Methodology — deep read

The core methodology introduces a query-scoped evidence state maintained per question, which explicitly tracks various aspects of evidence closure needed to answer omni-modal queries. The system architecture and inference loop work as follows:

1. Threat Model & Assumptions: The work does not focus on adversarial threat models but rather on the system design to improve multi-step evidence gathering and reasoning. The 'adversary' here is the inherent difficulty of sparse, distributed, multi-modal evidence that is partially observable and requires multi-stage reasoning.

2. Data: The benchmarks used are OmniGAIA, emphasizing open-world, multi-modal evidence with tool use (web, browsing, computation), and WorldSense, which focuses on self-contained audio-video multiple-choice QA. Data provenance is from public leaderboards; experiments use standard splits. Labels are QA pairs with answer correctness annotations. The system uses official evaluation protocols.

3. Architecture/Algorithm: Omni-Decision maintains an explicit evidence state S_t = {E_t, C_t, F_t, U_t} per query, where E_t contains confirmed evidence atoms with sources, C_t unresolved conflicts, F_t tracks external facts and computation results, and U_t lists open evidence needs. An immutable context R_0 packages fixed info (query, assets, tool availability). At each step, a state digest d_t = state_digest(R_0, S_t) summarizes current evidence closure. The planner takes d_t and chooses actions from an action space A(R_0), including media grounding, retrieval, browsing, computation, verification, or finish. Each tool action returns an observation o_t; a critic returns a verdict v_t. A reducer commits normalized, typed state events (from observations or verdicts) to update S_t+1 = S_t ⊕ ΔS_t by deterministic field-wise updates (append to E, update U and F, record conflicts in C). The finish action is gated on an evidence-ready state where all evidence needs are closed, dependencies resolved, and no conflicts exist. If no actions can plausibly advance the state, inference returns an insufficient status.

4. Training Regime: Omni-Decision is a training-free inference system leveraging pretrained LLMs (e.g., gpt-5.2-2025-12-11 for planner, critic, finalizer) and perception backends (gemini-3.1-pro, qwen3-omni-flash). It orchestrates tools and LLM modules at inference time without new model training, relying on prompt conditioning with the evidence state digest.

5. Evaluation Protocol: Accuracy is measured against established official benchmarks OmniGAIA (open-world QA requiring tool use) and WorldSense (self-contained audio-video MCQ). The system is evaluated on full test sets with category/difficulty splits. Ablations remove the evidence state to isolate its impact. Additional audits involve manual subgoal annotations decomposing evidence chains to classify partial/full/no progress and analyze failure modes. Controlled diagnostics vary planners and perception backends to analyze separate contributions.

6. Reproducibility: The paper does not indicate a public code or model release; dataset splits and model choices align with public leaderboards. Exact prompt templates, reducer rules, and runtime fields are documented in appendices. The system uses proprietary LLMs and perception APIs, limiting direct exact replication but enabling conceptual reproduction.

Example end-to-end usage (Fig 1): Given a query about the month difference between Instagram launch and a watch brand's first Instagram post, the system initializes S0 with open evidence needs (brand grounding, Instagram launch date, first post date, computation of month difference). It grounds the brand from the video, fills missing factual slots via web search and verification, computes the month difference, updates the evidence state accordingly each step, and finalizes the answer when all needs are satisfied.

Technical innovations

• Formulating omni-modal evidence-seeking QA as query-scoped evidence-closure with an explicit, structured evidence state rather than implicit dialogue history or free-form tool call trajectories.
• Designing a training-free evidence-state system that uses deterministic field-wise updates via a reducer to commit normalized multi-modal observations and critic verdicts into a shared evidence state.
• Integrating heterogeneous evidence sources (video, audio, images, web, computation) into a unified state interface conditioning planning, evidence acquisition, verification, repair, and stopping decisions.
• Using a state-conditioned control loop with a planner that consults a bounded state digest, enabling meaningful action feasibility checks and explicit handling of unresolved conflicts and open evidence needs.

Datasets

• OmniGAIA — size unspecified (benchmark for open-world omni-modal QA with videos, web, tools) — public benchmark
• WorldSense — 3,172 examples (self-contained audio-video multiple-choice QA) — public benchmark

Baselines vs proposed

• OmniGAIA base agent: 18.33% overall accuracy vs Omni-Decision: 45.56%
• Minimal agent baseline: 5.56% accuracy vs Omni-Decision: 45.56% on OmniGAIA
• OmniAgent: 25.51% accuracy vs Omni-Decision: 45.56% on OmniGAIA
• Removing evidence state lowers accuracy by 9.17 points (from 41.67% to 32.50%) in default planner/perception setting on 120-case subset
• WorldSense native best end-to-end model Gemini-3.1-Pro-Preview: 65.5% vs Omni-Decision: 58.32%
• OmniAgent on WorldSense: 28.06% vs Omni-Decision: 58.32%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.11433.

Fig 1: Omni-Decision on one OmniGAIA task. The system first turns the question into open

Fig 2: Overview of the Omni-Decision inference loop. Given the immutable context R0 con-

Fig 3 (page 3).

Fig 4 (page 3).

Fig 5 (page 3).

Fig 6 (page 3).

Fig 7 (page 3).

Fig 8 (page 3).

Limitations

• No learning-based policy or training is applied; system depends on large pretrained models and discrete deterministic reducer rules.
• Evaluation limited to two benchmark datasets (OmniGAIA and WorldSense) that do not cover all real-world complexities like private media, dynamic web, or personal histories.
• No adversarial robustness or misuse threat model considered; focus is on system-level evidence organization rather than security.
• Relies on external perception and backend tools whose errors block progress; the evidence state does not replace or improve low-level perception.
• No formal statistical significance tests on ablation studies; diagnostic subsets only.
• Limited evaluation on process-level behaviors like follow-up question generation, uncertainty localization, or abstention beyond final-answer accuracy.

Open questions / follow-ons

• How might learned policies or reinforcement learning incorporate and optimize over the query-scoped evidence state for improved multi-step omni-modal QA?
• Can the evidence state interface be extended to handle dynamic or interactive multi-turn dialogues where evidence needs evolve over time?
• How to integrate calibrated uncertainty estimates and perception confidence into the evidence state to better handle noisy or ambiguous observations?
• What richer evaluation metrics and benchmarks can be designed to assess evidence coverage, verification robustness, and abstention behavior in omni-modal QA?

Why it matters for bot defense

For bot-defense and CAPTCHA engineers, Omni-Decision's approach highlights the importance of an explicit, structured evidence state to track multi-step, multi-source information integration and verification. Rather than relying on unstructured or implicit memory, a clear query-scoped state allows fine-grained control over evidence acquisition, conflict resolution, and stopping decisions. This can improve system interpretability, diagnose failure modes, and guide resource allocation—critical for robust automated challenge-response systems that may incorporate heterogeneous signals such as images, audio, and web data. Insights on deterministic reducer updates and state-conditioned planning could inspire more rigorous state management in bot-detection pipelines that combine diverse sensor inputs.

However, practitioners should note that Omni-Decision depends heavily on strong backend perception and reasoning models and does not inherently defend against adversarial manipulations. The framework is not a direct defense but a systematic method for organizing and validating multi-step evidence gathering that could serve as a backbone for more sophisticated, multi-modal automated decision systems, including in CAPTCHA challenge design or bot behavior analysis.

Cite

@article{arxiv2607_11433,
  title={ Omni-Decision: A Progressive Evidence-State Agent System for Omni-Modal QA },
  author={ Ming Ma and Yi Zhu and Yiran Zhong and Feida Zhu and Weigao Sun and Junhan Shi and Lingrui Mei and Tianming Yang and Steven Hoi },
  journal={arXiv preprint arXiv:2607.11433},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.11433}
}

Read the full paper

• arXiv:2607.11433 (abstract)
• arXiv:2607.11433 (PDF)