Beacon: Knowing When and How to Perform Agentic Visual Reasoning

Source: arXiv:2607.28595 · Published 2026-07-30 · By Qixun Wang, Yang Shi, Letian Cheng, Zhuoran Zhang, Yan He, Yuqi Tang et al.

TL;DR

This paper addresses a core challenge in agentic visual reasoning for multimodal large language models (MLLMs): improving their ability to selectively and effectively use external visual tools to solve complex tasks. Existing agentic visual reasoning models often invoke tools inefficiently—either using them unnecessarily on easy problems solvable by text-only reasoning or failing to invoke them when needed on difficult problems—and gain little net benefit from tool use. The authors introduce two key concepts to analyze this phenomenon: Mode Adaptiveness (whether the model calls tools only when necessary) and Tool Effect (whether tool use genuinely extends capabilities without causing harm on easy tasks). They provide a thorough empirical analysis showing that prior models exhibit limited adaptiveness and tool-effectiveness.

Motivated by these insights, the authors propose Beacon, a novel agentic visual reasoning model trained via a supervised fine-tuning and reinforcement learning pipeline designed to improve adaptive tool invocation and expand tool-use capabilities on challenging examples. The core technical innovations include a Necessity-Aware Adaptive Reward that encourages selective tool use, and a Hint-Guided Capability Expansion mechanism that leverages expert hints to discover effective tool-use trajectories on hard cases. Experiments across 13 diverse benchmarks demonstrate that Beacon achieves stronger overall performance, substantially better Mode Adaptiveness, and positive net tool-induced gains, surpassing prior open-source and some closed-source agentic visual reasoning models.

Key findings

• Beacon improves average performance over the base Qwen3-VL-8B-Instruct model by +6.07 percentage points on 13 benchmarks.
• Beacon achieves a higher tool-call ratio on hard samples and lower unnecessary tool calls on easy samples compared to prior models (Fig 2a).
• Beacon attains a Mode Adaptiveness mean score well above prior models, which often score near or below 50% (random baseline).
• Beacon’s net tool-effect (tool gain minus tool harm) is the largest (+3.1%), whereas prior models’ tool gains are largely offset by tool harms (Fig 2c).
• The Necessity-Aware Adaptive Reward assigns higher credit to correct text-only reasoning on easy samples and full reward to code solutions on hard samples, enabling balanced adaptive tool use.
• Hint-Guided Capability Expansion enables discovering successful tool-use trajectories on hard problems via expert hints from Gemini 3.1 Pro, improving RL training efficiency.
• On high-resolution visual search, spatial reasoning, quantitative, and compositional reasoning tasks, Beacon ranks first or second among open-source models on 11/13 benchmarks.
• Beacon’s tool-free reasoning alone also improves over prior models, showing the model benefits from RL beyond just tool use.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is not explicitly modeled since this is a model capability improvement paper, not a security work. The model operates in an agentic reasoning setting where it autonomously decides when to invoke external visual tools (code execution) to assist in complex multimodal reasoning. The main assumption is that task difficulty varies: some problems can be reliably solved with text-only reasoning, others require external tool use. Optimal performance requires adaptive invocation.

2. Data: Training data is curated from 16 open-source visual reasoning benchmarks spanning diverse tasks including real-world perception, spatial reasoning, diagram interpretation, STEM tasks, and agentic visual reasoning challenges. Overlaps with evaluation sets are removed. Training involves 3 stages: (a) selecting hard samples the base model (Qwen3-VL-8B-Instruct) struggles with, (b) generating and refining code-assisted reasoning trajectories on these hard samples using Gemini 3.1 Pro as an expert, and (c) selecting remaining unsolved hard samples for reinforcement learning.

3. Architecture / Algorithm: Beacon is built on Qwen3-VL-8B-Instruct and produces answers either by text-only reasoning or by generating Python code snippets enclosed in <tool_call> tags, which are executed externally for image/numerical operations. Outputs are returned in <observation> tags for continued reasoning.

Training combines supervised fine-tuning (SFT) on code-assisted trajectories and reinforcement learning (RL) with two novel components: (i) Necessity-Aware Adaptive Reward (NAAR), which weights rewards based on whether text-only reasoning suffices for a sample to encourage adaptive tool use, and (ii) Hint-Guided Capability Expansion (HCE), which injects answer-free hints derived from expert model trajectories to help exploration and capability expansion on hard samples. RL optimization employs the GRPO algorithm with importance sampling adjusted for hinted and normal rollouts.

4. Training Regime: SFT: 4 epochs at 1e-5 learning rate, masking code tokens in loss to avoid overfitting on code. RL: 1 epoch at 1e-6 learning rate, rollout group size 128.

5. Evaluation Protocol: Evaluated on 13 diverse benchmarks covering visual search, spatial reasoning, STEM, compositional and agentic visual tasks. Each sample is run 5 times with and without tools to assess tool-invocation adaptiveness and tool effectiveness. Key metrics include Mode Adaptiveness components (MAtext, MAtool) measuring avoidance of unnecessary and missing tool calls, Tool-Gain and Tool-Harm quantifying net benefit of tool use, and overall accuracy. Baselines include prior state-of-the-art open-source agentic visual reasoning models Thyme, DeepEyesV2, CodeV, Metis, and the base Qwen3-VL-8B-Instruct.

6. Reproducibility: Code and model weights are hinted as publicly released (not confirmed here). Dataset sources are open benchmarks; synthesis and refinement pipelines are described in Appendix B. Hyperparameters and prompt templates are detailed in Appendix C.

Example end-to-end: Given a hard problem requiring counting blue marbles in an image, Beacon first tries text-only reasoning (prompt ptext). If uncertain or likely to fail, it generates Python code to perform image cropping and pixel-level computation (enclosed in <tool_call> tags). The code output is executed, returns an observation (like number of detected blue marbles), and the model incorporates this to finalize the answer. During RL, the model receives higher rewards for tool use if needed, but is penalized for unnecessary tool calls on easy problems solvable by text-only reasoning, nudging adaptive invocation behavior.

Technical innovations

• Introduction of Necessity-Aware Adaptive Reward (NAAR) that dynamically assigns rewards based on the necessity of tool use, encouraging selective tool invocation rather than indiscriminate or always/no tool use.
• Hint-Guided Capability Expansion (HCE) mechanism that injects intermediate answer-free hints generated by an expert model into RL rollouts to help the policy discover successful tool-use trajectories on hard examples.
• Unified evaluation framework for agentic visual reasoning that quantifies Mode Adaptiveness (MA) and Tool Effect (TE) separately, using multiple inference runs to improve robustness against stochasticity.
• Adaptive RL training pipeline combining supervised fine-tuning and reinforcement learning with specialized rewards and expert hints to expand multimodal reasoning capabilities beyond text-only reasoning.

Datasets

• V* — unknown size — open source visual search benchmark
• HRBench 4K and 8K — unknown size — open source high-resolution visual benchmarks
• VisualProbe — unknown size — spatial and perceptual reasoning
• RealWorldQA — unknown size — real-world spatial reasoning
• BLINK — unknown size — open benchmark for vision-language tasks
• BabyVision — unknown size — perceptual reasoning
• ChartQAPro — unknown size — quantitative and diagrammatic reasoning
• MathVista — unknown size — quantitative reasoning
• MathVision — unknown size — quantitative and diagrammatic reasoning
• VisualPuzzles — unknown size — compositional visual reasoning
• GameQA — unknown size — agentic visual reasoning
• TIRBench — unknown size — agentic visual reasoning
• Training data pool constructed from 16 datasets spanning diverse visual reasoning domains (specific datasets listed in Appendix B).

Baselines vs proposed

• Qwen3-VL-8B-Instruct: average accuracy = 60.57% on visual/spatial benchmarks vs Beacon: 66.05%
• Metis-8B: 64.57% vs Beacon: 66.05%
• Qwen3-VL-8B-Instruct: average accuracy = 43.97% on quantitative/compositional benchmarks vs Beacon: 50.73%
• Metis-8B: 47.46% vs Beacon: 50.73%
• Beacon achieves net tool-effect gain of +3.1% vs prior models (Thyme, CodeV, DeepEyes-V2, Metis) whose tool gains are largely offset by tool harms (Fig 2c).
• Beacon improves tool-free accuracy and tool-available accuracy more than baselines (Section 5.3 analysis).

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.28595.

Fig 1: Agentic visual reasoning models should use tools adaptively and effectively.

Fig 2: (a) The tool-call ratio at different levels of text-only accuracy. (b) Average performance

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• The evaluation focuses on open-source and a few closed-source models but does not fully characterize manual effort or compute tradeoffs in training Beacon.
• The RL training was only conducted for 1 epoch due to resource constraints, potentially limiting convergence and the maximum capability gains.
• The hint-guided rollout relies on a strong expert model (Gemini 3.1 Pro), which may limit generalizability if such experts are unavailable.
• The impact of stochasticity in problem difficulty classification on Mode Adaptiveness metrics, although mitigated by multiple rollouts, may still cause some noise.
• The approach focuses on Python-based tool invocation; extending to other modalities or tool types is not evaluated.
• No adversarial evaluation or stress testing on out-of-distribution visual tasks was reported, limiting understanding of robustness.

Open questions / follow-ons

• Can the Necessity-Aware Adaptive Reward and Hint-Guided Capability Expansion methods be generalized to other agentic reasoning domains beyond visual reasoning?
• How does Beacon perform under distribution shifts or in zero-shot scenarios with unseen tool types or tasks?
• Can the requirement for an expert hint generator be reduced or replaced with self-play or unsupervised methods without performance loss?
• What are the compute and latency trade-offs of Beacon’s adaptive tool use in real-time or production environments?

Why it matters for bot defense

Beacon's principled approach to adaptive tool use in agentic visual reasoning offers valuable insights for bot-defense and CAPTCHA practitioners aiming to improve AI-powered challenge solvers. The concepts of Mode Adaptiveness and Tool Effect emphasize the importance of deciding when to invoke computationally expensive or specialized tools versus relying on simpler reasoning modes. This adaptiveness reduces unnecessary overhead and mitigates error introduction, which is critical in security-sensitive applications such as CAPTCHAs where efficient, robust visual understanding is needed.

Moreover, Beacon’s reinforcement learning framework with necessity-aware rewards and expert guidance mechanisms provides a blueprint for training models that judiciously rely on external modules (e.g., OCR, object detectors) only when needed. This could inform strategies to design CAPTCHA solvers or bot detectors that balance performance and computational cost. The evaluation methodology grounding adaptive tool invocation decisions in robust multi-run metrics also offers a useful analytic framework to rigorously assess tool-augmented reasoning systems in security contexts.

Cite

@article{arxiv2607_28595,
  title={ Beacon: Knowing When and How to Perform Agentic Visual Reasoning },
  author={ Qixun Wang and Yang Shi and Letian Cheng and Zhuoran Zhang and Yan He and Yuqi Tang and Qi Zhang and Xinlei Yu and Ruizhe Chen and Tianrun Xu and Yuanxing Zhang and Pengfei Wan and Haotian Wang and Xianghua Ying },
  journal={arXiv preprint arXiv:2607.28595},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.28595}
}

Read the full paper

• arXiv:2607.28595 (abstract)
• arXiv:2607.28595 (PDF)