VisualRepair: Dynamic Tool Calling and Region Focusing for Visual Software Issue Repair

Source: arXiv:2607.14075 · Published 2026-07-15 · By Jingyu Xiao, Zhongyi Zhang, Haoran Hou, Yuxuan Wan, Yuan Jiang, Yintong Huo et al.

TL;DR

VisualRepair addresses the challenge of automated program repair (APR) in modern software systems where bug reports contain heterogeneous and complex visual inputs such as UI screenshots, IDE snapshots, GIFs, and text images. Existing APR methods mostly focus on text-only inputs, but visual artifacts are critical for understanding bugs in GUI-rich applications. VisualRepair leverages multimodal large language models (MLLMs) enhanced with two key innovations: Image Type-aware Tool Calling (ITTC) to classify and process diverse image types using tailored toolchains, and Dynamic Test-time Region Focusing (DTRF) which grounds multiple bug-related regions in images and adaptively zooms in/out to refine fault localization. This multimodal approach results in more accurate bug understanding and diverse patch generation.

Experiments on the SWE-bench Multimodal benchmark demonstrate VisualRepair’s effectiveness, with 196 bugs fixed on the test set and 25 on the dev set, surpassing the best baselines by 10 and 11 instances respectively. VisualRepair not only outperforms open-source and closed-source competitors but also achieves higher repair accuracy at a substantially lower computational cost. The work highlights the importance of heterogeneous visual input understanding and dynamic region grounding in enhancing multimodal APR systems.

Key findings

• Visual inputs in real-world issue reports are highly heterogeneous: UI screenshots (43.4% test, 63.8% dev), IDE snapshots (25.9% test, 18.1% dev), GIFs (11.6% test, 2.9% dev), and text images (9.1% test, 7.6% dev).
• VisualRepair resolves 196 bugs on the SWE-bench M test set (37.91% fix rate), surpassing the best baseline by 10 more fixed instances.
• On the dev set, VisualRepair fixes 25 bugs, outperforming the best prior approach by 11 instances.
• The deterministic Image Type-aware Tool Calling module achieved 100% classification accuracy on SWE-bench M with zero misclassifications.
• Dynamic Test-time Region Focusing improved bug localization, allowing VisualRepair to better capture peripheral bug regions missing in MLLM-only grounding (e.g. Fig 3 example).
• VisualRepair attained a low average computational cost of $0.47 per issue, significantly below competing frameworks like OpenHands-Versa ($1.79) and SWE-agent ($1.52–$3.11).
• Repository-level results show VisualRepair achieving best or tied-best fix rates across major projects, e.g. 75.93% on bpmn-js and 97.47% on openlayers.
• Ablation and case studies demonstrated that ITTC’s tailored tool chains and DTRF’s zoom-in/out multi-cropping provide significant effectiveness gains over uniform visual processing.

Threat model

The adversary is the natural occurrence of software bugs causing defects in GUI-rich applications, with issue reports containing heterogeneous visual evidence. The model must comprehend and localize these defects to automatically generate patches. The system assumes no malicious manipulation of inputs or model internals, focusing on automated repair assistant capabilities rather than adversarial robustness.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is implicit—software bugs manifest as defects in codebases with corresponding issue reports that include multimodal text and image attachments. The system assumes an MLLM backbone with no adversarial manipulation of inputs, focusing on automated repair of typical software defects involving GUIs and code. The attacker does not have direct control over model internals or patch selection.

2. Data: VisualRepair is evaluated on SWE-bench Multimodal, a benchmark with 619 JavaScript library bug instances from 17 projects, split into 517 test and 102 dev samples. Inputs are natural language bug descriptions plus associated images of types UI screenshots, IDE snapshots, animated GIFs, and text images. Labels are verified correct patches. 10% of images were manually annotated for type distributions.

3. Architecture / Algorithm: VisualRepair comprises two main components:

• Image Type-aware Tool Calling (ITTC): First classifies input images via file suffix and UI element detection into GIF, UI, IDE code, or plain text image. Each type is routed through a tailored processing chain:
  • GIF Tool: extracts keyframes using mean absolute error (MAE) thresholds to reduce redundancy.
  • OCR Tool: extracts text from IDE and text images using PaddleOCR.
  • Crop Tool: removes uninformative blank regions from UI screenshots using UIED bounding boxes and scanline analysis.
  • Code Library Tool: generates minimal runnable code templates for well-structured repositories to ease bug reproduction.
• Dynamic Test-time Region Focusing (DTRF): Given grounded bug regions in images (via MLLM prompt), DTRF generates M=3 candidate bounding boxes, and augments each with zoom-in (half area) and zoom-out (double area) views, producing 10 total visual inputs per issue. Each is paired with class/function localized buggy code snippets and fed to the MLLM (o3 model) to generate P=1 patch candidate per view, resulting in 10 patch candidates.

Patch formats follow a minimal search/replace edit scheme to reduce hallucination risks. Patches undergo compilation filtering and visual validation by browser rendering and pixel-level screenshot comparison against original bug images. Visual validation discards no-op patches. The MLLM then assesses the patched visual outputs in context to select a final fix.

4. Training Regime: VisualRepair uses the o3 multimodal LLM as backbone with temperature 1 and 2 sampling iterations during bug file and hunk localization. Thresholds for GIF keyframe extraction are fixed (δ=1.2). The system reuses hyperparameters from prior baselines to ensure fair comparison. No model fine-tuning is reported; rather, inference prompting with tailored tool stacks is emphasized.

5. Evaluation Protocol: The primary metric is number of correctly resolved bug instances on SWE-bench M test and dev sets. Baselines include 10 state-of-the-art APR frameworks from the official leaderboard. Repair counts and fix rates are averaged over 3 runs with variance reported. Repository-level fix rates and cost per issue are also measured. Ablation studies isolate contributions of ITTC and DTRF. Qualitative examples highlight region localization benefits.

6. Reproducibility: The code will be released upon acceptance. VisualRepair relies on publicly available SWE-bench Multimodal benchmark. Some components like PaddleOCR and UIED are open source. The Code Library tool auto-generates templates per repository at runtime, preventing data leakage.

Example: For a GIF bug report, ITTC detects .gif suffix, extracts keyframes, and passes them plus bug description to MLLM, which generates reproduction code. DTRF grounds multiple suspect regions in extracted frames, performs zoom augmentations, and generates diverse patches. Compiled and visually validated fixes are ranked by MLLM, outputting the final patch. This modular, type-aware approach improves bug understanding and repair success over baselines.

Technical innovations

• Image Type-aware Tool Calling (ITTC) that deterministically classifies and applies tailored visual processing tools per heterogeneous bug image type, unlike prior free-form LLM tool selection.
• Dynamic Test-time Region Focusing (DTRF) which grounds multiple visual bug regions and uses adaptive zoom-in/out cropping to enable diverse multi-granularity patch generation.
• Use of code template libraries generated on-demand from repository structure to assist bug reproduction in structurally homogeneous front-end libraries, improving patch validity.
• Integration of visual validation—patches are compiled and validated via rendering and pixel-level screenshot comparison to enforce correctness beyond code compilation.

Datasets

• SWE-bench Multimodal — 619 instances (517 test, 102 dev) — publicly available benchmark featuring multimodal JavaScript UI bugs

Baselines vs proposed

• GUIRepair (o3 backbone): resolved 186 instances (35.98%) vs VisualRepair: 196 instances (37.91%)
• SVRepair (o3 backbone): resolved 186 instances (35.98%) vs VisualRepair: 196 instances (37.91%)
• Agentless Lite (o3): 137 instances (26.50%) vs VisualRepair: 196 (37.91%)
• OpenHands-Versa (Claude-Sonnets-4): 178 instances (34.43%) vs VisualRepair: 196 instances (37.91%)
• Zencoder (closed-source): 158 instances vs VisualRepair: 196
• Globant Code Fixer Agent (closed-source): 153 instances vs VisualRepair: 196
• Average cost per issue: VisualRepair $0.47 vs OpenHands-Versa $1.79 and SWE-agent up to $3.11

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.14075.

Fig 1: Four visual tasks with heterogeneous inputs for SWE-bench M [22].

Fig 2: Image types distribution in SWEbench MM benchmark. Others means

Fig 3: Bug-related region grounding example.

Fig 4: illustrates the overall architecture of VisualRepair.

Fig 5: The reproduced code of two issues of PrismJS.

Fig 6 (page 3).

Fig 7 (page 4).

Fig 8 (page 4).

Limitations

• No reported adversarial evaluation; resistance to deliberately crafted visual bug reports is unclear.
• Reliance on fixed parameter thresholds (e.g., GIF MAE cutoff) may limit adaptability to diverse image characteristics.
• The approach requires clear image type discrimination; ambiguous or corrupted images might reduce effectiveness.
• Region grounding still relies on MLLM prompt-based bounding box generation, which may miss subtle bug areas.
• The evaluation is limited to JavaScript front-end codebases; generalization to other languages or GUI frameworks is not shown.
• No online or continuous learning analyzed; system performance under evolving codebases and UI changes is unknown.

Open questions / follow-ons

• How well does VisualRepair handle adversarial or intentionally obfuscated visual bug reports?
• Can the ITTC and DTRF modules be further generalized or learned jointly rather than deterministic pipelines?
• How does VisualRepair perform on other languages, GUI frameworks, or non-frontend domains?
• What is the impact of increasing candidate region count M or patches per region P on repair effectiveness and computational cost?

Why it matters for bot defense

VisualRepair’s methodology of dynamically classifying heterogeneous visual inputs and focusing attention on multiple spatial regions for effective interpretation is highly relevant to CAPTCHAs and bot-defense systems that rely on robust visual understanding. Its deterministic tool routing approach could inspire modular pipelines for detecting and analyzing challenging, multimodal CAPTCHA images such as interactive or animated content. Dynamic region focusing via zoom-in/out strategies reflects how attention mechanisms might be leveraged to pinpoint crucial visual features while ignoring noise, addressing challenges with large uninformative backgrounds common in CAPTCHA snaps. Furthermore, the practice of combining multimodal textual and visual cues to validate patches provides an example of cross-modal verification that could be useful for adaptive CAPTCHA verification involving text extraction and GUI element recognition. Practitioners designing CAPTCHA systems or automated defenses might consider adopting type-aware processing and multi-granularity focus strategies to improve accuracy and reduce false positives or negatives when analyzing complex or heterogeneous visual inputs from bots or humans.

Cite

@article{arxiv2607_14075,
  title={ VisualRepair: Dynamic Tool Calling and Region Focusing for Visual Software Issue Repair },
  author={ Jingyu Xiao and Zhongyi Zhang and Haoran Hou and Yuxuan Wan and Yuan Jiang and Yintong Huo and Michael R. Lyu },
  journal={arXiv preprint arXiv:2607.14075},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.14075}
}

Read the full paper

• arXiv:2607.14075 (abstract)
• arXiv:2607.14075 (PDF)