MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization

Source: arXiv:2607.15205 · Published 2026-07-16 · By Shaoxiong Zhan, Shi Hu, Boyu Feng, Hai Lin, Andrew Gong, Zhengda Zhou et al.

TL;DR

MM-IssueLoc addresses the challenge of repository-level issue localization in software engineering by explicitly incorporating visual evidence such as screenshots, error dialogs, UI states, and logs, which are common in real-world GitHub issues but overlooked in existing benchmarks. Unlike prior multimodal SE benchmarks that evaluate issue fixing end-to-end—thereby conflating localization with patch generation—MM-IssueLoc isolates localization as the target task. It provides 652 issue-PR instances across 23 programming languages, annotated with 7 image categories and 4 relevance levels. It offers file-level and function-level ground truth labels and enables paired evaluations with text-only inputs, raw images, and a novel Visual Content Evidence (VCE) modality that translates images into structured textual evidence. Evaluation of state-of-the-art LLM-based and retrieval-based localization systems reveals that current models perform significantly worse on this multimodal benchmark versus text-only benchmarks, underscoring multimodal localization as an open challenge. The best system achieves only 38.96% file Acc@5 and 22.45% function Acc@10, and image evidence provides useful but inconsistent signal depending on model and evaluation settings.

Key findings

• MM-IssueLoc contains 652 issue-PR instances with 1,050 associated images across 23 languages, annotated by relevance and 7 distinct image categories.
• The best LLM-based agent, OpenHands with GPT-5.2, achieves 38.96% file-level Acc@5 and 22.45% function-level Acc@10; the best retriever, MM-IssueLoc-VL-Embedding-8B, achieves 33.86% function Acc@10 (Table 2).
• Localization accuracy sharply degrades with difficulty: file Acc@10 drops from 83.10% on easy to 2.84% on hard issue instances (Fig 3A).
• Removing images reduces MM-IssueLoc-VL-Embedding file-level accuracy by 4.44–4.91 points, evidencing useful visual signals (Table 3).
• VCE texturalization of images improves localization over raw images in some settings (e.g., AgentLess Claude-Sonnet-4.6 File@5 improves from 25.15% to 28.53% with VCE), but raw pixels add inconsistent residual gains (Table 4).
• Cross-benchmark comparisons show strong localization on text-only SWE benchmarks (up to 94.53% File@5) does not transfer to multimodal issue localization (max 38.96% File@5).
• LLM agents have higher file-level accuracy while retrieval methods better rank functions due to alignment with evaluation units.
• Harmful images, synthesized to mislead localization, serve as robustness tests, but occur rarely (under 1% real).

Threat model

The adversary is a repository-level software issue producing a bug report composed of textual descriptions and visual artifacts; the defender is a localization system tasked with identifying all pre-existing code locations that must be edited to fix the issue. The adversary does not conduct active attacks or obfuscations beyond natural multimodal evidence presentation. The system may not receive newly introduced functions (added by a patch) as candidates. Harmful images are synthetically introduced to mimic misleading cues, testing robustness but not representing adversarial threat models such as intentional spoofing or poisoning.

Methodology — deep read

The research starts with the observation that real-world bug reports include both textual and visual evidence, but prior SE benchmarks largely ignore or conflate the role of images during repository-level issue localization. To study and isolate the impact of visual evidence, the authors design MM-IssueLoc as a controlled benchmark.

Threat model: The task is repository-level localization, assuming the adversary is a typical software issue resulting in code edits represented by a merged pull request. No explicit adversarial attack is modeled; instead, the challenge is to identify all files/functions modified to fix a given issue report with its multimodal content.

Data:

• 652 GitHub issue-PR pairs from public repos with ≥50 stars, with closed and merged PRs.
• Each instance contains: issue title and body texts, attached images (1050 in total) of 7 categories (UI screenshot, error message, rendering, code screenshot, log output, data viz, behavior demo).
• Relevance annotation scores (-1 harmful, 0 neutral, +1 helpful, +2 vital) assigned manually on 450 instances and AI-assisted on 202 instances.
• File-level gold labels derived from edited files in fixing PR; function-level gold labels from AST and diff intersection in 343 instances.
• Controlled harmful-image synthesis to test robustness.

Architecture/Algorithms:

• Propose MM-IssueLoc-VL-Embedding retriever initialized from Qwen3-VL-Embedding, fine-tuned contrastively with a curriculum from file-level to function-level. Input query includes issue text and attached images; candidates are file/function names.
• Visual Content Evidence (VCE): a non-trainable image-to-text pipeline that extracts structured fields (OCR, error messages, UI elements, saliency etc.) from images and appends them as text to the issue description, enabling disentangling image content from raw pixels.

Training:

• Train contrastive embedding model with InfoNCE loss.
• Use hard negatives mined from same repo non-gold files/functions ranked via pretrained embedding.
• Two-stage curriculum: (1) file-level fine-tuning, (2) function-level continuation.
• Training data contains 7,664 instances for file-level and 3,999 for function-level.

Evaluation:

• Evaluate strict Acc@K metrics requiring all gold files/functions to appear in top-K to count as success.
• File-level metrics reported at K=1,3,5; function-level at K=5,10.
• Compare text-only, text+image, +VCE text, and +VCE+image input modes.
• Baselines: several retrieval methods (BM25, SWERank, Qwen3-VL), LLM-based agents (OpenHands, LocAgent, AgentLess), and own multimodal retriever.
• Perform ablative experiments toggling image inputs and VCE.
• Cross-benchmark comparison against SWE-bench variants.

Concrete example: An issue with title and textual description plus screenshots is input to MM-IssueLoc. The model ranks files/functions. The ground truth is the edited files and functions in the fixing PR. The system’s Acc@5 and Acc@10 are measured strictly requiring full recovery. When images are removed, a drop in accuracy quantifies the contribution of visual evidence. When images are replaced by VCE text, improvement or degradation measures how well structured visual content aids localization.

Reproducibility:

• Dataset, annotations, evaluation protocol, and baseline code/models are publicly released on GitHub.
• The benchmark JSONL files contain rich metadata enabling independent analysis.
• The training data is separated from the eval instances to prevent leakage.

Overall, the methodology carefully isolates visual evidence’s contribution in multimodal issue localization through dataset design, annotation, and controlled evaluation enabling rigorous model comparisons.

Technical innovations

• Introduction of MM-IssueLoc: a controlled benchmark isolating visual evidence in repository-level issue localization across 652 real-world multimodal GitHub issue-PR pairs.
• Development of Visual Content Evidence (VCE), a non-trainable image-to-text extraction pipeline that converts images into structured textual visual evidence to enable disentangled evaluation of image content versus raw pixels.
• Design of a multimodal contrastive retriever (MM-IssueLoc-VL-Embedding) trained with a two-stage file-to-function curriculum and hard-negative mining to effectively integrate textual and visual inputs for localization.
• Controlled evaluation protocol supporting paired input modes (text-only, with-image, +VCE, +VCE+image) and strict all-gold Accuracy@K metrics that precisely quantify visual evidence contribution.

Datasets

• MM-IssueLoc Benchmark — 652 issue-PR instances, 1,050 images — Public GitHub repositories with ≥50 stars, curated and annotated by human and AI-assisted pipelines
• MM-IssueLoc Training Data — 7,664 file-level and 3,999 function-level multimodal instances (13,196 and 13,769 rows respectively) — Constructed disjointly from benchmark set

Baselines vs proposed

• OpenHands GPT-5.2 agent: File@5 Acc = 38.96% vs MM-IssueLoc-VL-Emb-8B retriever: File@5 Acc = 32.82%
• OpenHands GPT-5.2 agent: Function@10 Acc = 22.45% vs MM-IssueLoc-VL-Emb-8B retriever: Function@10 Acc = 33.86%
• BM25 (text-only): File@5 Acc = 21.01% vs MM-IssueLoc-VL-Emb-8B (multimodal): 32.82%
• Removing images reduces MM-IssueLoc-VL-Emb-8B File@5 Acc by 4.44 points (Table 3)
• VCE improves AgentLess Claude-Sonnet-4.6 File@5 from 25.15% to 28.53%, but raw image pixels add only +0.22 points
• OpenHands achieves 94.53% File@5 on text-only SWE-bench Lite vs 38.96% on MM-IssueLoc (multimodal)
• Difficulty level impact: OpenHands GPT-5.2 File@10 drops from 83.10% (easy) to 2.84% (hard) (Fig 3A)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.15205.

Fig 1: Dashboard of MM-IssueLoc. MM-IssueLoc contains 652 file-level instances and 343 function-level

Fig 2 (page 2).

Fig 3 (page 2).

Fig 4 (page 2).

Fig 5 (page 2).

Fig 2: End-to-end construction and evaluation flow for MM-IssueLoc. The benchmark separates human

Fig 3: further explains the aggregate results.

Fig 5: Representative MM-IssueLoc examples grouped by image category.

Limitations

• Function-level gold labels depend on tree-sitter AST parsing, which may be noisy or incomplete for complex language features like macros, templates, or generated code.
• Harmful image instances are primarily synthetic constructions used for robustness tests and do not represent natural frequency in the wild.
• The current retriever uses at most two images per issue, leaving scalability to many-image issues unexplored.
• VCE extraction fields and downstream localization models are fixed; they do not represent an upper bound on potential visual reasoning or integrated multimodal understanding.
• MM-IssueLoc focuses solely on localization, not end-to-end automated patch generation or repair, thus does not measure downstream fixing success.
• Cross-language differences in localization performance are observed but not deeply analyzed due to dataset composition and language parsing constraints.

Open questions / follow-ons

• Can improved multimodal models better leverage raw pixel data—not only structured VCE—to enhance localization accuracy?
• How does multimodal localization performance affect downstream automated repair, debugging, and triage tasks in real SE pipelines?
• What role can advanced language-specific parsers or hybrid static-dynamic code analyses play in improving function-level gold label accuracy and localization evaluation?
• How can many-image issues with complex visual-text interactions be modeled and scaled within the MM-IssueLoc framework?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, MM-IssueLoc highlights the criticality of multimodal evidence integration in complex real-world tasks. Analogously, in bot-defense, suspicious behaviors or attack indicators may manifest through multiple modalities such as textual signals combined with images or UI state snapshots. MM-IssueLoc’s design and evaluation methodology serve as a model to carefully isolate the contribution of visual versus textual cues, enabling granular diagnostics rather than conflated end-goal performance metrics. The benchmark shows that naïve inclusion of images does not guarantee improved localization performance; instead, rigorous analysis with structured visual content extraction and controlled evaluation modes is necessary.

Systems that operate in security contexts involving CAPTCHA or bot detection may draw parallels from MM-IssueLoc’s pipeline: combining raw pixel analysis with interpretable visual content representations to improve detection/localization reliability, and enforcing strict evaluation criteria that require full correct identification of malicious elements rather than partial hits. Additionally, harmful-image robustness testing provides a useful template for adversarial robustness assessments in multimodal threat detection settings.

Cite

@article{arxiv2607_15205,
  title={ MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization },
  author={ Shaoxiong Zhan and Shi Hu and Boyu Feng and Hai Lin and Andrew Gong and Zhengda Zhou and Jiaying Zhou and Yunyun Hou and Hao Su and Hai-Tao Zheng },
  journal={arXiv preprint arXiv:2607.15205},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.15205}
}

Read the full paper

• arXiv:2607.15205 (abstract)
• arXiv:2607.15205 (PDF)