DualG-MRAG: Decoupling Macro-Reasoning and Micro-Matching for Multimodal Retrieval-Augmented Generation
Source: arXiv:2607.28580 · Published 2026-07-30 · By Jiacheng Tao, Qingyun Sun, Haonan Yuan, Ziwei Zhang, Jianxin Li
TL;DR
This paper addresses the limitations of existing Multimodal Retrieval-Augmented Generation (MM-RAG) systems in handling complex multi-hop reasoning tasks involving heterogeneous data sources such as text, images, and tables. Prior methods mainly rely on instance-level matching or coarse graph-based approaches that either introduce excessive retrieval noise or discard critical fine-grained evidence needed for precise verification. DualG-MRAG proposes a dual-tier graph architecture that decouples macro-level global reasoning from micro-level fine-grained evidence matching, mitigating these tradeoffs. A Macro Graph captures global topological routing, while a Micro Graph encodes local visual and textual features for verification. Retrieval is performed via a novel query-driven graph neural network (GNN) that dynamically propagates relevance scores, replacing static graph topologies and enabling query-specific reasoning paths. Finally, explicit multi-hop reasoning paths are decoded from the GNN’s message passing and injected as structured evidence to guide multimodal Large Language Model (MLLM) answer generation. Extensive experiments on three multimodal multi-hop QA benchmarks demonstrate that DualG-MRAG significantly improves both evidence recall and final QA accuracy over strong vector matching and graph-based retrieval baselines. The method achieves notable gains in exact match, F1, ROUGE-L, and BERTScore metrics, confirming the effectiveness of the decoupled graph construction, query-driven GNN retriever, and explicit path decoding components.
Key findings
- DualG-MRAG improves exact match (EM) score on MMQA from baseline 37.2% (ViDoRAG) to 44.2%, an absolute gain of 7% (Tab. 1).
- Evidence recall at R@2 on MMQA rises from 31.8% (best graph baseline MMGraphRAG) to 49.4% with DualG-MRAG, a 17.6% increase (Fig. 4).
- On WebQA, DualG-MRAG achieves 𝑅@5 recall of 58.2%, surpassing VLM2Vec-V2.0’s 58.0%, indicating better topological retrieval at wider windows.
- ScienceQA results show DualG-MRAG attains 90.99% accuracy overall, outperforming HM-RAG (89.06%) and VLM2Vec-V2.0 (88.12%), especially on image-rich queries (91.52%).
- Decoupling macro reasoning and micro matching reduces visual retrieval noise while preserving local evidence, verified by superior retrieval and QA metrics.
- Query-driven GNN retriever dynamically initializes node embeddings conditioned on query, enhancing selective multi-hop message passing versus static GNNs.
- Dynamic programming path decoding extracts explicit multi-hop reasoning chains directly from GNN’s forward pass, improving structural guidance for MLLM generation.
- DualG-MRAG’s hybrid dual-branch evidence activation combining graph-level subgraph matching and node-level retrieval increases robustness against parser failures.
Methodology — deep read
The authors start by defining the threat model and assumptions in the multimodal retrieval and generation context, focusing on complex multi-hop question answering where queries seek evidence spanning heterogeneous modalities (text, images, tables). The adversary or adversarial scenario is not explicitly detailed since this is a retrieval/QA enhancement method rather than a security-specific paper.
Data provenance: Experiments use three established multimodal multi-hop QA datasets—MultiModalQA (MMQA), WebQA, and ScienceQA. MMQA and WebQA test retrieval and reasoning on mixed modality corpora sampled from their held-out validation sets (1,000 queries each). ScienceQA tests comprehensive reasoning across scientific topics using training sets as knowledge base and 4,241 test samples.
The core technical contribution is the dual-tier multimodal knowledge graph construction separating Macro Graph (global entities and relations) from Micro Graph (fine-grained visual/textual relations linked to original source pointers). Macro Graph entities incorporate concise image captions obtained via frozen Vision-Language Models combined with OpenIE-extracted textual triples and semantic entity resolution (cosine similarity thresholding). Micro Graph maintains localized detailed visual features and relations within images/tables as anchored textual nodes with fine-grained relation types and document pointers.
To activate relevant evidence, a dual-branch matching strategy runs graph-level subgraph matching on Micro Graph driven by a constraint graph parsed from the query with a dynamic visual budget controlling raw image injection. A heuristic branch-and-bound approximate search constrained by top-K semantic candidates limits NP-hard subgraph matching cost. Simultaneously, an auxiliary node-level anchor search retrieves nearest micro-nodes from explicit query entities to increase recall. Both scoring branches combine to score micro-level evidence.
From the activated micro-nodes, a semantic alignment mapping projects them into the Macro Graph nodes, which combined with Named Entity Recognition entities form the initial node mask m_q. This mask initializes a query-aware GNN retriever over the Macro Graph, based on NBFNet architecture. Here, node embeddings are dynamically initialized to the query encoding if activated, else zero, focusing propagation from query-relevant anchors. The GNN runs L layers of relational message passing using the DistMult scoring function with learnable relation transformations for each layer. Message aggregation via sum pooling and linear transformation updates node states.
After propagation, a node relevance MLP scores each Macro Graph node regarding the query. These node-level scores project back to documents, separating textual/tabular and visual scores. Visual scores are modulated by micro-graph matching scores to suppress unverified visual noise. The final document ranking score is the maximum of the normalized textual or modulated visual score.
Crucially, the GNN's message passing trajectory is exploited to extract explicit multi-hop reasoning paths via a dynamic programming algorithm. This interprets the L-hop message passing as a DAG and finds the most probable paths by recursively maximizing cumulative local flow probabilities, calculated per edge by scaled dot-product similarities. The resulting topological evidence paths are serialized and aligned back to original multimodal source snippets, forming structured evidence graphs input to the MLLM for answer generation.
All model components including Micro/Macro Graph construction, the query-driven GNN retriever, and path decoding are trained or constructed offline using pretrained encoders and heuristics. Downstream LLMs for generation are fixed Qwen3-VL-4B or 8B. Experiments run on NVIDIA A100 hardware. No mention is made of release of code or frozen weights.
End-to-end example: Given a complex multimodal query, first a structured pattern graph is parsed. Subgraph matching retrieves candidate micro-facts grounded in local images or tables. These micro-facts map to Macro Graph nodes and activate masked nodes m_q. The GNN retriever propagates relevance dynamically conditioned on the query, yielding ranked document scores. The dynamic programming extracts reasoning chains from the GNN message flows, explicitly connecting evidence steps across modalities and documents. Finally, these serialized reasoning paths, along with the raw multimodal snippets, form the input prompt to the MLLM that autoregressively generates the final answer.
Technical innovations
- Dual-tier decoupled graph construction separating Macro Graph for global structural reasoning from Micro Graph for fine-grained local evidence verification to mitigate retrieval noise in multimodal RAG.
- Query-driven GNN retriever dynamically initializes node embeddings conditioned on the query mask to enable selective multi-hop message passing rather than static, query-agnostic propagation.
- Dynamic programming-based path decoding algorithm extracting explicit multi-hop reasoning chains directly from GNN forward pass message flows, enabling structural evidence serialization.
- Hybrid dual-branch evidence activation combining rigorous graph-level subgraph matching over Micro Graph with auxiliary node-level retrieval to improve robustness and retrieval effectiveness.
Datasets
- MultiModalQA (MMQA) — 1,000 validation queries sampled for evaluation — public benchmark
- WebQA — 1,000 validation queries sampled for evaluation — public benchmark
- ScienceQA — 4,241 test samples — public benchmark for scientific domain QA
Baselines vs proposed
- ViDoRAG: EM = 37.2% vs DualG-MRAG: EM = 44.2% on MMQA
- MMGraphRAG: R@2 = 31.8% vs DualG-MRAG: R@2 = 49.4% on MMQA
- VLM2Vec-V2.0: R@5 = 58.0% vs DualG-MRAG: R@5 = 58.2% on WebQA
- HM-RAG: accuracy = 89.06% vs DualG-MRAG: accuracy = 90.99% on ScienceQA
- Base LLM Qwen3-VL-8B: EM = 22.3% vs DualG-MRAG: EM = 46.0% on MMQA (8B backbone)
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.28580.
Fig 1: Comparison of existing graph-enhanced MRAG and
Fig 2: The overall architecture of DualG-MRAG. The framework operates in three query-driven phases. (A) Dual-Tier
Fig 3 (page 12).
Fig 4 (page 12).
Limitations
- The approach requires substantial offline graph construction with OpenIE and fine-grained visual parsing, potentially expensive for very large corpora.
- Subgraph matching on Micro Graph is approximated heuristically due to NP-hardness, possibly missing some evidence.
- No adversarial robustness evaluation against retrieval attacks or query manipulation reported.
- The explicit path decoding assumes L-hop acyclic computation DAG, which may limit handling cycles or highly interconnected graphs.
- Evaluation focuses on curated benchmarks; real-world open-domain retrieval scaling and latency not fully explored.
- Code and pretrained models not currently released, limiting immediate reproducibility and adoption.
Open questions / follow-ons
- Can the dual-tier framework be extended to online dynamic graph updates or streaming multimodal data?
- How robust is the system against adversarial evidence injection or query paraphrasing attacks in multimodal retrieval?
- Could incorporating learned edge weights or attention mechanisms further refine message passing beyond DistMult in the GNN retriever?
- How well does DualG-MRAG scale and perform in open-domain settings with millions of heterogeneous documents?
Why it matters for bot defense
From a bot-defense and CAPTCHA perspective, the key takeaway is the paper’s demonstration that finely decoupling global reasoning and local evidence verification using dual-tier graph structures dramatically improves cross-modal retrieval and reasoning. For CAPTCHA systems, which often rely on multimodal evidence verification (e.g., images combined with textual clues), such architectures could offer more robust defense by explicitly modeling complex relationships and multi-hop dependencies, making it harder for bots to spoof answers through simple surface-level matching.
Moreover, the query-driven GNN retriever that dynamically adapts its relevance propagation paths to user intent might inspire more sophisticated challenge designs that require reasoning over connected evidence, rather than isolated verification. The explicit path decoding and serialization can also be viewed as a scheme for explainability in challenge generation, allowing systems to track step-by-step evidence accumulation. While the paper focuses on retrieval-augmented generation for QA, the underlying principles could facilitate building stronger CAPTCHA schemes that inhibit automated abuse by demanding reasoning across multimodal, graph-structured evidence rather than shallow pattern recognition.
Cite
@article{arxiv2607_28580,
title={ DualG-MRAG: Decoupling Macro-Reasoning and Micro-Matching for Multimodal Retrieval-Augmented Generation },
author={ Jiacheng Tao and Qingyun Sun and Haonan Yuan and Ziwei Zhang and Jianxin Li },
journal={arXiv preprint arXiv:2607.28580},
year={ 2026 },
url={https://arxiv.org/abs/2607.28580}
}