Cognitive-structured Multimodal Agent for Multimodal Understanding, Generation, and Editing

Source: arXiv:2607.08497 · Published 2026-07-09 · By Feng Wang, Canmiao Fu, Zhipeng Huang, Chen Li, Jing Lyu, Ge Li

TL;DR

To enable training with explicit turn-level retrieval supervision, the authors develop a Unified Scenario Engine that programmatically generates multi-turn multimodal dialogues with fine-grained retrieval annotations. They also build a dedicated long-horizon benchmark (M2CA-Bench) with 20-turn sessions annotated by retrieval difficulty. Through supervised fine-tuning and reinforcement learning, the 8B-parameter agent achieves 91.4% retrieval accuracy on the full 20-turn benchmark, outperforming larger 32B unified models by over 8% and reducing inference latency per turn by nearly half. The paper also presents CMA-Harness, a tool-augmented framework implementing the same cognitive architecture with persistent memory, web access, and multimodal editing tools. Results suggest that structured memory and modular decision-making provide a more scalable and efficient alternative to monolithic scaling for sustained multimodal interaction.

Key findings

• The 8B-parameter Cognitive-structured Multimodal Agent achieves 91.4% retrieval accuracy on the full 20-turn M2CA-Bench, exceeding the 32B unified baseline by +8.2% (English).
• Retrieval accuracy increases with task-specific training and RL: SFT and RL improve retrieval from 81.8% to 91.4% on the full benchmark (Table 5).
• Per-turn inference time is reduced from 23.1 seconds (unified 32B baseline) to 12.7 seconds, nearly halving latency.
• The Multi-agent architecture outperforms unified models substantially on long-horizon retrieval: unified baseline (BAGEL) falls below 3% retrieval on the hardest turns vs 82%+ for the proposed agent (Table 3).
• Improved retrieval accuracy correlates with generation quality gains: our agent scores 8.49 overall (English) on Gemini-3-Pro quality metric, besting 32B baseline by +0.25 (Table 4).
• The agent maintains stable performance across 20-turn sessions involving mixed tasks (generation, editing, understanding), unlike unified models which degrade with dialogue length and retrieval difficulty.
• The Unified Scenario Engine successfully generates structured multimodal dialogue data with turn-level retrieval supervision to enable RL optimization, filling a dataset gap.
• Ablation studies indicate that jointly optimizing memory construction (PAE) and retrieval (CoRE) via reinforcement learning maximizes retrieval performance and downstream task fidelity.

Methodology — deep read

1. Threat Model & Assumptions: The adversary model is not explicitly defined in this research, as the focus is on system architecture and multimodal dialogue performance improvements rather than security. The design assumes real users engage in multi-turn visual-text dialogue requiring long-term visual context recall. It is assumed that cross-turn visual grounding is necessary for consistent generation and understanding, and that visual token explosion impedes unified approaches. No adversarial manipulation or attack scenarios are considered.

2. Data: The core training data is programmatically generated by the Unified Scenario Engine, which simulates multi-turn structured multimodal dialogues with corresponding image sets and precise turn-level retrieval annotations. Training comprises 270 sessions × 20 turns (~5,400 turns) with 55 curated topics across 8 domains to ensure compositionality and topic shifts. Validation uses 30 sessions × 20 turns (~600). The held-out M2CA-Bench evaluation set has 100 sessions × 20 turns (2,000 annotated turns) with no topic overlap. The data construction pipeline includes human simulation of user queries, zero-shot multi-agent generation of model responses, and ground-truth justification verification to guarantee retrieval correctness.

3. Architecture:

• Perceptual Abstraction Engine (PAE): Inputs incoming images and produces structured JSON memory entries containing semantic tags, textual descriptions, and compressed thumbnails. These abstractions serve as compact visual summaries stored externally.
• Episodic Visual Memory (EVM): Stores PAE's structured image representations externally, allowing selective retrieval without flooding the dialogue context window.
• Cognitive Retrieval Engine (CoRE): Given the current user query and dialogue state, CoRE performs cross-modal retrieval from EVM to identify the minimal subset of relevant historical images for the current turn.
• Multimodal Executive Controller (MEC): Integrates dialogue history and retrieved memory, autonomously classifies the user request into one of five task modes (generation, editing, composition, understanding, pure chat), and dispatches control accordingly to either the image generation pipeline (Qwen-Image-Edit) or text response.

The modular design decouples perception, memory, retrieval, and control, enabling independent training and efficient inference.

4. Training Regime:

• Supervised Fine-Tuning (SFT) trains CoRE using retrieval annotations from the Unified Scenario Engine dataset to predict appropriate image indices.
• Reinforcement Learning via DAPO optimizes both CoRE and PAE with a retrieval-focused reward: the Jaccard similarity between predicted and ground-truth retrieval sets, weighted by difficulty. RL samples emphasize harder turns, incorrect baseline predictions, and negative samples.
• PAE is optimized with frozen CoRE to improve memory entry formats to yield better downstream retrieval success rather than classical captioning metrics.

Training uses Qwen3-VL-8B as the backbone for vision-language modules. Specific hyperparameters, epochs, batch size, or seeds are not detailed in the source text.

5. Evaluation Protocol:

• Metrics include Context Usage Accuracy (exact-match retrieval accuracy of relevant image indices per turn) and Generation Quality (per-turn Gemini-3-Pro scores from 0-10 evaluating faithfulness and visual correctness).
• Evaluation subsets 'Full' (all 20 turns), 'Medium' (turns 11-20), and 'Hard' (very hard subset within 11-20) measure performance degradation over longer contexts, topic shifts, and ambiguous references.
• Baselines include a unified multimodal model (BAGEL), all-context 8B and 32B prompt baselines feeding full history, and a multi-agent baseline without training optimizations.
• The training and evaluation datasets have disjoint topics and seeds.

6. Reproducibility:

• Code, dataset, and the CMA-Harness project page are publicly released (GitHub link provided).
• Frozen weights or exact training details are not specified but modular components are built on the publicly known Qwen VL models and Qwen-Image-Edit.

Technical innovations

• Introduction of a Cognitive-structured Multimodal Agent architecture that externalizes visual input into an Episodic Visual Memory with modular components for perceptual abstraction, retrieval, and executive control, avoiding exponential token growth.
• Development of a Unified Scenario Engine to programmatically synthesize multi-turn multimodal dialogues with fine-grained turn-level retrieval supervision, enabling effective RL training for retrieval policies.
• Application of difficulty-weighted reinforcement learning (DAPO) directly optimizing perceptual abstraction and retrieval policies for long-horizon cross-turn visual grounding rather than traditional caption similarity metrics.
• Design of a Multi-turn Context Agent Benchmark (M2CA-Bench) stratified by retrieval difficulty, specifically targeting episodic visual memory performance in sustained multimodal dialogue.
• Implementation of a tool-augmented deployment framework (CMA-Harness) that expands the cognitive-structured architecture to persistent multi-session memory and integrates web access and advanced image generation/editing/composition tools.

Datasets

• Unified Scenario Engine dataset — ~5,400 multi-turn dialogue turns — programmatically generated synthetic multimodal conversations with retrieval annotations
• M2CA-Bench — 2,000 turns (100 sessions × 20 turns) — held-out benchmark generated with disjoint seeds and topics for long-horizon visual-dialogue evaluation

Baselines vs proposed

• Unified Model (BAGEL): Retrieval accuracy (Full subset, English) = 23.3% vs Ours (8B) = 91.4%
• Agent Baseline-8B (All-Context): Retrieval accuracy (Full, English) = 78.9% vs Ours = 91.4%
• Agent Baseline-32B (All-Context): Retrieval accuracy (Full, English) = 81.9% vs Ours = 91.4%
• Multi-Agent Baseline-8B: Retrieval accuracy (Full, English) = 81.8% vs Ours = 91.4%
• Multi-Agent Baseline-32B: Retrieval accuracy (Full, English) = 83.2% vs Ours = 91.4%
• Generation Quality (Gemini score, Overall English): BAGEL = 4.97 vs Ours = 8.49
• Generation Quality (Gemini score, Edit task, Chinese): Multi-Agent Baseline-32B = 7.03 vs Ours = 7.44
• Training Ablation (Retrieval accuracy Full subset English): Multi-Agent Baseline-8B = 81.8%, + SFT CoRE = 86.6%, + RL CoRE = 90.1%, + RL PAE = 91.4%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.08497.

Fig 1: A multi-turn multimodal dialogue produced by our Cognitive-structured Multimodal Agent, spanning 20 turns across four topics

Fig 2: End-to-end pipeline of the proposed Cognitive-structured Multimodal Agent. Each incoming image passes through the Perceptual

Fig 3 (page 2).

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 2).

Fig 7 (page 2).

Fig 8 (page 2).

Limitations

• The dataset and benchmark are synthetically generated via programmatic simulation rather than drawn from natural human dialogues, which may limit real-world generalization.
• Evaluation is limited to English and Chinese, leaving open performance in other languages or multimodal settings.
• The model and approach have not been tested under adversarial conditions or attempts at malicious retrieval manipulation.
• Deployment details and end-to-end user studies of the CMA-Harness tool framework remain preliminary and lack quantitative evaluation.
• Training and inference speed improvements are reported but absolute throughput and resource usage details (e.g. GPU specs, batch size) are not fully disclosed.
• While better than unified baselines, retrieval accuracy still drops with increasing retrieval difficulty, showing room to improve robustness on very ambiguous cross-turn references.

Open questions / follow-ons

• How well does the cognitive-structured approach generalize to natural human multimodal conversations beyond simulated scenarios?
• Can the external memory and retrieval module be further scaled or replaced with neural retrieval paradigms for even larger episodic contexts?
• How robust is the system to noisy or ambiguous visual inputs and natural language queries in real-world deployments?
• What strategies can effectively integrate persistent multi-session memory to support longer-term personalization and multimodal state tracking?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this paper highlights a promising architectural direction for multimodal agents needing sustained visual dialogue capabilities. Traditional unified models that repeatedly process entire visual histories are computationally prohibitive and degrade in long interactions, limiting their practical deployment. The cognitive-structured agent's explicit episodic visual memory with selective retrieval offers a scalable, latency-efficient paradigm to maintain cross-turn visual context integrity and task consistency.

In CAPTCHA or bot-detection systems leveraging multimodal interactions, the modular design allowing efficient retrieval-driven grounding can reduce costly token overhead and maintain accurate reference to past visual challenges or user inputs. Reinforcement learning optimizing perceptual abstraction for retrieval success rather than caption quality could inform more effective memory representations relevant to verifying user authenticity in ongoing sessions. However, practitioners should consider that the training data is synthetic and that robustness under adversarial attempts to spoof episodic memory retrieval remains unexplored—critical for security-centered applications.

Cite

@article{arxiv2607_08497,
  title={ Cognitive-structured Multimodal Agent for Multimodal Understanding, Generation, and Editing },
  author={ Feng Wang and Canmiao Fu and Zhipeng Huang and Chen Li and Jing Lyu and Ge Li },
  journal={arXiv preprint arXiv:2607.08497},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.08497}
}

Read the full paper

• arXiv:2607.08497 (abstract)
• arXiv:2607.08497 (PDF)