SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding

Source: arXiv:2608.05137 · Published 2026-08-05 · By Yue Zhang, Yingzhao Jian, Yunqiu Xu, Xiaoxiao Sun, Hehe Fan

TL;DR

SmartMage addresses a core challenge in 3D scene understanding using multimodal inputs: existing multimodal large language models (MLLMs) rely on fixed modality combinations, treating all modalities equally for every query. This rigid fusion introduces semantic noise from irrelevant modalities and underutilizes key ones, causing wasted computation and degraded performance. The paper proposes a dynamic modality orchestration framework that adaptively selects relevant modalities per query and routes tokens to modality-specialized experts for reasoning. The two key modules are SMART (Semantic-guided Modality Adaptive RouTing), which performs semantic-aware global modality selection based on semantic priors, text-modality similarity, and modality quality, and MAGE (Modality-Aware Gating Expert), which guides token routing to experts specialized in particular modalities. Empirically, SmartMage achieves state-of-the-art results on five 3D scene understanding benchmarks (ScanQA, SQA3D, Scan2Cap, ScanRefer, Multi3DRefer), outperforming prior approaches by notable margins (e.g., +5.1 [email protected] on ScanRefer, +6.4 [email protected] on Multi3DRefer). On newly introduced ScanFacet, which categorizes queries by semantic facets, SmartMage’s adaptive modality selection especially benefits material and color questions (+27.1 and +15.9 CIDEr boosts). The method retains strong performance on RGB-only video benchmarks, showing robustness to missing 3D modalities. This paper advances 3D scene understanding by converting fixed modality fusion into adaptive, interpretable selection and specialized multimodal expert reasoning, yielding improved accuracy and efficiency.

Key findings

• SmartMage improves ScanQA EM@1 from 30.8 (Ross3D) to 32.6 (+1.8)
• SQA3D EM@1 increases from 63.0 (Ross3D) to 66.8 (+3.8)
• Scan2Cap [email protected] achieves 93.8 vs 87.1 of PQ3D, a +6.7 gain
• ScanRefer [email protected] rises from 54.4 (Ross3D) to 59.5 (+5.1)
• Multi3DRefer [email protected] improves from 54.3 (Ross3D) to 60.7 (+6.4)
• Ablation removing semantic similarity scorer (SSS) drops ScanQA EM@1 by 2.7 points (29.8 to 27.1)
• Removing modality-aware expert speculation (MES) lowers SQA3D EM@1 from 64.5 to 63.1
• Adaptive modality selection outperforms best fixed combination by +1.9 EM@1 on ScanQA and +2.3 EM@1 on SQA3D

Threat model

The adversary in this context is the fixed-modality fusion approach that acts as a 'naive' baseline by treating all available modalities as equally relevant for every query. This leads to inefficiencies and semantic noise, hurting reasoning performance. The threat model is therefore the rigid modality fusion scheme unable to adapt to query-specific relevance of heterogeneous modalities. The defense is to dynamically select and route modalities based on semantic content and modality quality to reduce noise and wasted computation.

Methodology — deep read

The threat model focuses on relevance and informativeness of multiple heterogeneous modalities (RGB, depth, BEV, point cloud, voxel) for 3D scene understanding queries. The adversary is a rigid fixed-modality approach that treats all modalities equally, causing semantic noise and efficiency loss.

Data is a unified corpus from ScanNet annotations, covering 3D QA, dense captioning, and visual grounding tasks, enabling joint multimodal training. Evaluation uses five standard 3D benchmarks plus three RGB-only video benchmarks and a newly introduced diagnostic benchmark ScanFacet that categorizes questions into eight semantic facets. Data is preprocessed to encode modalities with respective specialized extractors: FoVSR for RGB-D keyframe selection, Mask3D sparse U-Net for voxels, PointNet++ for point clouds, vision backbone encoders for RGB/D/BEV, projecting all into a unified embedding space.

The architecture is a large multimodal large language model (MLLM) augmented with two key modules: (1) SMART (Semantic-guided Modality Adaptive RouTing), which predicts a query-conditioned modality relevance distribution by combining Semantic Prior Estimator (SPE) from text embeddings, Semantic Similarity Scorer (SSS) that measures cosine similarity between text and modality-conditioned visual embeddings via cross-attention, and Modality Quality Evaluator (MQE) that analyzes feature activation strength, sparsity, and stability to measure modality reliability. These signals are fused using learnable weights and an RGB evidence gate to adaptively select modalities per query.

(2) MAGE (Modality-Aware Gating Experts) introduces a sparse Mixture-of-Experts (MoE) architecture into multiple LLM layers by assigning tokens soft routing distributions over modality-specialized experts. The Modality-Aware Expert Speculation (MES) module predicts, per token, a probability distribution over modalities which is transformed into a modality-to-expert affinity prior guiding the router's expert selection probabilities. This bias promotes modality-consistent expert specialization and balanced expert usage.

Training freezes the main visual encoders but fine-tunes adapter layers and the router-expert components over 1 epoch with AdamW on 2 H800 GPUs. Losses include cross-entropy language modeling, semantic alignment losses (semantic correlation and discrimination), and expert assignment losses (modality attribution, expert calibration, and balancing losses). Top-2 experts are used per token at six LLM layers. The training pipeline converts text + selected modalities into token embeddings, routes tokens to experts using MAGE, and decodes answers.

Evaluation metrics are EM@1 for QA, CIDEr@IoU for captioning, and Acc/F1@IoU for visual grounding, plus answer accuracy for RGB-only video. Ablations study the impact of SMART submodules, MAGE submodules, loss components, and adaptive vs fixed modality selection. Diagnostic tools such as ScanFacet reveal semantic-dependent modality preferences. Efficiency is profiled end-to-end, showing moderate overhead relative to baseline multimodal methods.

A concrete example: given the question “What color is the blanket?”, SMART uses SPE to predict RGB preference, SSS confirms strong text-RGB similarity, MQE validates RGB quality, so RGB and relevant modalities are routed. MAGE predicts token modality distributions guiding experts specialized in RGB semantics. The model generates the correct answer with reduced input noise and compute compared to fixed fusion.

Technical innovations

• Semantic-guided Modality Adaptive RouTing (SMART) dynamically combines semantic priors, text-modality similarity, and modality quality signals to select task-relevant modalities adaptively per query.
• Modality-Aware Gating Expert (MAGE) introduces a learnable modality-to-expert affinity prior into sparse Mixture-of-Experts routing, enabling token-level modality-consistent expert specialization.
• Unified omni-modal 3D feature extractor integrating RGB-D, BEV, point cloud, and voxel representations into a shared embedding space supporting dynamic modality coordination.
• Joint end-to-end training scheme fine-tunes lightweight adapters and MoE experts while freezing large pretrained visual and language backbones, stabilizing multimodal reasoning.

Datasets

• ScanNet-based Unified Scene Corpus — size not explicitly reported — private combined dataset constructed from ScanNet annotations
• ScanQA — standard 3D QA benchmark
• SQA3D — 3D embodied spatial question answering
• Scan2Cap — 3D dense captioning dataset
• ScanRefer — 3D visual grounding benchmark
• Multi3DRefer — multi-object 3D visual grounding dataset
• VSI-Bench, VSI-SUPER, MMSI-Bench — RGB-only video understanding benchmarks
• ScanFacet — diagnostic benchmark reorganizing ScanQA and SQA3D QA pairs into eight semantic facets (color, location, material, etc.) with LLM-assisted taxonomy and human verification

Baselines vs proposed

• Ross3D: ScanQA EM@1 = 30.8 vs SmartMage = 32.6
• Ross3D: SQA3D EM@1 = 63.0 vs SmartMage = 66.8
• PQ3D: Scan2Cap [email protected] = 87.1 vs SmartMage = 93.8
• Video-3D LLM: Scan2Cap [email protected] = 83.8 vs SmartMage = 88.7
• Ross3D: ScanRefer [email protected] = 54.4 vs SmartMage = 59.5
• Ross3D: Multi3DRefer [email protected] = 54.3 vs SmartMage = 60.7
• w/o Semantic Similarity Scorer (SSS): ScanQA EM@1 = 27.1 vs full SMART 29.8
• w/o Modality-Aware Expert Speculation (MES): SQA3D EM@1 = 63.1 vs full MAGE 64.5
• Fixed modality best: ScanQA EM@1 = 30.9 vs adaptive SmartMage = 32.6

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2608.05137.

Fig 1: Motivation of SmartMage. (a) Different question types exhibit inherent modality preferences. (b) Previous MLLMs fuse

Fig 2: Overview of SmartMage. The omni-modal feature extractor first encodes text, RGB, depth, BEV, point cloud, and

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• Training performed over only 1 epoch on ScanNet-derived data; potential for further gains with extended training not explored.
• The diagnostic benchmark ScanFacet uses semi-automated LLM taxonomy with limited light human verification, potentially introducing label noise.
• No explicit adversarial evaluation or robustness tests under modality corruption attacks are reported.
• The joint model relies on frozen pretrained backbones; applicability to other backbone architectures or real-time online adaptation not studied.
• Efficiency analysis shows increased preprocessing latency for 3D features (64.8ms) compared to baselines, which may limit deployment in latency-sensitive applications.
• Limited analysis on generalization to out-of-distribution scenes or heavily cluttered environments.

Open questions / follow-ons

• How well does SmartMage generalize to scenes with missing or heavily degraded 3D modalities, beyond the tested RGB-only cases?
• Can the modality-adaptive routing and expert specialization techniques be extended to outdoor or large-scale 3D environments involving LiDAR?
• What are the impacts of longer or more complex query instructions on the robustness of modality selection?
• How does the method perform under cross-dataset domain shifts or with noisy sensor inputs beyond the evaluated distributions?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners focused on multimodal human verification or interactive 3D scene understanding, SmartMage provides a compelling approach to dynamically optimize which sensory modalities to process based on semantic intent. Instead of always fusing all sensor channels—such as cameras, depth sensors, or point clouds—this method selectively attends to the most task-relevant modalities per query. This reduces unnecessary compute and semantic noise, boosting efficiency and accuracy of reasoning over complex multimodal inputs. The modality-aware expert routing further enables specialized reasoning paths tailored to each input modality's strengths. Such adaptive multimodal pipelines could inspire new CAPTCHA designs that flexibly leverage multiple sensing inputs without overwhelming computation or confusing signals. Moreover, the ScanFacet diagnostic benchmark highlights semantic facet-dependent sensitivity to modalities, signaling that CAPTCHA challenges could better target distinct sensory cues for robust human verification. Overall, bot-defense models that understand and incorporate the dynamic interplay of multimodal inputs as SmartMage does can build stronger, more efficient defenses against automated agents attempting to spoof or manipulate multi-sensor verifications.

Cite

@article{arxiv2608_05137,
  title={ SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding },
  author={ Yue Zhang and Yingzhao Jian and Yunqiu Xu and Xiaoxiao Sun and Hehe Fan },
  journal={arXiv preprint arXiv:2608.05137},
  year={ 2026 },
  url={https://arxiv.org/abs/2608.05137}
}

Read the full paper

• arXiv:2608.05137 (abstract)
• arXiv:2608.05137 (PDF)