ViewMind3D: Modular View-Aware Inference for Training-Free 3D-QA

Source: arXiv:2607.28442 · Published 2026-07-30 · By Ping-Kun Chiang, Kun-Ru Wu, Po-han Li, Sandeep Chinchali, Ufuk Topcu, Yu-Chee Tseng

TL;DR

ViewMind3D addresses the challenge of 3D question answering (3D-QA) in indoor scenes without relying on costly 3D-specific training or fine-tuning. Existing 3D-QA approaches typically require annotated 3D data and task-specific models, limiting scalability and real-world application. ViewMind3D proposes a fully training-free, modular pipeline that leverages general-purpose pretrained large language models (LLMs) and vision-language models (VLMs) to perform spatial reasoning over multi-view RGB images and 3D point clouds. The key novelty lies in decomposing the 3D-QA task into interpretable stages: question-driven multi-view selection, guided visual grounding with language-conditioned object cues, bird’s-eye-view (BEV) spatial encoding of camera poses, and structured multi-stage answer synthesis. This modular orchestration enables robust, interpretable 3D reasoning without any parameter updates or fine-tuning on 3D data.

Evaluated on ScanQA and SQA3D benchmarks, ViewMind3D matches or surpasses prior training-free and fine-tuned baselines. On ScanQA validation, it achieves 73.4 CIDEr and 50.8% accuracy on SQA3D, exceeding 3D-LLM and GPT-based baselines on spatially grounded question types. Ablations demonstrate that guided detection and BEV viewpoint encoding provide complementary gains in spatial understanding. The approach’s modularity also offers efficiency benefits by reducing irrelevant multi-view inputs. These results highlight that effective 3D spatial reasoning can be realized solely through structured use of powerful pretrained LLMs and VLMs without 3D-specific training.

Key findings

• ViewMind3D achieves 73.41 CIDEr on ScanQA, surpassing 3D-LLM’s 69.40 and fine-tuned models like ScanQA (64.9 CIDEr).
• On the SQA3D test set, ViewMind3D attains 50.75% accuracy overall, outperforming 3D-LLM (49.79%) and GPT4Scene zero-shot variants (~44-46%).
• In ‘What’-type questions requiring spatial reasoning, ViewMind3D exceeds 3D-LLM by over 7 percentage points (44.38% vs 37.05%).
• Guided Detection module adds +1.15 CIDEr on ScanQA and +1.57% accuracy on SQA3D average, but reduces accuracy on ‘How’ questions (47.31% → 39.78%) due to noisy detections.
• Viewpoint Indicator using BEV improves CIDEr on ScanQA from 57.41 to 63.48 and boosts ‘Which’ questions accuracy on SQA3D from 30.20% to 37.89%.
• Role decomposition into specialized agents improves ScanQA CIDEr from 54.77 to 58.05, proving modular design benefits reasoning quality.
• Term-Based Instructions aligning model outputs to dataset vocabularies raise ScanQA CIDEr by 5.43 points (58.05 → 63.48).
• Clock-based vs compass-based BEV direction encodings yield comparable performance, with clock-based better for fine spatial questions ('What'), compass for coarser reasoning ('Is').

Threat model

n/a — this work focuses on modular zero-shot 3D spatial reasoning for embodied AI rather than security or adversarial robustness. It assumes the adversary cannot manipulate 3D inputs or language prompts maliciously.

Methodology — deep read

1. Threat Model & Assumptions: The adversary scenario is not explicitly security-critical here but the system assumes multi-view RGB images, 3D point clouds, a natural language query, and optionally spatial context (position and orientation of the questioner). It presumes no retraining or fine-tuning on 3D data is possible, working fully zero-shot. The system expects general pretrained large language and vision-language models to possess the necessary capabilities for zero-shot grounding and spatial reasoning.

2. Data: The method uses two standard 3D-QA benchmarks: ScanQA and SQA3D, both based on ScanNet. ScanQA validation and SQA3D test sets are used for evaluation. ScanQA lacks explicit situation descriptors, while SQA3D provides textual situation and exact poses. Multi-view RGB images are extracted from ScanNet’s 25k frames subset. Corresponding 3D point clouds are used to construct the BEV spatial context.

3. Architecture & Algorithm: ViewMind3D consists of four modular components:

• Relevance View Selection filters question-relevant views from all multi-view images. It uses an agent selector to classify each view as relevant or not, retaining a recall-oriented subset.
• Guided Detection extracts object categories referenced by the question using a Natural Language Guider, enriches object attributes per view, and runs an open-vocabulary detector (Florence-2) to annotate bounding boxes over relevant objects in the selected views, providing explicit visual grounding.
• Viewpoint Indicator encodes camera poses as bird’s-eye-view (BEV) maps: each camera’s 3D position and orientation (SE(3)) are overlaid as directional markers on a global BEV map. This provides a shared spatial reference across views.
• 3D-QA Module orchestrates multi-stage reasoning via specialized agents: Question Interpreter parses structured intent; View Captioner generates viewpoint-aware captions for each annotated view using the BEV cues; Summarizer aggregates multi-view captions into spatially coherent region summaries; Answerer generates final text answers combining spatial, visual, and linguistic information; Answer Normalizer aligns outputs to canonical vocabularies. All agents rely on pretrained LLMs and VLMs (e.g., GPT-4.1, OpenAI o3, Florence-2) and execute inference-only prompts without parameter updates.

4. Training Regime: No training or fine-tuning occurs; the entire pipeline is zero-shot and training-free. Modular components run sequentially with designed prompts. Hyperparameters like IoU threshold for bounding box NMS are fixed (e.g., 0.6). The number of selected views is adaptive per-question.

5. Evaluation Protocol: Standard language generation metrics (BLEU-1, METEOR, ROUGE-L, CIDEr, BERTScore-F1) measure answer quality on ScanQA. SQA3D uses accuracy per question type. Comparisons include fine-tuned 3D-QA models, zero-shot GPT-based methods, and prior training-free baselines. Ablations test individual component removals (Guided Detection, Viewpoint Indicator), encoding schemes (compass vs clock), and role decomposition vs monolithic agent.

6. Reproducibility: Some agents rely on proprietary LLM/VLM backbones without public pretrained weights or code, but the paper details prompt designs and modular architecture to guide replication. Datasets used are publicly known benchmark splits. The design encourages extensibility with multiple LLM/VLM implementations.

Concrete Example Pipeline: Given a query "What type of kitchen cabinets are in the room?" and multi-view images with a 3D point cloud, the system first selects relevant views mentioning kitchen cabinets. Then, it extracts the phrase "kitchen cabinets" as object cues, runs object detection to annotate bounding boxes in the filtered views. Simultaneously, it encodes the camera poses as directional markers on a BEV map. The 3D-QA module parses structured question semantics, captions individual views using visual and spatial inputs, summarizes regional spatial information to reduce redundancy, synthesizes an answer integrating all clues (e.g., "Brown wooden built-in cabinets with drawers and doors"), then normalizes wording for evaluation. This entire process requires no gradient updates or training data specific to 3D-QA, fully leveraging large pretrained generalist models.

Technical innovations

• A fully training-free modular 3D-QA framework decomposing reasoning into question-driven multi-view selection, guided visual grounding, BEV-based viewpoint encoding, and structured answer synthesis.
• Use of bird’s-eye-view (BEV) spatial encoding of camera poses as explicit, interpretable spatial cues enabling consistent cross-view reasoning in 3D scenes.
• Guided Detection combining language-conditioned object cue extraction with an open-vocabulary detector over filtered views to provide precise visual grounding without 3D training.
• Multi-stage role decomposition of reasoning tasks via specialized agents (question interpreter, view captioner, summarizer, answerer) to improve semantic alignment without fine-tuning.
• Term-Based Instructions constraining generated answers to canonical vocabularies, improving standard language generation metrics without changing underlying reasoning.

Datasets

• ScanQA validation — ~1,500 QA pairs — derived from ScanNet
• SQA3D test set — ~477 QA pairs — derived from ScanNet with situation descriptors

Baselines vs proposed

• ScanQA: 3D-LLM (BLIP2-flant5) CIDEr = 69.4 vs ViewMind3D (o3) CIDEr = 73.41
• ScanQA: ScanQA fine-tuned baseline CIDEr = 64.9 vs ViewMind3D (GPT-4.1) CIDEr = 68.3
• SQA3D: 3D-LLM accuracy = 49.79% vs ViewMind3D (OpenAI o3) accuracy = 50.75%
• SQA3D: GPT4Scene zero-shot ~44-46% accuracy vs ViewMind3D 50.75%
• Component Ablation on ScanQA: With Guided Detection CIDEr improves 60.30 → 61.45
• Component Ablation on ScanQA: With Viewpoint Indicator CIDEr improves 57.41 → 63.48
• Role Decomposition improves ScanQA CIDEr 54.77 → 58.05
• Term-Based Instructions improve ScanQA CIDEr 58.05 → 63.48

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.28442.

Fig 1: Overview of the ViewMind3D framework for 3D-QA. Given a natural language query, multi-view images, and a 3D point cloud, the system

Fig 2 (page 3).

Fig 3 (page 3).

Fig 4 (page 3).

Fig 5 (page 3).

Fig 6 (page 3).

Fig 7 (page 3).

Fig 8 (page 3).

Limitations

• The Guided Detection module may introduce noisy or inaccurate annotations, notably reducing performance on procedural or numeric ‘How’ questions.
• The approach is limited by available multi-view coverage and quality; sparse or incomplete views may limit spatial reasoning accuracy.
• Reliance on proprietary or large pretrained LLM/VLM backbones without fully open implementations limits reproducibility.
• Performance on 'Which'-type questions requiring explicit candidate ranking or comparison is weaker due to generation-based framework without dedicated ranking mechanisms.
• No adversarial robustness evaluation against ambiguous or misleading queries is reported.
• Efficiency bottleneck remains in the 3D-QA module with substantial token consumption (~35K tokens), implying computational cost for real-time deployment.

Open questions / follow-ons

• How to integrate candidate selection or explicit ranking mechanisms to improve performance on ‘Which’-type comparative questions in 3D-QA?
• How to improve Guided Detection robustness for open-world and ambiguous object categories without degrading other question types?
• Can adaptive selection between clock-based and compass-based BEV encodings per query improve spatial reasoning granularity dynamically?
• What are the limits of training-free modular orchestration when scaling to more complex scenes or outdoor 3D environments beyond indoor ScanNet data?

Why it matters for bot defense

ViewMind3D demonstrates that sophisticated 3D spatial reasoning and question answering can be achieved in zero-shot, training-free settings by modularly orchestrating general-purpose pretrained LLMs and VLMs. For bot-defense and CAPTCHA practitioners, this implies that future CAPTCHA systems relying on 3D spatial understanding may need to consider the growing zero-shot reasoning capabilities of large multimodal models without explicit 3D-specific training. The modular design, especially the use of spatially explicit BEV encodings and language-conditioned visual grounding, suggests that defending against automated agents with such reasoning abilities may require novel challenge designs that go beyond static multi-view scene understanding. Understanding these modular inference processes can guide the development of CAPTCHA tasks that better test truly embodied spatial reasoning under adversarial settings.

Cite

@article{arxiv2607_28442,
  title={ ViewMind3D: Modular View-Aware Inference for Training-Free 3D-QA },
  author={ Ping-Kun Chiang and Kun-Ru Wu and Po-han Li and Sandeep Chinchali and Ufuk Topcu and Yu-Chee Tseng },
  journal={arXiv preprint arXiv:2607.28442},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.28442}
}

Read the full paper

• arXiv:2607.28442 (abstract)
• arXiv:2607.28442 (PDF)