3D-Aware VLMs with Implicit and Explicit Geometries

Source: arXiv:2607.21595 · Published 2026-07-23 · By Wenhao Li, Xueying Jiang, Quanhao Qian, Deli Zhao, Ran Xu, Shijian Lu et al.

TL;DR

This paper addresses the challenge that existing vision-language models (VLMs), primarily built on 2D visual input, lack fine-grained 3D spatial understanding, which is critical for many 3D scene reasoning tasks. The authors propose VLM-IE3D, a unified framework that enriches VLMs with both implicit and explicit 3D geometric representations learned from RGB videos, without requiring specialized 3D sensors or explicit 3D input. Implicit Geometry Tokens (IGTs) are learned via a 3D geometry encoder to capture coarse global spatial priors, while Explicit Geometry Tokens (EGTs) encode detailed local geometric structures from reconstructed explicit 3D attributes like depth maps. These complementary representations are fused via a novel 3D-aware adapter module together with 2D visual tokens to produce a strongly 3D-aware visual embedding for language-driven 3D spatial reasoning. Extensive experiments across 3D video detection, visual grounding, dense captioning, and spatial reasoning benchmarks demonstrate that VLM-IE3D consistently outperforms previous SOTA RGB-only VLMs, narrowing the gap with methods that rely on explicit 3D inputs. The approach injects strong 3D inductive biases while maintaining efficiency with only a minor parameter increase and computational overhead.

Key findings

• VLM-IE3D achieves 80.4% [email protected] on 3D dense captioning (Scan2Cap), surpassing Qwen2.5-VL-3B baseline by 22.4 points and VG LLM by 1.8.
• On 3D visual grounding (ScanRefer), VLM-IE3D attains 43.2% and 16.9% accuracy at IoU thresholds 0.25 and 0.5 respectively, improving over Qwen2.5-VL-3B by 9.2 and 6.3 points.
• On 3D video detection, VLM-IE3D achieves 44.2% precision, 41.9% recall, and 42.8% F1 (IoU 0.25), improvements of 12.1, 11.8, and 11.9 over baseline and 2.5, 6.2, 4.6 over VG LLM.
• Adding IGTs alone improves F1 from 30.9% to 40.5%, EGTs alone to 34.7%, and combining both reaches 42.8%, showing complementarity of implicit and explicit geometries.
• The proposed implicit-explicit attention fusion outperforms concatenation, weighted addition, and direct addition, achieving the best F1 gain in 3D video detection.
• Different explicit 3D attributes (depth maps, point maps, Gaussian splats) all boost performance similarly; depth maps are chosen for efficiency.
• A lightweight explicit embedding module (1-layer patch + 2-layer MLP) outperforms deeper encoders in effectiveness and training stability, with negligible parameter increase (0.008B).
• VLM-IE3D achieves state-of-the-art average performance (47.6%) on the spatial reasoning benchmark VSI-Bench with only 4B parameters, outperforming larger open-source and proprietary models.

Threat model

The key challenge addressed is how to equip vision-language models with comprehensive 3D spatial understanding solely from RGB video data without access to explicit 3D inputs like point clouds or depth from sensors. The 'adversary' is essentially the incomplete and ambiguous geometric information available from 2D images, complicating fine-grained spatial reasoning. The method cannot leverage ground-truth 3D or rely on specialized sensors; instead it depends on learning implicit priors and reconstructing explicit geometric cues from RGB alone.

Methodology — deep read

The core aim is to enhance 3D spatial reasoning of large vision-language models (VLMs) that traditionally rely only on 2D inputs. The threat model is the challenge of 3D spatial understanding and fine-grained geometry extraction purely from RGB videos, without ground-truth 3D data or specialized sensors. The adversary is thus inherent ambiguity and lack of explicit geometric cues in 2D data.

The input data consists of RGB video sequences of length f frames, resized to 392x518 resolution and tokenized into patches of size 14x14 pixels producing n = 1036 tokens per frame. No ground-truth 3D annotations are required for training. Datasets used for downstream tasks include Scan2Cap, ScanRefer, EmbodiedScan-based 3D detection, and spatial reasoning benchmarks (VSI-Bench).

Architecture-wise, VLM-IE3D builds on a frozen 2D visual encoder from Qwen2.5-VL, extracting 2D visual tokens T2D. A pre-trained 3D geometry encoder (AnySplat) processes the RGB frames to extract Implicit Geometry Tokens (IGTs) TI representing latent global 3D scene priors via multi-frame self-attention fusion decoder outputs. Explicit 3D attributes (e.g., depth maps, camera poses) are reconstructed as intermediate outputs from the 3D encoder and converted to Explicit Geometry Tokens (EGTs) TE via a lightweight 3D explicit embedding module (1-layer patch embedding + 2-layer MLP).

A novel 3D-aware adapter fuses these three token sets: first, spatially compressing tokens by merging adjacent 2x2 patches per type with an MLP; then aligning and fusing IGTs and EGTs via a multi-head cross-attention based implicit-explicit attention (IEA) module that treats IGTs as queries and EGTs as keys/values; lastly, adding the resulting 3D fused tokens with compressed 2D tokens element-wise to produce unified 3D-aware tokens. These fused representations are fed into the frozen VLM backbone with the input query text for final reasoning.

Training uses one epoch with Adam optimizer on 8 H100 GPUs, batch size 1 per GPU, learning rate warmed up to 1e-5 then decayed. The 2D encoder and 3D geometry encoder remain frozen; only the VLM backbone and explicit embedding are trained. Evaluation is performed on individual benchmarks for 3D dense captioning, visual grounding, 3D video detection, and spatial reasoning.

Metrics include accuracy at different IoU thresholds for visual grounding, precision/recall/F1 for video detection at IoU 0.25, captioning metrics like [email protected] and [email protected], and aggregate accuracy percentages. Baselines compared include Qwen2.5-VL, VG LLM, and various 3D-scene-input methods. Extensive ablations dissect contributions of implicit vs explicit tokens, fusion strategies, explicit attribute types, and explicit embedding module designs.

Code and pre-trained weights are released publicly for reproducibility. Overall, the method is RGB-only, efficient, with modest parameter and compute overhead for significant 3D understanding gains.

Example end-to-end: An RGB video is passed through the 2D encoder and 3D geometry encoder (AnySplat). The 3D encoder outputs implicit tokens (IGTs) and explicit 3D reconstructions like depth maps. Depth maps are embedded into EGTs by a lightweight module. The IEA module fuses IGTs and EGTs, then these fused features are added to 2D visual tokens. The combined 3D-aware tokens and text query tokens are input to Qwen2.5-VL for spatial reasoning, enabling output generation such as 3D bounding box localization or dense caption description without explicit 3D inputs.

Technical innovations

• Introduction of complementary Implicit Geometry Tokens and Explicit Geometry Tokens to jointly represent global coarse and local fine-grained 3D scene structures from RGB videos.
• Design of a novel 3D-aware adapter that employs a multi-head cross-attention based implicit-explicit attention (IEA) module to fuse implicit and explicit 3D geometric tokens effectively.
• Development of a lightweight and efficient 3D explicit embedding module transforming reconstructed explicit 3D attributes (depth maps, point clouds) into latent embeddings compatible with VLM token space.
• A purely RGB input pipeline that injects strong 3D inductive biases into large vision-language models without requiring any explicit 3D data or specialized sensors.

Datasets

• Scan2Cap — size unspecified — public indoor 3D dense captioning benchmark
• ScanRefer — 36,665 descriptions with bounding boxes across 562 indoor scans — public
• EmbodiedScan-based 3D video detection dataset — 958 train and 243 eval scenes, sequences of 4 frames — curated by [50]
• VSI-Bench — spatial reasoning benchmark with egocentric-allocentric tasks — public
• SPAR-7M subset — 234K samples for training spatial reasoning
• LLaVA-Hound subset — 63K samples for spatial reasoning

Baselines vs proposed

• Qwen2.5-VL-3B baseline: Scan2Cap [email protected] = 58.0 vs VLM-IE3D 80.4
• VG LLM: Scan2Cap [email protected] = 78.6 vs VLM-IE3D 80.4
• Qwen2.5-VL-3B: ScanRefer [email protected] = 34.0 vs VLM-IE3D 43.2
• VG LLM: ScanRefer [email protected] = 36.4 vs VLM-IE3D 43.2
• Qwen2.5-VL-3B: 3D video detection [email protected] = 30.9 vs VLM-IE3D 42.8
• VG LLM: 3D video detection [email protected] = 38.2 vs VLM-IE3D 42.8
• Fusion ablations: concat fusion F1=41.5 vs IEA fusion F1=42.8
• Explicit attribute ablations: depth map F1=42.8, point map F1=42.6
• Spatial reasoning (VSI-Bench): Gemini-1.5-Pro 45.4% vs VLM-IE3D 47.6%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.21595.

Fig 1: (a) Existing VLMs acquire 3D awareness by learning solely implicit geometric

Fig 2: Overview of the VLM-IE3D framework. VLM-IE3D directly processes RGB

Fig 3: Illustration of our 3D-aware adapter architecture. It takes the implicit and

Fig 4 (page 4).

Fig 5 (page 4).

Fig 4: Qualitative comparison of 3D visual grounding among Qwen2.5-VL, VG LLM,

Fig 5: Qualitative comparison of 3D video detection among Qwen2.5-VL, VG LLM, and

Fig 8 (page 11).

Limitations

• Model training is limited to one epoch on 8 H100 GPUs which may limit exploration of convergence behavior or robustness.
• Evaluation focuses on indoor 3D scene understanding datasets; effectiveness in outdoor, highly dynamic, or cluttered scenes remains untested.
• The method relies on reconstructed explicit 3D attributes derived from the 3D encoder's predictions rather than ground-truth 3D data, which may propagate errors in challenging scenarios.
• Fusion and embedding module designs were empirically optimized, but no theoretical analysis on fusion mechanisms or token alignment was provided.
• The approach introduces mild inference slowdown (1 FPS drop) compared to implicit-only baselines, which may impact real-time applications with constrained compute.
• Qualitative failure cases or adversarial robustness against ambiguous or occluded spatial queries were not evaluated.

Open questions / follow-ons

• How well does the approach generalize to outdoor or highly dynamic scenes where 3D reconstruction from RGB videos is more challenging?
• Can the implicit-explicit fusion mechanism be further optimized or made adaptive to different scene complexities or tasks?
• How robust is VLM-IE3D to noisy or incomplete reconstructed 3D attributes, and can uncertainty-aware representations improve reliability?
• Could additional cues like multi-view geometry, motion parallax, or learned 3D priors be integrated to further enhance performance without explicit 3D sensors?

Why it matters for bot defense

For bot-defense engineers and CAPTCHA practitioners, this research shows a promising avenue to bolster visual language models’ 3D spatial awareness without requiring additional hardware like depth sensors. VLM-IE3D’s approach of jointly leveraging implicit and explicit 3D representations reconstructed from RGB videos encourages designing defenses that rely on fine-grained 3D spatial cues in videos rather than only 2D image features. This can improve robustness against bots that process only 2D signals or struggle with 3D spatial reasoning, essential for verifying human-like scene understanding in dynamic video challenges. Moreover, the efficient architecture and minimal overhead imply feasibility for practical deployment in security-sensitive applications requiring spatial reasoning.

Cite

@article{arxiv2607_21595,
  title={ 3D-Aware VLMs with Implicit and Explicit Geometries },
  author={ Wenhao Li and Xueying Jiang and Quanhao Qian and Deli Zhao and Ran Xu and Shijian Lu and Gongjie Zhang },
  journal={arXiv preprint arXiv:2607.21595},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.21595}
}

Read the full paper

• arXiv:2607.21595 (abstract)
• arXiv:2607.21595 (PDF)