Knowing the Self, Understanding the World: A Dual-Cognition Benchmark for UAV Spatio-temporal Reasoning with MLLMs

Source: arXiv:2607.16193 · Published 2026-07-17 · By Like Liu, Zhengzheng Xu, Haitao He, Hongzhe Li, Shuchang Zhang, Dian Shao

TL;DR

This paper introduces UAV-DualCog, a novel benchmark designed to evaluate the dual-cognition capabilities of multimodal large language models (MLLMs) in UAV aerial scenarios. Unlike prior UAV benchmarks that typically focus on either environment understanding or task completion, UAV-DualCog explicitly assesses reasoning about both the UAV's own self-state (e.g., position, viewpoint, motion) and the external environment state in multiview, spatio-temporal contexts. It provides a scalable automated pipeline to generate image and video tasks from scene-level semantic point clouds with rich spatial and temporal grounding annotations, covering 12 UAV scenarios and over 500 landmarks with thousands of QA samples. Extensive experiments across proprietary, open-source, and specialized spatial reasoning models reveal persistent challenges in self-state reasoning, viewpoint transformation, precise spatial grounding, and temporal interval localization, with environment-state cognition generally easier than self-state cognition. Additional diagnostics with thinking-enabled and frontier models, as well as human baselines, confirm the benchmark is understandable to humans yet very challenging for current MLLMs. Finally, they demonstrate the benchmark’s training split can provide useful structured supervision, indicating its potential to advance UAV-oriented multimodal reasoning through both evaluation and data resources.

Key findings

• UAV-DualCog covers 12 UAV scenarios with 512 valid landmarks, yielding 4,096 image and 2,048 video QA samples with structured spatial and temporal annotations.
• MLLMs show moderate answer accuracy (~30%-60%) on image tasks but substantially weaker spatial grounding (mean IoU often below 30%), indicating localization is a key bottleneck.
• Environment-state cognition tasks generally outperform self-state cognition on images, highlighting difficulty in UAV-relative position reasoning.
• Video tasks involving temporal localization of flight behaviors and landmark visibility show highly uneven model performance (composite behavior recognition accuracy as low as ~20%), and temporal grounding remains a major challenge.
• Specialized spatial reasoning models do not consistently outperform general MLLMs, revealing that accurate UAV dual-cognition requires beyond coarse recognition.
• Thinking-enabled MLLM variants yield mixed improvements; higher answer accuracy does not reliably translate to better spatial grounding.
• Human baseline significantly outperforms all models in grounding, confirming benchmark clarity but model challenge.
• A lightweight optimization probe using the UAV-DualCog-Train split demonstrates the benchmark’s utility as a structured supervision resource for UAV-oriented MLLMs.

Threat model

Not a security-focused paper; threat model not applicable. However, implicitly the benchmark tests adversarial model failures in dual-cognition UAV reasoning where models must reliably infer both self state and environment state under dynamic viewpoint and temporal conditions.

Methodology — deep read

The paper formulates UAV embodied reasoning as a dual-cognition problem requiring simultaneous modeling of self-state cognition (UAV pose, position, viewpoint, motion) and environment-state cognition (landmarks, visibility, spatial relations) in dynamic aerial contexts. The authors develop six key tasks across image and video modalities: four image tasks evaluating self-relative position, future observation prediction (self-state), landmark-relative direction, and landmark-driven action decision (environment-state); and two video tasks evaluating flight behavior recognition with temporal localization (self-state) and landmark visibility counting with visible interval reasoning (environment-state). All tasks require structured output with spatial (bounding boxes) or temporal (intervals) grounding beyond discrete answers.

Data construction relies on a scalable automatic pipeline over 18 simulated UAV scenes from AerialVLN, producing semantic scene-level point clouds that fuse LiDAR, RGB imagery, segmentation, and pose data. Landmark assets are extracted, associated with multiview observations, and manually verified for geometry and semantics. Flight trajectories are generated through a hierarchical flight behavior framework that defines patterns such as orbits and inspections, validated for collision and visibility constraints, and rendered into first-person videos. Image QA samples are synthesized by controlled viewpoint sampling and visibility checks from landmark assets.

Evaluation involves submitting strict JSON outputs with predicted discrete answers plus bounding boxes or temporal intervals. Metrics include answer accuracy, bounding box accuracy at IoU≥0.5, mean IoU, temporal intersection-over-union (tIoU) at ≥0.5, and averaged temporal IoU for video tasks. The benchmark provides a uniform prompt schema and ensures consistent parsing and evaluation rules.

The authors benchmark three model groups under UAV edge deployment constraints: proprietary commercial MLLMs (GPT-5.x series, Claude Sonnet, Gemini 3, Qwen variants), open-source MLLMs (GLM, InternVL, Kimi, Mimo), and specialized spatial reasoning models fine-tuned on spatial tasks. They run evaluation on a moderate scale (thousands of QA samples with compressed 640×480 imagery and 10M bitrate videos).

For interpretability, they conduct a parsing audit to correct for bounding box formatting errors and diagnose failures. Further analyses include ablations by task type and cognition dimension, comparisons of instant vs thinking modes of models to isolate the impact of textual deliberation, and trials with frontier large models and a human baseline for upper bound clarity.

A lightweight test fine-tuning experiment on the UAV-DualCog-Train split, constructed from disjoint scenes, evaluates whether the benchmark data can provide useful structured supervision to improve UAV MLLM performance. All code and materials are publicly released for reproducibility, though the dataset is simulation-based rather than collected from real UAV flights, with sim-to-real transfer remaining future work.

Technical innovations

• Formulation of UAV embodied reasoning as dual-cognition: jointly reasoning about self-state and environment-state in multiview spatio-temporal contexts.
• Design of UAV-DualCog benchmark tasks with both image and video modalities requiring discrete answers plus explicit spatial (bounding box) and temporal (visible interval) grounding.
• Scalable automated data construction pipeline using semantic point clouds and hierarchical behavior-driven flight trajectory generation for diverse UAV scenarios.
• Comprehensive evaluation protocol combining answer accuracy, spatial and temporal grounding metrics under UAV edge deployment constraints.
• Lightweight optimization probe demonstrating UAV-DualCog’s value as both an evaluation benchmark and a structured data resource for UAV MLLM training.

Datasets

• UAV-DualCog — 12 simulated UAV scenes, 512 landmarks — constructed from AerialVLN simulator scenes
• UAV-DualCog-Train — disjoint subset of UAV-DualCog scenes — released as a structured supervision training split

Baselines vs proposed

• GPT 5.5 image answer accuracy: up to 65.4% on environment-aware tasks vs spatial grounding mIoU often <25%
• Gemini 3 Flash image self-aware answer accuracy: 47.6% but spatial grounding under 1.5% mIoU
• Mimo v2.5 open-source model outperforms many proprietary ones in spatial grounding metrics (mean IoU ~30%)
• Flight behavior recognition composite accuracy: Gemini 3 Flash 47.9% vs open-source best ~39.8% (InternVL 3.5-4B)
• Landmark visibility counting accuracy: up to 59.9% by Gemini 3 Flash, but temporal localization remains below 50%
• Specialized spatial reasoning models generally perform worse than general MLLMs on UAV-DualCog spatial/temporal tasks

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.16193.

Fig 1: Overview of UAV-DualCog, a benchmark for aerial multiview spatio-temporal reasoning. It explicitly formulates

Fig 2: The construction process of UAV-DualCog benchmark. Above: The automatic process includes scene scanning and

Fig 3: Statistics of UAV-DualCog benchmark, showing

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 2).

Fig 7 (page 2).

Fig 8 (page 2).

Limitations

• Benchmark constructed entirely in simulation, so real-world transfer (sim-to-real gap) is untested and left for future work.
• Current MLLM evaluations use compressed images/videos which may limit visual fidelity compared to raw UAV data.
• Evaluation uses lightweight 'Instant' mode without exploration of heavy reasoning or multimodal interactive capabilities.
• Self-state cognition tasks remain challenging, revealing fundamental limitations of existing MLLMs in egocentric spatial reasoning.
• Parsing errors and format incompatibilities cause some underestimation of spatial grounding, though corrected diagnostics address this partially.
• Specialized spatial reasoning fine-tuned models do not clearly outperform general MLLMs, indicating insufficient adaptation for UAV contexts.

Open questions / follow-ons

• How can sim-to-real transfer be improved to enable UAV Dual-Cognition reasoning with real-world UAV data and imagery?
• What architectural improvements or auxiliary supervision strategies can close the gap in self-state spatial reasoning and temporal grounding?
• Can integrating longer-horizon planning or multimodal interaction (e.g., active querying) improve UAV embodied cognition beyond instant reasoning modes?
• How might contrastive or representation learning techniques leverage the benchmark’s structured annotations to better unify self- and environment-state modalities?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, UAV-DualCog illustrates the complexity of spatio-temporal grounding in dynamic embodied vision-language systems, highlighting challenges in models' ability to jointly reason about an agent's self-position and external environment from multiple viewpoints over time. This dual-cognition perspective parallels challenges in developing defenses that require nuanced multimodal understanding beyond static imagery or simple discrete answers — such as grounding responses spatially and temporally. Those designing CAPTCHAs or bot-detection schemes involving aerial or dynamic camera data can draw from UAV-DualCog’s rigorous structured annotation and evaluation methodology to test model robustness in grounding and state-awareness, which bots currently fail at reliably. Moreover, the benchmark's framework of assessing both discrete answer correctness and evidence grounding could inspire richer CAPTCHA challenges that demand interpretable, spatially or temporally grounded responses rather than solely classification accuracy.

Cite

@article{arxiv2607_16193,
  title={ Knowing the Self, Understanding the World: A Dual-Cognition Benchmark for UAV Spatio-temporal Reasoning with MLLMs },
  author={ Like Liu and Zhengzheng Xu and Haitao He and Hongzhe Li and Shuchang Zhang and Dian Shao },
  journal={arXiv preprint arXiv:2607.16193},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.16193}
}

Read the full paper

• arXiv:2607.16193 (abstract)
• arXiv:2607.16193 (PDF)