Seek to Segment: Active Perception for Panoramic Referring Segmentation

Source: arXiv:2607.02497 · Published 2026-07-02 · By Song Tang, Shuming Hu, Xincheng Shuai, Henghui Ding, Yu-Gang Jiang

TL;DR

This paper addresses active perception for referring image segmentation in continuous 360° panoramic environments, a key challenge in Embodied AI. Traditional referring segmentation works on static images and fixed views, but real-world agents need to actively explore such panoramic spaces to find an object based on natural language instructions. The authors propose the novel Active Panoramic Referring Segmentation (APRS) task and introduce a new benchmark with 7,420 samples spanning diverse indoor and outdoor scenes that require complex spatial reasoning over four types of referring expressions. To solve the task, they build PanoSeeker, an agent combining a Vision-Language Model (VLM) with EgoSphere, an explicit spatial visual memory module that incrementally integrates local observations into a global 360° equirectangular panorama representation. PanoSeeker learns efficient search trajectories via expert-annotated supervision and reinforcement learning with a reward emphasizing exploration efficiency and accurate segmentation. Experiments demonstrate that PanoSeeker significantly outperforms static, heuristic scanning, and prior active VLM agents on metrics of success rate, steps taken, exploration efficiency (SPL), and segmentation mean IoU on the APRS benchmark.

Key findings

• PanoSeeker achieves 75.4% success rate (SR) on the APRS benchmark, surpassing prior best GPT-5.2 agent at 69.1%.
• It reduces Average Steps (AS) to 4.8, compared to 6.2 for GPT-5.2 and 12.4 for zero-shot baseline.
• PanoSeeker improves Success weighted by Path Length (SPL) to 0.57, a ~58% relative improvement over GPT-5.2 (0.36).
• Memory ablations show EgoSphere spatial visual memory boosts SR by 6.2% and SPL by 0.05 over textual log memory.
• Supervised Fine-Tuning (SFT) on expert trajectories raises SR from 56.5% to 70.3% and mIoU from 39.9% to 50.7%.
• Reinforcement Learning with GRPO further improves efficiency, reducing AS from 6.2 to 4.8 steps and increasing SPL to 0.57.
• Heuristic scanning yields poor efficiency (SPL=0.26) and low success rates (<50%), highlighting need for active spatial reasoning.
• Static methods processing whole panorama at once perform worse due to distortion and lack of exploration, with max SR 66.2%.

Threat model

n/a - This work focuses on active perception and vision-language grounding for panoramic segmentation, not on adversarial or security threats.

Methodology — deep read

The paper formalizes APRS as an active search and segmentation task in a 360° environment E driven by a language instruction I. The agent starts at initial orientation (θ0, ϕ0) with limited FoV (120°×90°), and iteratively takes actions (∆θt, ∆ϕt) to adjust viewpoint and collect observations Vt = Proj(E, θt, ϕt, ψ). At each step, the agent maintains a memory state Mt encoding cumulative explored scene knowledge. The core function F maps current visual input Vt, instruction I, and memory Mt into next action at and updated memory Mt+1.

Data was gathered by collecting high-res equirectangular panoramic images from 360-Indoor, PANDORA, and SUN360 datasets (4,971 scenes). Expert human annotators generated search trajectories for Supervised Fine-Tuning (SFT) and created spatial referring expressions with 4 categories (Egocentric, Unique-Attribute, Allocentric, Multi-hop) spanning indoor and outdoor contexts. Initial camera viewpoints were chosen away from the target to require active exploration.

PanoSeeker architecture combines a pre-trained Vision-Language Model (Qwen3-VL-8B-Instruct) with the novel EgoSphere spatial memory module. EgoSphere progressively back-projects each local observation into a fixed-resolution 360° equirectangular panorama canvas, augmented with visual spatial prompts (crosshair, latitude-longitude grid, tracked trajectory). This constructs a global visual-memory map that supports non-redundant exploration.

Training follows two stages: first SFT on expert-annotated trajectories using a cross-entropy loss to imitate optimal actions; then Reinforcement Learning (RL) via Group Relative Policy Optimization (GRPO) to improve search efficiency and success rate. Rewards balance geodesic distance progress, exploration cost penalty, and final segmentation accuracy.

Evaluation uses multiple complementary metrics on held-out test set: Success Rate (SR) measuring localization + segmentation IoU ≥0.5, Average Steps (AS) to measure exploration cost, Success weighted by Path Length (SPL) quantifying path optimality, and mean Intersection over Union (mIoU) for segmentation quality.

The framework was implemented using DeepSpeed ZeRO-2 for efficient training on 4× RTX A6000 GPUs with mixed precision. Ablations analyzed the importance of EgoSphere memory and RL optimization compared to baselines using textual or visual memory buffers. Visualization of trajectories confirmed the benefits of the explicit 360° memory representation facilitating spatial mapping and target localization.

The authors provide detailed dataset statistics, annotation protocols, and benchmark splits, though no code release or frozen weights were explicitly mentioned. The task poses new challenges by requiring 360° active perception under partial observability with spatial linguistic grounding.

Technical innovations

• Introduction of APRS, a novel task requiring active 360° exploration for language-guided object segmentation.
• EgoSphere: an explicit spatial visual memory module that back-projects local views onto a fixed-resolution equirectangular panorama canvas for unified global scene representation.
• Supervised Fine-Tuning on expert-annotated search trajectories combined with Reinforcement Learning (GRPO) to optimize exploration efficiency.
• Active Alignment and Segmentation that refines the viewpoint and segmentation mask after target detection to improve final output quality.

Datasets

• APRS Benchmark — 7,420 samples across 4,971 panoramic scenes — constructed by authors from 360-Indoor, PANDORA, SUN360 panoramic image datasets with expert-annotated spatial referring expressions and search trajectories.

Baselines vs proposed

• Static Methods (e.g., VisionReasoner): SR=66.2% vs PanoSeeker: 75.4%
• Heuristic Scanning (VisionReasoner): SR=49.5%, AS=4.9, SPL=0.26 vs PanoSeeker: SR=75.4%, AS=4.8, SPL=0.57
• Active VLM Agents (GPT-5.2): SR=69.1%, AS=6.2, SPL=0.36 vs PanoSeeker: SR=75.4%, AS=4.8, SPL=0.57
• Memory Variant - Text Log: SR=50.3%, SPL=0.21 vs EgoSphere: SR=56.5%, SPL=0.26 (zero-shot setting)
• Ablation: +SFT improves SR from 56.5% to 70.3%, +RL further improves to 75.4% SR and reduces AS to 4.8

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.02497.

Fig 1: Active Panoramic Referring Segmentation (APRS) requires an agent to search for and segment a target (e.g.,

Fig 2 (page 2).

Fig 3 (page 2).

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 2).

Fig 7 (page 2).

Fig 8 (page 2).

Limitations

• No explicit evaluation under adversarial conditions or with noisy language instructions.
• The benchmark uses simulated panoramic views from static images, lacking real-time embodied agent physical constraints.
• No publicly released code or pretrained weights at this time, limiting reproducibility.
• The environment is limited to pre-collected panoramas rather than fully dynamic 3D environments with interactive objects.
• The spatial memory EgoSphere size is fixed, which may limit scalability to very large or complex scenes.
• The action space discretization into 30° increments may limit fine-grained viewpoint control.

Open questions / follow-ons

• How would PanoSeeker perform with noisy or ambiguous natural language instructions impacting search?
• Can EgoSphere spatial memory be adapted for real-time 3D environment exploration with dynamic objects and agent movement?
• What are the implications of finer-grained viewpoint adjustments on search efficiency and segmentation accuracy?
• How robust is the model to distributional shifts, e.g., outdoor vs indoor or novel environments unseen in training?

Why it matters for bot defense

Although primarily focused on embodied AI and active panoramic segmentation, this work introduces valuable concepts for bot defense and CAPTCHA systems involving active perception under complex visual-linguistic instructions. The EgoSphere memory mechanism offers a novel approach to efficiently aggregate multi-view observations into a unified spatial representation, which could inspire CAPTCHA designs that require efficient exploration or interactive spatial reasoning that bots must learn to emulate. Furthermore, the reinforcement-learned exploration policies and explicit trajectory reasoning point toward adaptive challenge generation tailored to agent behavior. However, direct adaptation would require domain-specific tuning and deeper security threat modeling in adversarial contexts. The presented APRS benchmark and evaluation metrics provide a rigorous framework for analyzing multi-step, instruction-guided visual search complexity, potentially informing robust evaluation of human vs automated agent capabilities in CAPTCHA-like tasks.

Cite

@article{arxiv2607_02497,
  title={ Seek to Segment: Active Perception for Panoramic Referring Segmentation },
  author={ Song Tang and Shuming Hu and Xincheng Shuai and Henghui Ding and Yu-Gang Jiang },
  journal={arXiv preprint arXiv:2607.02497},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.02497}
}

Read the full paper

• arXiv:2607.02497 (abstract)
• arXiv:2607.02497 (PDF)