Learning to See While Learning to Act: Diffusion Models for Active Perception in Robot Imitation

Source: arXiv:2606.23625 · Published 2026-06-22 · By Kuancheng Wang, Vaibhav Saxena, Shuo Cheng, Yotto Koga, Danfei Xu

TL;DR

This paper addresses the challenge of robotic imitation learning under partial observability caused by occlusion, where robots must not only execute actions but also actively choose informative viewpoints to see critical scene parts. Conventional methods rely on fixed or limited viewpoints, struggling when relevant objects or locations are hidden. See2Act proposes a novel diffusion-model-based framework that simultaneously denoises robot actions and refines camera viewpoints in a coupled manner during both training and inference. This joint action-viewpoint denoising enables the robot to iteratively reposition its camera to disambiguate occlusions before executing manipulation tasks. The authors demonstrate that See2Act significantly outperforms fixed or passively selected viewpoint baselines on occlusion-heavy Ravens benchmark tasks, achieving up to 96% success compared to near 0%-50% for others. On RLBench, it matches or exceeds state-of-the-art methods using richer 3D/multi-camera inputs. Critically, See2Act also transfers zero-shot from simulation to real robots, achieving 95% success on complex occluded shelf insertion tasks with only 50 digital-twin demonstrations. This shows that learned active perception tightly coupled with action generation facilitates robust robot manipulation under challenging partial observability, reducing reliance on large data or multi-view sensors.

Key findings

• See2Act achieves 96% success on Ravens' occluded bin-picking compared to 0%-22% from fixed or passive view baselines (Table 1).
• On four Ravens tasks with occlusion, See2Act averages 91% success, outperforming C3DM by up to 88%.
• On nine RLBench manipulation tasks, See2Act matches or outperforms 3D/multi-view methods with 81.1% average success (Table 2).
• Active viewpoint refinement yields up to 34% improvement in success rate over prior methods on RLBench tasks.
• Performance improves as the number of diffusion denoising/refinement steps increase (Fig 6), e.g., bin-picking improves from ~60% at 5 steps to 96% at 20 steps.
• See2Act handles out-of-distribution initial camera poses with only ~8-10% drop in success rate (Fig 7).
• On real robot occluded pick-and-place tasks, See2Act achieves 95% success zero-shot after training on only 50 digital twin demos, versus 25% for fixed-view diffusion baselines (Table 3).
• In the bin-search task, See2Act's active viewpoint inference naturally induces search behavior, achieving 100% success where baselines fail.

Threat model

The implicit adversary is the environment itself, producing occlusions that block the robot's line of sight to task-relevant objects or placements. The robot cannot initially observe critical regions and must overcome this via active viewpoint selection. There is no adversarial agent actively manipulating sensors or environment; rather, the challenge is partial observability due to natural occlusions. The robot lacks explicit knowledge of which viewpoints are informative and must infer this from limited offline demonstrations alone.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is essentially environmental occlusion and partial observability, where the robot lacks upfront full visibility of task-relevant objects. No explicit adversary manipulation is considered. The robot must learn to actively seek informative views to infer actions accurately. The approach assumes access to offline demonstrations paired with state and keyframe actions, but no explicit viewpoint annotations.

2. Data: Training data consists of offline manipulation demonstrations containing initial object states and target pick/place keyframe actions (6-DoF poses). Data is generated in simulation using a digital twin capable of rendering observations from arbitrary camera poses. Each demonstration yields multiple noisy action samples paired with observations from interpolated camera poses anchored at a global view (broad overview) and a keyframe-centric view (close, target centric) during diffusion training. The Ravens environment benchmarks occluded tasks, and RLBench provides diverse realistic robot tasks. Real-world testing uses 50 digital twin demonstrations.

3. Architecture/Algorithm: The policy is a diffusion model over latent robot actions conditioned on image observations. It operates in camera coordinate frame, coupling action denoising with camera pose update at each diffusion timestep. The visual encoder extracts features from rendered RGB-D observations. The noise prediction network estimates Gaussian noise components to iteratively denoise robot action and simultaneously update the camera pose using spherical linear interpolation between a global and target-centric anchor pose. This coupling enables active perception as the camera repositioning trajectory emerges organically during denoising rather than requiring an explicit viewpoint planner.

4. Training Regime: Traces keyframe actions a0 are transformed into multiple camera frames Ct along a smooth interpolation trajectory from global to target-centric views. Actions are progressively noised to ˆat with Gaussian noise at diffusion timestep t, paired with rendered images Ot at Ct. The model jointly trains the visual encoder and noise prediction network to minimize mean squared error between predicted and true noise over N offline demos and K samples each. Training specifics such as epochs, batch size, hardware are in appendix (not fully specified). No real-world fine-tuning is done.

5. Evaluation Protocol: Success rate is the primary metric measured across 50 trials for Ravens and 4 trials per task for RLBench, comparing against baselines including fixed-view behavioral cloning, multi-view with and without diffusion, entropy-based passive view selection, and restricted camera motion methods like C3DM. In real world, success measures precise insertion or placement within tight tolerances under occlusion. Ablations track performance versus refinement steps and sensitivity to initial camera poses. Qualitative analysis of viewpoint trajectories show emergent search behavior. Statistical tests are not explicitly reported.

6. Reproducibility: The paper provides a project website and pseudocode. Training relies on simulation with a digital twin renderer and 50 real-world demos for transfer. Baselines use the same backbone for direct comparison. However, exact training hyperparameters and code repository links are not detailed within the given excerpt, so full reproduction may require contacting authors or awaiting release.

Concrete Example: Consider the bin-picking Ravens task where the target red block is fully occluded in the global overhead view. At training, the model learns to denoise the target end-effector pick pose progressively while interpolating camera poses from a broad overview to a close-up target-centric position derived from the true pick pose. At inference, starting with the initial global view, the algorithm alternates action denoising and camera pose updates, repositioning the camera to viewpoints that successively reveal the occluded block, ultimately enabling an accurate pick action generated fully from partial visual input. This contrasts with fixed-view baselines which never see the block and fail the task.

Technical innovations

• Coupling diffusion-based action denoising with full 6-DoF camera viewpoint refinement within a unified generative process for active perception.
• Anchoring camera pose interpolation to keyframe action poses, enabling implicit learning of viewpoint selection from demonstration action labels alone without explicit viewpoint supervision.
• Active Viewpoint Inference during test-time uses predicted action states to iteratively refine camera poses and observations to disambiguate occlusions without separate planning or reinforcement learning.
• Demonstration-driven viewpoint-agnostic training via simulation-based online rendering to generate diverse paired action-viewpoint-noise samples enabling robust zero-shot sim-to-real transfer.

Datasets

• Ravens Benchmark — variable per task, 50 trials per task — public simulation environment
• RLBench — 9 tasks, 4 trials per task evaluation — public benchmark for robotic manipulation
• Digital Twin Simulation — 50 demonstrations per real-world task — private synthetic dataset replicating real robot and environment

Baselines vs proposed

• Conv-MLP (fixed single view): 0% success on Ravens occluded tasks vs See2Act 96% on bin-picking
• Conv-MLP-MV (fixed multi-view): max 2% success on Ravens occluded tasks vs See2Act 96%
• Diff-MV (diffusion on fixed multi-view): 0% success on Ravens occluded vs See2Act 96%
• Diff-MV-Entropy (passive view selection): 22% success on bin-picking vs See2Act 96%
• C3DM (restricted zoom camera motion): 8% on bin-picking vs See2Act 96%
• On RLBench, PerAct avg success 55.5% vs See2Act 81.1%; RVT-2 avg 77.9% vs See2Act 81.1%
• Real-world pick-and-place shelf-kitting: Diff-MV 25% success vs See2Act 95% using 50 demos

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.23625.

Fig 1: See2Act couples seeing and acting in a single denoising loop. At each step the camera

Fig 2: Training and Inference Pipeline in See2Act. (Left: Training) Given demonstrations

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• Inference requires sequential viewpoint refinement with multiple camera movements and observations, incurring latency compared to fixed-viewpoint methods.
• Training relies heavily on simulation and digital twins with accurate online rendering; realism gaps may affect transfer.
• Experiments focus on pick/place keyframe actions rather than dense continuous trajectories, limiting extension to dynamic tasks.
• Evaluation under adversarial occlusions or environmental perturbations is not explored.
• Performance degrades 8-10% on out-of-distribution initial camera poses, indicating sensitivity to starting viewpoint.
• Exact training hyperparameters and code are not fully disclosed, possibly limiting immediate reproducibility.

Open questions / follow-ons

• How well does See2Act generalize beyond tabletop pick-and-place tasks to more dynamic or continuous manipulation requiring full trajectory prediction?
• Can the active viewpoint inference framework be integrated with real-time planning and control to reduce latency during iterative refinement?
• How robust is the method to adversarial or unexpected occlusions beyond those present in training simulations or digital twins?
• What are the impacts of noisy or imprecise camera calibration on active viewpoint refinement over extended operational time?

Why it matters for bot defense

For bot-defense and CAPTCHA-like challenges where partial observability and adversarial occlusion could impair automated vision systems, See2Act provides insights into coupling observation acquisition and action prediction in a unified framework. Although robotics and CAPTCHA domains differ, the principle of active perception—iteratively acquiring task-relevant views to reduce ambiguity—could guide design of adaptive sensing mechanisms in bot detection systems. The use of diffusion models to jointly model observation and action trajectories may inspire future CAPTCHA defenses requiring bots to 'explore' or adaptively query distorted or occluded visual puzzles rather than relying on fixed, passive inputs.

Additionally, the zero-shot sim-to-real transfer indicates potential for policies that generalize robustly under varied occlusion conditions with limited real-world data—a desirable property for bot defenses needing to perform reliably despite adversarially manipulated inputs. However, the latency and multi-step querying of See2Act may pose usability tradeoffs in fast CAPTCHA validation settings. Overall, its active viewpoint refinement approach expands the toolkit for tackling partial observability challenges relevant across automated vision tasks beyond robotics.

Cite

@article{arxiv2606_23625,
  title={ Learning to See While Learning to Act: Diffusion Models for Active Perception in Robot Imitation },
  author={ Kuancheng Wang and Vaibhav Saxena and Shuo Cheng and Yotto Koga and Danfei Xu },
  journal={arXiv preprint arXiv:2606.23625},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.23625}
}

Read the full paper

• arXiv:2606.23625 (abstract)
• arXiv:2606.23625 (PDF)