BridgeVLA++: A Data-Efficient, Generalizable, and Memory-Augmented Vision-Language-Action Framework for 3D Manipulation
Source: arXiv:2608.05042 · Published 2026-08-05 · By Peiyan Li, Yuze Zhu, Yixiang Chen, Qisen Ma, Yuan Xu, Jiabing Yang et al.
TL;DR
This paper addresses critical limitations in current 3D vision-language-action (VLA) models for robotic manipulation, namely their heavy data requirements, poor generalization under distribution shifts, and lack of memory for past observations. The authors build on their previous BridgeVLA framework, which aligns 3D manipulation learning with pre-trained vision-language models (VLMs) by projecting 3D point clouds into multi-view 2D images and predicting intermediate spatial heatmaps before generating robot actions. The key innovation in BridgeVLA++ is the addition of a unified spatio-temporal memory architecture that explicitly models both temporal interaction history and persistent spatial context. This enables reasoning over historical observations, addressing memory-dependent manipulation tasks while preserving BridgeVLA's data efficiency and generalization capabilities. Extensive experiments on five simulation benchmarks and two real-world robotic platforms demonstrate that BridgeVLA++ achieves state-of-the-art performance on memory-dependent benchmarks RMBench and MemoryBench, improves bimanual manipulation, and maintains strong results on standard spatial tasks with limited data.
Key findings
- BridgeVLA++ improves success rates on memory-dependent manipulation benchmarks from 20.0% (BridgeVLA) to 93.3%, a 73.3% absolute increase.
- BridgeVLA achieves state-of-the-art results on RLBench, COLOSSEUM, and GemBench with strong data efficiency and out-of-distribution generalization.
- The spatial memory module enables occlusion-robust fine localization via re-rendering stored initial point clouds aligned with current views.
- Temporal memory comprises initial anchor views, neighboring recent keyframes (n=2), and adaptively selected sub-goal keyframes to capture interaction history.
- Coarse-to-fine prediction with shared weights: coarse stage leverages temporal memory to decide what to do next; fine stage uses spatial memory to localize precisely.
- BridgeVLA++ supports bimanual manipulation by sharing spatial memory across two arms with arm-specific action heads.
- Robust real-world performance validated on Franka Research 3 and Dobot CR5A platforms, showing scalability across robotic embodiments.
- Language-conditioned heatmap pre-training on 120K RoboPoint object detection samples aligns VLM outputs with spatial localization objectives.
Threat model
Not an explicit security paper. The implicit challenge is robust robotic manipulation under partial observability, occlusion, and distribution shift, with adversaries represented by environment variations and occlusions. The adversary cannot modify the policy's internal memory or force platform failures.
Methodology — deep read
Threat Model & Assumptions: The work assumes an embodied robot policy learner operating in 3D manipulation tasks specified by language instructions. The adversary is implicit, focusing on challenges posed by occlusions, distribution shifts, and limited data; explicit adversarial threat modeling is not the focus.
Data: The pre-training uses RoboPoint's 120K object detection images paired with language prompts describing target objects. During policy fine-tuning, the dataset consists of multi-task, language-conditioned 3D manipulation trajectories from simulation benchmarks including RLBench, COLOSSEUM, GemBench, RMBench, and MemoryBench. The datasets provide RGB-D images or 3D point clouds with expert actions (translation, rotation, gripper state, collision avoidance).
Architecture/Algorithm: The core is a pre-trained vision-language model backbone (PaliGemma) consisting of a SigLIP vision encoder and Gemma transformer. It processes multi-view orthographic 2D projections of 3D point clouds along with language instructions. Outputs are rearranged image tokens decoded via convex upsampling into 2D spatial heatmaps, representing likelihoods of target object locations. Actions are predicted by back-projecting heatmap maxima to 3D translations and regressing rotation, gripper, and collision via an MLP on pooled global and local VLM features. BridgeVLA++ adds unified spatio-temporal memory: temporal memory aggregates an initial anchor view, recent keyframes (n=2), and adaptively selected sub-goal keyframes into coarse-stage tokens; spatial memory stores the initial point cloud and re-renders it aligned to the predicted fine-stage crop for occlusion-robust localization. Multi-head attention integrates memory tokens with current observation tokens.
Training Regime: Heatmap pre-training uses cross-entropy loss over Gaussian spatial masks derived from object bounding boxes in RoboPoint. Fine-tuning uses combined losses: heatmap cross-entropy for translation, Frobenius-norm rotation loss, and binary cross-entropy for gripper and collision. Data augmentation applies rigid-body transformations jointly to input point clouds and labels. Coarse-to-fine passes share weights, with the fine stage processing zoomed local crops centered on the coarse prediction.
Evaluation Protocol: The method is evaluated on five simulation benchmarks covering spatial and memory-dependent tasks. Metrics include task success rates with ablations on memory usage and generalization to unseen objects, backgrounds, and instructions. Real-robot experiments on two different arms further validate transferability. Ablations study temporal vs spatial memory contributions. Results are reported as average success percentages.
Reproducibility: The authors provide detailed implementation and training details in appendices and release code and project website, but some datasets (RLBench etc.) are standard but may not be open-source. Pre-trained weights of VLM backbone are reused from PaliGemma's public checkpoint. Minor details such as hyperparameters and seeds are documented in supplemental material. Overall, the work appears reproducible with standard robotics datasets and open-source VLM backbones.
Example Workflow: Given a current RGB-D observation and a language instruction, BridgeVLA++ reconstructs a colored 3D point cloud and renders top/front/right orthographic images. These images, plus the instruction, are fed into the VLM backbone to generate visual tokens. Temporal memory tokens from initial anchor and keyframes are integrated at the coarse stage via attention. The coarse heatmaps predict a rough 3D translation. Then the point cloud is cropped around this coarse location, re-rendered alongside spatial memory (the initial point cloud rendered at the same zoom), and passed through a shared VLM to predict fine heatmaps. These fine heatmaps precisely localize the end-effector translation, while an MLP predicts rotation, gripper, and collision. The final 3D action is executed by the robot, and the memories are updated.
Technical innovations
- Aligning pre-trained vision-language model outputs and 3D manipulation actions in a unified multi-view 2D spatial heatmap space to preserve spatial correspondence.
- Introducing a scalable language-conditioned 2D heatmap pre-training stage on large-scale detection data to transfer object-grounding knowledge to robotic action prediction.
- Developing a unified spatio-temporal memory architecture where temporal memory integrates initial anchor views, recent keyframes, and adaptively selected sub-goals for coarse-stage decision making.
- Incorporating spatial memory that stores and re-renders initial point clouds aligned to current fine-stage views to provide occlusion-robust spatial context for precise localization.
- Extending single-arm BridgeVLA to bimanual manipulation by sharing spatial memory and using arm-specific heads while preserving sample efficiency.
Datasets
- RoboPoint 120K split (object-detection) — 120,000 images — public
- RLBench — multi-task 3D manipulation trajectories — public robotics benchmark
- COLOSSEUM — multi-task 3D manipulation — public robotics benchmark
- GemBench — specialized 3D language-conditioned manipulation benchmark — public
- RMBench — memory-dependent manipulation benchmark — public
- MemoryBench — memory-dependent manipulation benchmark — public
Baselines vs proposed
- BridgeVLA vs strong baseline on real-world memory-independent tasks: success rate +32% average improvement
- BridgeVLA++ vs BridgeVLA on memory-dependent benchmarks (RMBench and MemoryBench): success rate improved from 20.0% to 93.3%
- BridgeVLA achieves state-of-the-art on RLBench, COLOSSEUM, GemBench with strong generalization; BridgeVLA++ matches or improves upon these results while adding memory capability
- Ablation without temporal memory on RMBench significantly reduces success rate, showing temporal memory's critical role
- Ablation without spatial memory degrades fine localization performance under occlusion
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2608.05042.
Fig 2: Model Architecture. Top: BridgeVLA first learns language-conditioned 2D heatmap prediction from detection data and transfers the resulting
Fig 2 (page 4).
Fig 3 (page 4).
Fig 4 (page 4).
Fig 5 (page 4).
Fig 3: Occlusion-robust fine localization using spatial memory. Left: the
Fig 7 (page 7).
Fig 4: Real-Robot Evaluation Setup. Top left: the general-manipulation platform—a 7-DoF Franka Research 3 arm observed by a static ZED 2i stereo
Limitations
- While strong in memory-dependent tasks, the spatial memory relies on storing an initial observation point cloud, which may limit adaptability to dynamic environments with large scene changes.
- The system requires multi-view orthographic projections, calibrated cameras, and point cloud reconstruction, which may not be feasible or efficient in some real-world setups.
- Adversarial robustness or evaluation against intentional occlusion or deception is not explicitly studied.
- The adaptive sub-goal keyframe selection module is shallow and heuristic; more sophisticated memory management or compression could improve scalability.
- Real-world experiments are limited to two robotic platforms and may not generalize to more diverse or mobile embodiments.
- Memory buffering uses a fixed recent keyframe count (n=2) and fixed selection thresholds, which may be suboptimal for all task types.
Open questions / follow-ons
- How to efficiently update or maintain spatial memory in dynamic environments where scene geometry changes over time?
- Can the memory architecture be extended to longer horizon tasks requiring hierarchical planning or multi-stage memory retrieval?
- What are the trade-offs in memory capacity and latency with larger historical context inclusion for temporal memory?
- How would explicit adversarial occlusions or deceptive environment conditions impact memory-augmented policies like BridgeVLA++?
Why it matters for bot defense
For bot-defense and CAPTCHA practitioners, BridgeVLA++ represents a principled approach to data-efficient, generalizable 3D vision-language-action modeling with explicit memory, which is crucial for handling tasks where prior state and history are important. While CAPTCHAs typically do not involve physical manipulation, the concepts of persistent spatial and temporal memory could inspire defenses against multimodal automated agents that exploit short-term observation. The framework's emphasis on preserving spatial-action alignment and memory could inform robust bot-detection signals based on consistency over time and scene context. Moreover, understanding how such memory-augmented models achieve generalization with limited data may guide CAPTCHA designers aiming to detect bots with temporally consistent, stateful behavior versus stateless scripted attacks. However, direct application is limited as the paper focuses on embodied robotics rather than adversarial human-machine interaction.
Cite
@article{arxiv2608_05042,
title={ BridgeVLA++: A Data-Efficient, Generalizable, and Memory-Augmented Vision-Language-Action Framework for 3D Manipulation },
author={ Peiyan Li and Yuze Zhu and Yixiang Chen and Qisen Ma and Yuan Xu and Jiabing Yang and He Guan and Yan Huang and Hongtao Wu and Xiao Ma and Tao Kong and Liang Wang and Tieniu Tan },
journal={arXiv preprint arXiv:2608.05042},
year={ 2026 },
url={https://arxiv.org/abs/2608.05042}
}