Learning Action Priors for Cross-embodiment Robot Manipulation

Source: arXiv:2606.26095 · Published 2026-06-24 · By Dong Jing, Tianqi Zhang, Jiaqi Liu, Jinman Zhao, Zelong Sun, Li Erran Li et al.

TL;DR

This paper addresses a key challenge in Vision-Language-Action (VLA) robot manipulation models: while Vision-Language Models (VLMs) encode strong visual and linguistic priors, their attached action modules typically lack any explicit prior knowledge of physical motion. As a result, VLA training must simultaneously learn cross-modal alignment and temporal action dynamics from scratch, which slows convergence and reduces performance especially in heterogeneous cross-embodiment scenarios. The authors propose a two-stage training framework that first pretrains the action module to learn a motion prior purely from unconditioned robot action trajectories without any visual or language input. This prior is then transferred to the VLA training stage by decoder reuse and latent distillation, allowing the VLA to start with a structured action representation that encodes temporal dynamics across embodiments. Additionally, the pretrained encoder compresses state-action history into a compact latent token to inject temporal context cheaply during VLA training. Extensive experiments spanning 13 tasks from two simulation benchmarks (LIBERO and RoboCasa) and a real Franka robot show that incorporating action priors enables faster policy convergence, higher success rates, and improved generalization particularly in data-scarce real-world tasks. Increasing the amount of action-only data further boosts the prior’s generality and downstream benefits. The approach reduces the training burden on the VLA by disentangling temporal action learning from cross-modal alignment.

Key findings

• Integrating pretrained action priors accelerates VLA convergence, yielding smoother action trajectories during training as illustrated in Fig 1(b,c).
• Across 13 diverse cross-embodiment tasks (LIBERO, RoboCasa, real-world Franka), the method achieves higher success rates than training without action priors, with improvements especially pronounced on real-world tasks with limited data.
• Scaling unconditioned action-only Stage 1 data results in broader, more generalizable action priors that further improve downstream VLA performance.
• The encoder-decoder action module compresses extended action chunks into compact latent embeddings that capture temporal motion structure, enabling robust modeling of cross-embodiment action distributions.
• Early-stage latent distillation aligns VLM predicted action embeddings with the pretrained action module’s latent space, stabilizing training gradients and enhancing policy learning efficacy.
• History compression via the pretrained encoder encodes past state-action trajectories into a single context token, improving long-horizon task performance with minimal overhead.
• Ablations show that removing Stage 1 learning or history compression degrades convergence speed and final success rates significantly, confirming each component’s contribution.
• Unified 37D action and 74D state representation across heterogeneous embodiments facilitates joint training across disparate robots without fine-tuning.

Threat model

n/a — this is a robot learning methodology paper focused on policy training efficiency and generalization rather than security or adversarial scenarios.

Methodology — deep read

1. Threat Model & Assumptions: The work focuses on robot policy learning rather than adversarial security, so the threat model is n/a. The challenge targeted is the optimization bottleneck when training a VLA model that combines a pretrained Vision-Language Model backbone with an action module initialized without motion priors. Assumptions include access to heterogeneous robot embodiment datasets containing state-action trajectories, visual observations, and language instructions.

2. Data: Experiments use a cross-embodiment dataset combining trajectories from three main domains — LIBERO (simulated tasks, 4 task suites with ~500 demos per task), RoboCasa GR1 humanoid robot simulator (5 tabletop tasks, 1000 trajectories total), and real-world Franka robot (~50 demos per 4 tasks). Combined training sets include approx 565,000 state-action transitions. Real-world data is the long-tail subset (~7.6%). Unified state (74D) and action (37D) vector spaces are defined with zero padding and soft dataset tokens disambiguating embodiments.

3. Architecture and Algorithm: The core novelty is a two-stage training pipeline. Stage 1 action prior learning uses an encoder-decoder Transformer module trained solely on interleaved state-action trajectories (no vision or language) to reconstruct action chunks via a flow-matching objective. The encoder compresses the temporal sequence into a single normalized latent embedding capturing cross-embodiment motion structure. The decoder reconstructs actions conditioned on this latent latent embedding and noise schedule, optimizing a velocity field prediction loss.

Stage 2 reuses the pretrained decoder as the VLA action head and distills the encoder's latent embeddings as supervised targets into the Vision-Language Model's predicted embeddings using a decaying latent alignment loss. The VLA takes inputs of soft dataset prompt tokens, visual observations, language instructions, and optionally a compact history token (also encoded with the pretrained action encoder). The total Stage 2 loss combines the main action prediction loss and the latent alignment plus reconstruction losses with a decaying weight to allow smooth transition to end-to-end optimization.

4. Training Regime: Stage 1 action prior training optimizes parameters of encoder and decoder on action-only data with a reconstruction objective implemented as flow matching. Stage 2 VLA training jointly trains the backbone VLM and the pretrained decoder, supervised by behavior cloning loss and latent alignment distillation. Early-step distillation decreases over time. Histories are optionally compressed into a latent token used as input to the VLM. Details on batch sizes, epochs, optimizer specifics, or hardware are not explicitly detailed in the provided text.

5. Evaluation Protocol: Models are trained on the full cross-embodiment mixture and evaluated across all 13 tasks without environment-specific fine-tuning. Metrics include success rates over repeated rollouts with fixed random seeds. Multiple baselines are compared, including the same architecture trained from scratch without Stage 1 prior and variants adding history compression. Ablations confirm key component contributions. No explicit cross-validation is mentioned but held-out test tasks and data-scarce real-world tasks serve as generalization tests.

6. Reproducibility: The paper references use of the StarVLA codebase for model training under consistent data and training protocols. It is unclear if code or pretrained weights are publicly released. Datasets (LIBERO, RoboCasa) are public but real robot data likely private.

Concrete example: For a pick-and-place task on the Franka arm, Stage 1 pretrains the encoder-decoder on just state-action trajectories recorded from demonstrations to learn temporal motion embeddings. Then in Stage 2, the pretrained decoder initializes the VLA action head that receives visual observations and language instructions; the encoder provides latent embeddings that supervise the VLM representation and history compression. This yields more stable, smooth trajectories and improved success versus training the entire policy end-to-end from scratch.

Technical innovations

• Introduction of a two-stage training framework that first learns an explicit cross-embodiment temporal action prior solely from unconditioned action trajectories before joint VLA training.
• Design of a flow-matching Transformer encoder-decoder architecture for modeling continuous action chunks as latent embeddings that capture macroscopic motion dynamics across heterogeneous robot embodiments.
• A novel latent alignment distillation method that transfers the pretrained action encoder latent space to supervise the predicted action embeddings from visual-linguistic inputs during early-stage VLA training.
• Reuse of the pretrained action decoder as the VLA action head to embed motion priors directly into policy outputs, resulting in smoother trajectories and faster convergence.
• Leveraging the action encoder as an efficient history compressor that condenses past state-action sequences into a single latent context token for history-aware robot policy modeling at negligible computational cost.

Datasets

• LIBERO — 4 task suites with 500 demonstrations each (~2000 total) — public simulation benchmark
• RoboCasa GR1 — 5 tabletop tasks, 1000 trajectories total — public humanoid simulation
• Real-world Franka arm tasks — 4 pick-and-place tasks, 50 demos each (~200 total) — proprietary real robot data

Baselines vs proposed

• No Action Prior baseline (same architecture, no Stage 1 prior learning): slower convergence, lower success rates across all tasks, especially on data-scarce real-world tasks
• Action-State Prior (Stage 1 action and proprioceptive state trajectories): significantly faster convergence and improved success rates compared to no prior
• Action-State Prior + History Compression (full method): best performance with further gains on long-horizon tasks and improved stability
• Scaling Stage 1 action-only data size leads to monotonic improvements in downstream VLA success rates (exact metrics unspecified)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.26095.

Fig 1: Illustration of motivation: a policy should first learn to move, and then learn to see and act. (a) In Stage 1, the

Fig 2: Architecture of our proposed framework. In Stage 1, interleaved state-action sequences and dataset-specific soft-prompt

Fig 3 (page 2).

Fig 4 (page 2).

Fig 5 (page 4).

Fig 6 (page 4).

Fig 7 (page 4).

Fig 3: Overview of the 13 cross-embodiment tasks across LIBERO, RoboCasa GR1, and real-world Franka manipulation. The

Limitations

• Training details such as batch size, epoch counts, optimizer hyperparameters, and hardware configurations are not fully detailed, potentially affecting reproducibility.
• No explicit adversarial robustness or safety evaluation of the policy under perturbations or environment changes is presented.
• While 13 cross-embodiment tasks are evaluated, the generalization to very different robot morphologies or drastically novel tasks is unclear.
• The approach requires substantial amounts of unconditioned action trajectory data for Stage 1 prior learning, which may be challenging to collect for some platforms.
• Real-world robot experiments are limited to only four pick-and-place tasks on Franka, restricting assessment of broader applicability.
• The latent alignment distillation weight decay schedule parameters and sensitivity are not extensively ablated or discussed.

Open questions / follow-ons

• How well does the learned action prior transfer to robots with drastically different kinematics or task distributions beyond the tested embodiments?
• Can the flow-matching encoder-decoder prior learning be extended or adapted to handle contact-rich or deformable object manipulations better?
• What are the best strategies to efficiently collect or augment unconditioned action trajectories for prior learning in real-world systems?
• How sensitive is the latent alignment distillation schedule and weighting to training hyperparameters and task complexity?

Why it matters for bot defense

While this work focuses on cross-embodiment robot manipulation, the general principle of pretraining dedicated action priors before multimodal model alignment is relevant to bot-defense and CAPTCHA system designers who integrate vision-language models with interactive outputs. Specifically, introducing explicit priors for the action or response generation module can reduce instability and improve convergence during joint training of perception and generation components. For CAPTCHA challenges involving complex user interactions or robotic simulators, pretrained action priors can help produce smoother, more human-like responses and speed up policy refinement. Additionally, the encoder-based history compression mechanism provides an efficient way to inject temporal context about past actions or user inputs, which might benefit history-aware bot detection or challenge adaptation strategies. However, the work does not directly address adversarial robustness or human/bot distinction, so practitioners should view this primarily as a general technique for stabilizing and improving multimodal policy training rather than a direct bot-defense method.

Cite

@article{arxiv2606_26095,
  title={ Learning Action Priors for Cross-embodiment Robot Manipulation },
  author={ Dong Jing and Tianqi Zhang and Jiaqi Liu and Jinman Zhao and Zelong Sun and Li Erran Li and Zhiwu Lu and Mingyu Ding },
  journal={arXiv preprint arXiv:2606.26095},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.26095}
}

Read the full paper

• arXiv:2606.26095 (abstract)
• arXiv:2606.26095 (PDF)