FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation

Source: arXiv:2606.26006 · Published 2026-06-24 · By Shuyi Zhang, Yunfan Lou, Hongyang Cheng, Yichen Guo, Chuyao Fu, Yaoxu Lyu et al.

TL;DR

This paper addresses the core challenge of sample inefficiency and instability in reinforcement learning (RL) fine-tuning for Vision-Language-Action (VLA) robotic models. Traditional imitation learning from suboptimal data imposes an imitation ceiling on performance, and naive RL fine-tuning suffers from catastrophic initial unlearning due to unstable Q-value estimates and inefficient policy updates from low-quality exploratory actions. To overcome these issues, the authors propose FORCE, a novel three-stage framework that first calibrates the Q-function support with a value-calibrated warm-up phase incorporating on-policy rollouts, then performs online fine-tuning with a value-guided policy self-distillation (VGPD) mechanism that filters updates to high-value actions and eliminates exploration noise without human intervention.

Empirically, FORCE achieves state-of-the-art results on a suite of challenging simulated robotic manipulation tasks (ManiSkill) and diverse real-world contact-rich tasks on a physical Franka Emika Panda robot. The method attains a 79% absolute improvement in success rate over behavioral cloning, outperforms prior RL fine-tuning baselines by over 10% average success, and accelerates training by over 30%. Notably, FORCE eliminates the common catastrophic performance drop seen during offline-to-online RL transfer, achieves near-perfect success in precision tasks without costly human-in-the-loop interventions, and significantly improves execution efficiency as measured by reduced action steps in real settings. The framework effectively addresses the distributional shift between offline data and online visitation, enabling robust, efficient, and autonomous reinforcement fine-tuning of VLA policies in both simulation and real-world.

Key findings

• FORCE achieves an average success rate of 82.3% across six ManiSkill tasks using the Octo backbone, surpassing the strongest prior RL baseline ConRFT by over 10%.
• FORCE reduces the sample complexity by 32.5% (Steps@80% success), accelerating convergence significantly across tasks like PushCube (4k steps vs 6k+ for alternatives).
• Real-world evaluation on a Franka robot shows FORCE raises average success rate from 45.0% (Behavioral Cloning baseline) to 98.3%, achieving 100% success on 4 of 6 tasks.
• Execution efficiency improved drastically in real tasks: mean execution steps dropped from 112.8 (BC) to 38.9, e.g., Stack Cube steps reduced from 170.4 to 46.2.
• Ablation studies show removing the Value-Calibrated Warm-up stage induces initial catastrophic performance collapse, confirming its importance in stabilizing offline-to-online transition.
• Removing the VGPD module results in premature convergence plateaus and failure to reach target success rates in some tasks, indicating its critical role in stable policy refinement.
• VGPD automatically transitions from cloning expert actions early to exploiting high-value on-policy actions later, implementing an adaptive, curriculum-like learning schedule.
• FORCE achieves stable learning curves with monotonic improvements and substantially mitigates the common success rate drop observed in prior RL fine-tuning frameworks.

Threat model

The threat model involves an RL fine-tuning scenario where the policy must safely and efficiently transition from offline demonstration data to online deployment without catastrophic performance degradation. The adversary can be thought of as the challenges imposed by distributional shifts and exploration noise in high-dimensional vision-language-action spaces, which can cause the policy to unlearn or degrade. The method assumes no malicious adversaries manipulating the environment or observations but faces natural uncertainties and error compounding. It does not consider direct attacks or adversarial manipulations.

Methodology — deep read

1. Threat Model & Assumptions: The adversary model is implicit in the RL fine-tuning context: the policy seeks to improve from offline data to online deployment in complex robotic environments without human interventions. Challenges arise from distributional shift (support mismatch) and exploration noise, but the setting does not consider explicit adversarial manipulation. It assumes access to offline expert and suboptimal datasets and that online interaction data is collected safely.

2. Data: The offline datasets consist of expert demonstrations and suboptimal trajectories collected from physical and simulated robotic manipulation environments. ManiSkill benchmark provides six manipulation tasks with corresponding labeled offline data. For real-world experiments, data is collected from a Franka Emika Panda with synchronized visual inputs (dual RealSense cameras). Data splits include offline expert buffer, mixed warm-up data, and online policy-generated buffers.

3. Architecture / Algorithm: FORCE comprises three stages: (1) Offline reinforcement fine-tuning using Calibrated Q-Learning (Cal-QL), a TD-error minimization with a calibration regularizer preventing overestimation for OOD actions, coupled with a behavior cloning (BC) policy regularization and a consistency policy actor network. (2) Distributional Warm-up stage collects a small batch of on-policy rollouts to expand the Q-function’s valid support, mixing offline and online data to mitigate distributional shift before policy updates. The policy training objective balances BC loss and Q-value guided gradient ascent. (3) Online intervention-free fine-tuning maintains expert and policy replay buffers. It uses the Value-Guided Policy Self-Distillation (VGPD) method, which approximates a regularized policy improvement by projecting an energy-based optimal policy with advantages above the mean value. VGPD filters policy updates by masking out low-advantage actions and performs a KL-constrained weighted maximum likelihood update of the actor.

The VGPD module dynamically selects distillation targets by calculating a state-dependent baseline Qmean(s) from K sampled candidate actions per state. Actions with Q-values exceeding the baseline form a weighted energy-based target distribution for self-distillation, promoting monotonic policy improvement and reducing gradient variance.

4. Training Regime: Initial offline RL is performed with Cal-QL on expert demonstrations to get a conservative critic and stable actor. The warm-up phase collects on-policy trajectories to revise Q-function calibration. Finally, online fine-tuning uses balanced sampling from expert and policy buffers with VGPD. Hyperparameters include calibration regularization weight alpha, tradeoff parameters eta (BC weight) and lambda (Q weight), and temperature tau for exponentiation in VGPD. K (candidate samples) is chosen as a hyperparameter (exact K unspecified). Experiments run on multiple seeds with both simulated and real robotic setups. The consistency policy reduces denoising steps compared to diffusion policies.

5. Evaluation Protocol: Metrics include task success rate, average execution steps, and sample efficiency (Steps@80% success). Comparisons are against behavior cloning baselines and several prior RL methods (Cal-QL, CQL, PA-RL, ConRFT without human in loop). Evaluation tasks include six ManiSkill benchmark manipulation tasks and six contact-rich real robot tasks (Pick Cup, Open Drawer, etc.) with multi-view visual input. Ablations test the removal of warm-up and VGPD modules to isolate contributions. Learning curves across seeds and tasks measure convergence speed and stability. Statistical significance testing is not explicitly mentioned.

6. Reproducibility: Code release is not explicitly stated in the excerpt; offline dataset (ManiSkill) is public, while real robot dataset is private. Hyperparameters and equations are detailed. The paper uses standard RL and behavior cloning components adapted with novel calibration and distillation. Details on hardware seeds or exact training compute are not given. The VGPD sampling incurs computational overhead, noted as a limitation.

Example end-to-end flow: Starting from a pre-trained VLA policy, Cal-QL is applied offline over demonstration data for initial conservative Q-function learning; then a small batch of online rollouts is gathered in warm-up to update and calibrate the Q-function on the policy visitation distribution; finally, online fine-tuning proceeds with VGPD selectively distilling only actions with positive value advantage from both expert and policy buffers, resulting in stable iterative policy improvements and accelerated task mastery without human interventions.

Technical innovations

• Introduction of a Value-Calibrated Warm-Up stage using on-policy rollouts to expand the Q-function's valid support and mitigate offline-to-online distributional shift.
• Development of Value-Guided Policy Self-Distillation (VGPD), a novel regularized policy improvement operator that dynamically filters policy updates to only high-value, positive advantage actions to improve stability and sample efficiency.
• Hybrid offline-to-online three-stage RL fine-tuning framework combining calibrated Q-learning, distributional warm-up, and value-filtered self-distillation without requiring human-in-the-loop interventions.
• Use of a consistency policy actor network to distill multi-step diffusion/flow policies into a one-step policy, enabling faster inference and more stable Q-gradient propagation.

Datasets

• ManiSkill Benchmark — thousands of manipulation trajectories across six tasks — public
• Real-world Franka Emika Panda robot interaction data — hundreds of contact-rich task episodes — private

Baselines vs proposed

• Behavior Cloning (BC): average success rate = 45.0% vs FORCE = 82.3%+ on ManiSkill tasks
• ConRFT (no Human-in-Loop): average success rate = 71.1% vs FORCE (Octo backbone) = 82.3%
• Cal-QL: success rate StackCube = 45.5%, PlaceSphere = 0% vs FORCE StackCube = 94.2%, PlaceSphere = 85.1%
• Steps@80% PickCube: w/o VGPD = 20k steps vs FORCE = 12k steps
• Real-world task success rate Pick Cup: BC = 35% vs FORCE = 90%, execution steps reduced from 134.5 to 36.4

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.26006.

Fig 1: Overview of the FORCE framework. Our method employs a three-stage reinforcement fine-tuning pipeline that progressively

Fig 2: VGPD module in the online phase. VGPD serves as a

Fig 3: Real-world Experiment Tasks. We conducted real-

Fig 4: Learning Curves. We present the performance of our proposed FORCE and baselines in ManiSkill tasks. The evaluation

Fig 5: Ablation Study. Training curves across four tasks comparing full FORCE against variants without warm-up or VGPD. The

Fig 6 (page 5).

Fig 7 (page 5).

Fig 8 (page 5).

Limitations

• FORCE requires computationally expensive sampling of multiple candidate actions per state (K samples) during VGPD, increasing inference cost especially for large VLA backbones.
• The framework currently uses task-specific critics rather than generalized reward models, limiting scalability to broader open-ended or long-horizon tasks.
• Real-world evaluation is limited to a single robotic platform (Franka Panda) and a small set of six contact-rich tasks, constraining generality.
• No explicit adversarial or safety robustness evaluation under unexpected perturbations or distribution shifts beyond initial covariate shift.
• Reproducibility depends on access to potentially proprietary offline datasets and no mention of public code or checkpoint releases.

Open questions / follow-ons

• How can the computational overhead of the VGPD sampling procedure be reduced or approximated, e.g., via action caching or reuse of past samples?
• Can the FORCE framework be extended to employ generalized reward models to enable open-ended, multi-task, or long-horizon fine-tuning without retraining critics per task?
• How robust is FORCE under adversarial environmental perturbations, sensor noise, or distribution shifts beyond the initial transition phase?
• What is the impact of varying the candidate sample size K and temperature tau hyperparameters on VGPD’s stability versus computational cost tradeoff?

Why it matters for bot defense

FORCE primarily targets reinforcement learning fine-tuning of vision-language-action policies for robotic manipulation, which involves continuous control and complex multimodal inputs. For bot-defense and CAPTCHA practitioners, the core insights around mitigating distributional shift and leveraging value-based filtering for stable policy updates may inspire related approaches for reducing noisy or adversarial behavior in sequential decision systems. In particular, the concept of calibrating value estimators with on-policy data before policy updates could help online defense systems avoid catastrophic degradation when adapting to novel or adversarial inputs.

Furthermore, the value-guided self-distillation technique highlights a principled way to filter and weight behavior updates dynamically based on estimated returns, potentially informing adaptive filtering of suspicious or low-value actions in CAPTCHA or bot traffic models. However, given the robotics-specific target domain and reliance on complex offline-to-online reinforcement learning pipelines, direct application would require significant adaptation. Nonetheless, the emphasis on stable, sample-efficient fine-tuning without human heuristics or interventions is highly relevant for automated bot-defense systems needing online adaptation.

Cite

@article{arxiv2606_26006,
  title={ FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation },
  author={ Shuyi Zhang and Yunfan Lou and Hongyang Cheng and Yichen Guo and Chuyao Fu and Yaoxu Lyu and Xiaojie Zhang and Haoran Li and Pengwei Wang and Zhongyuan Wang and Shanghang Zhang },
  journal={arXiv preprint arXiv:2606.26006},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.26006}
}

Read the full paper

• arXiv:2606.26006 (abstract)
• arXiv:2606.26006 (PDF)