RECALL: Recovery Experience Collection for Active Lifelong Learning in Vision-Language-Action Models

Source: arXiv:2606.23617 · Published 2026-06-22 · By Ulas Berk Karli, Tesca Fitzgerald

TL;DR

This paper addresses the inefficiencies of passive imitation learning for fine-tuning Vision-Language-Action (VLA) models, where additional demonstrations are collected only after policy failures, often wasting effort on well-performing states. The authors propose an active, continual learning framework for VLAs that leverages INSIGHT, an uncertainty quantification method, to identify high-uncertainty states during execution and collect targeted recovery demonstrations from those points. They empirically show that active, uncertainty-guided data collection improves fine-tuning efficiency over passive, start-state-based data collection, especially when collecting a single recovery demonstration online from the first uncertain state per rollout.

However, fine-tuning only on this uncertainty-guided recovery data leads to catastrophic forgetting of previously learned skills due to the narrow data distribution. The authors investigate continual learning strategies including replay-based data mixing and elastic weight consolidation (EWC) regularization to mitigate forgetting. They find replay approaches, especially when including a broad subset of prior data, offer the best tradeoff between retaining old behaviors and adapting to new recovery data. EWC and low learning rates reduce forgetting but limit adaptation. The paper provides an empirical design study of active continual learning for large autoregressive VLAs and highlights open challenges in balancing plasticity and stability.

Key findings

• Online recovery data collected from the first high-uncertainty state per rollout improves overall policy success on LIBERO-10 to 72.4%, compared to 60.2% for matched passive collection (p=0.0001).
• Offline recovery collection from all uncertain states is not significantly better (68.4% success) than online first-uncertainty-state collection (72.4% success) despite more demonstrations (p=0.1659).
• Fine-tuning using only the newly collected recovery data causes catastrophic forgetting, dropping overall success to 28.4% and retained-task success to 56.4% vs. 72.4% and 95.6% when replaying prior data.
• Low constant learning rate (2.5×10^-8) fine-tuning preserves retained-task success at 91.2% but achieves limited collected-task success (34.4%) and overall success of 62.8%, showing a stability-plasticity tradeoff.
• Elastic weight consolidation (EWC) with λ=10^12 achieves 61.4% overall success, 32.0% collected-task success, and 90.8% retained-task success, but still underperforms full replay training.
• Full replay of all prior data combined with recovery data achieves highest overall success (72.4%) and collected-task success (49.2%), best balancing adaptation and retention.
• Replay restricted only to targeted low-accuracy tasks reduces retained-task success significantly from 90.0% to 83.6%, indicating replay coverage is critical.
• EWC regularization cannot reliably prevent forgetting when replay data coverage is narrow, highlighting limitations of regularization alone.

Threat model

The implicit adversary is the environment and deployment conditions causing the VLA policy to encounter novel states and distribution shifts inducing failure. The system assumes access to expert demonstrations upon uncertainty triggers but does not consider strategic manipulation of the learning process or adversarial attacks. Failures arise from lack of supervision at critical states, which the active learning aims to mitigate.

Methodology — deep read

1. Threat Model & Assumptions: The adversary here is an implicit concept: the model faces distribution shifts and failures when deployed to new robots, environments, or tasks within the LIBERO framework. The threat is the lack of supervision on uncertain or failure-prone states leading to poor policy performance. The model can query for help (demonstrations) at uncertain states identified by INSIGHT during rollouts; the adversary cannot manipulate this querying or demonstrations.

2. Data: The primary data comes from the LIBERO-10 benchmark dataset and corresponding simulated environment for robotic manipulation tasks. The baseline model π0-FAST is trained for 30k steps on LIBERO-10 tasks to produce πθ0. Expert recovery demonstrations are collected from states flagged as uncertain by INSIGHT — either offline (all uncertain states per rollout) or online (first uncertain state per rollout). Passive data collection gathers demonstrations from start states to match task distribution and data volume.

3. Architecture / Algorithm: The base model is π0-FAST, an autoregressive Vision-Language-Action model predicting tokenized robot action chunks conditioned on visual input, robot state, and language instruction. INSIGHT is an inference-time introspection method extracting token-level uncertainty features (entropy, log-probability, Dirichlet-based aleatoric and epistemic uncertainties) to detect when the policy requires help (binary label H_t). Recovery data collection uses the help predictions to reset the simulator to the uncertain state and collect expert demonstrations.

Fine-tuning computes cross-entropy loss on token predictions and optionally applies continual learning regularization. Replay combines prior datasets with new recovery data. Elastic Weight Consolidation (EWC) adds a Fisher-information-based quadratic penalty to preserve parameters important to prior tasks.

4. Training Regime: Fine-tuning uses standard π0-FAST hyperparameters with a linear warmup learning rate of 2.5×10^-5 decaying to 2.5×10^-6. For continual learning experiments, constant low learning rates (2.5×10^-8) and EWC with various penalties (λ up to 10^12) are evaluated. Training batches, epochs, and hardware details are not explicitly specified.

5. Evaluation Protocol: Policy success is measured as task completion over 50 rollouts per task (10 tasks total) = 500 rollouts per checkpoint. Metrics are overall success, collected-task success on 5 tasks targeted by recovery data, and retained-task success on the remaining 5 tasks. Statistical significance is assessed with two-sided two-proportion z-tests (p<0.05 considered significant). Comparisons include matched passive data, online vs offline active data, new-only fine-tuning vs replay mixtures, and regularization ablations. Normalization statistics for input features are also ablated.

6. Reproducibility: The paper refers to publicly available datasets (LIBERO-10) and published prior methods (π0-FAST, INSIGHT). The released INSIGHT model and π0-FAST weights are presumably available. Specific training code or checkpoints for this study are not explicitly mentioned as released. The dataset for active recovery demonstrations is generated via simulator resets in LIBERO-10.

Concrete example end-to-end: Starting from the π0-FAST baseline trained on LIBERO-10, a rollout is performed in simulation. At each timestep, INSIGHT evaluates uncertainty to predict if the policy needs help. The first timestep where INSIGHT outputs H_t=1 triggers an online active data collection: the simulator resets to that state and collects an expert recovery demonstration to completion. This recovery data is added to prior data and fine-tuned with replay mixture and cross-entropy loss. The fine-tuned policy shows statistically significant improvements over matched passive data in overall and collected-task success metrics.

Technical innovations

• Use of INSIGHT uncertainty estimator at inference time to actively identify high-value recovery states for targeted demonstration collection in VLAs, moving beyond passive data collection.
• Demonstration that collecting only one recovery demonstration online at the first uncertain state per rollout matches or exceeds dense offline recovery from all uncertain states in efficiency.
• Empirical evidence that fine-tuning solely on uncertainty-guided recovery data causes catastrophic forgetting in VLAs, necessitating continual learning methods.
• Evaluation of continuous learning techniques (data replay mixing and Elastic Weight Consolidation) adapted to large autoregressive VLAs, highlighting a stability-plasticity tradeoff.
• Identification that replay coverage breadth critically affects retention of previous tasks, with targeted replay insufficient and full or broad replay necessary.

Datasets

• LIBERO-10 — simulation-based robotic manipulation benchmark — public
• Recovery demonstration datasets (D_online, D_offline, D_passive) — collected via simulator resets at uncertain states — internal/generated

Baselines vs proposed

• Passive start-state data collection: overall success = 60.2% vs Online active recovery collection: 72.4%
• Offline active recovery collection (all uncertain states): overall success = 68.4% vs Online (first uncertain state): 72.4% (p=0.1659, not significant)
• Fine-tune new-only (Strong INSIGHT online recovery data): overall success = 28.4% vs full replay: 72.4%, catastrophic forgetting evident
• New-only low constant learning rate: overall success = 62.8%, retained-task success = 91.2% vs standard new-only LR: severe forgetting
• EWC with λ=10^12: overall success = 61.4%, collected-task success = 32.0%, retained-task success = 90.8% vs full replay: 72.4% overall, 49.2% collected-task
• Replay with LIBERO-10 subset: overall success = 68.6% close to full replay 72.4% (p=0.1877), targeted replay: 63.2% with reduced retained-task success (p=0.0345)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.23617.

Fig 1: Overview of uncertainty-guided active continual learning. An initial π0-FAST policy

Fig 2: Online recovery collection improves over passive start-state collection. We compare

Fig 3: compares online and offline recovery collection. Online collection reaches

Fig 4 (page 3).

Fig 5 (page 3).

Fig 6 (page 3).

Fig 7 (page 3).

Fig 8 (page 3).

Limitations

• Experiments only in simulation (LIBERO-10) abstract away real-world challenges such as sensor noise, safety constraints, imperfect resets, and human demonstrator costs.
• Active recovery requires simulator resets at uncertain states, which may not be practical on real robots where interventions need runtime human involvement or natural failure states.
• The quality of INSIGHT's uncertainty estimates is a bottleneck; false positives waste demonstrator effort and false negatives miss critical states.
• Study limited to one autoregressive VLA architecture and one benchmark; results may not generalize to diffusion-based or hybrid action models.
• Continual learning methods evaluated are relatively simple (replay, EWC, learning rate schedules); more sophisticated approaches (LoRA, adapters, selective freezing) could improve results.
• Evaluation focuses on task success metrics; ignores costs like intervention frequency, demonstration counts, safety violations, uncertainty calibration, and repeated active learning cycles.

Open questions / follow-ons

• How can better-calibrated, risk-sensitive uncertainty estimators improve the efficiency and reliability of active recovery data collection for VLAs?
• What continual learning methods beyond replay and EWC (e.g., parameter-efficient finetuning, adaptive regularization, data reweighting) can better balance stability and plasticity in large VLAs?
• How can active continual learning and uncertainty-guided recovery be integrated and cost-optimized in real-world robotic deployments with noisy sensors, safety constraints, and limited human availability?
• Can active learning paradigms generalize to diffusion or hybrid action representation VLAs that require different uncertainty signals and selection mechanisms?

Why it matters for bot defense

Bot-defense practitioners and CAPTCHA engineers can draw parallels from this study regarding targeted, uncertainty-driven data collection to improve system robustness. The core insight that judicious selection of challenging states for human (expert) intervention improves model adaptation efficiency may aid in designing CAPTCHA systems that require minimal user interaction but maximize security gains.

Additionally, the research highlights the critical importance of continual learning techniques when updating large, complex models with targeted data to avoid forgetting previously acquired capabilities. For CAPTCHA systems that incorporate evolving behavioral or interaction models, maintaining a balance between plasticity to new, challenging attack modalities and retention of mature defenses is vital. Replay-like approaches or regularization methods may inform such system update strategies.

Cite

@article{arxiv2606_23617,
  title={ RECALL: Recovery Experience Collection for Active Lifelong Learning in Vision-Language-Action Models },
  author={ Ulas Berk Karli and Tesca Fitzgerald },
  journal={arXiv preprint arXiv:2606.23617},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.23617}
}

Read the full paper

• arXiv:2606.23617 (abstract)
• arXiv:2606.23617 (PDF)