RoboTTT: Context Scaling for Robot Policies

Source: arXiv:2607.15275 · Published 2026-07-16 · By Yunfan Jiang, Yevgen Chebotar, Ruijie Zheng, Fengyuan Hu, Yunhao Ge, Jimmy Wu et al.

TL;DR

RoboTTT addresses the critical limitation of existing robot foundation models that operate on single-step or short visuomotor context by introducing a method that scales the context length up to 8,000 timesteps—three orders of magnitude longer—without increasing inference latency. This substantial increase in context length enables new capabilities such as one-shot in-context imitation from human video demonstrations, on-the-fly policy improvement via meta-learning, increased robustness to perturbations, and improved performance on long-horizon, multi-stage robotic assembly tasks. The key innovation behind RoboTTT is the integration of Test-Time Training (TTT), where a set of fast weights updated by gradient descent during inference compress and store history information, allowing continuous adaptation to long trajectories without ever growing computational cost at inference per step.

Key findings

• RoboTTT improves overall task completion score by 87% over the single-step context baseline GR00T N1.7 on three real-robot long-horizon assembly tasks.
• RoboTTT fully completes a previously unsolved 5-minute, 10-stage Gear Bot assembly task, achieving 2/10 full successes while no baselines do.
• Increasing pretraining context length from 1,000 to 8,000 timesteps raises closed-loop task completion score by 63% (from 43.9% to 71.5%).
• One-shot imitation from a single in-context human video demonstration succeeds in 6/10 trials with RoboTTT versus 0/10 with a gated recurrent baseline (GDN).
• Under external perturbations, RoboTTT recovers removed parts successfully in 75%-90% of trials, outperforming baselines by over 30% in some settings.
• DAgger Distillation training yields a 36% improvement in task performance via on-the-fly failure recovery, more than 2x the gain of standard DAgger fine-tuning on human corrections alone.
• Sequence action forcing during training is critical, with removal causing substantial performance degradation.
• Replacing RoboTTT’s nonlinear MLP fast model with a linear layer reduces performance by 27%, showing expressivity of fast weights matters.

Threat model

n/a — This paper addresses robot policy learning and adaptation rather than security threats. The 'adversary' is not specified; the focus is on improving closed-loop task performance, context scaling, and adaptation under natural perturbations rather than adversarial attacks.

Methodology — deep read

1. Threat model & assumptions: RoboTTT assumes a standard robot policy learning setting targeting long-horizon visuomotor control; the adversary model or security assumptions are not explicitly discussed as this is robotics policy research.

2. Data: The training data consists of multi-task, multi-stage robot manipulation trajectories and paired human video demonstrations. Dataset sizes vary per task: 8 hours for Pup Go Car (2-min episodes), 6 hours for Circuit (1-min episodes, 80 assembly configs), and 5 hours for Gear Bot (5-min, 10 stages). Language instructions are included as tokens shared across trajectories.

3. Architecture/algorithm: RoboTTT builds on the pretrained GR00T N1.7 Vision-Language-Action (VLA) model, augmenting it with Test-Time Training (TTT) layers inserted after self- and cross-attention in each of 16 diffusion transformer (DiT) layers. The policy predicts length-H action chunks at each timestep. The core novelty is that fast weights parameterize small, two-layer MLPs which update gradients at each timestep, encoding and recalling long contexts compressed into weight space instead of explicit memory. Learned register tokens attend across time to propagate VL information efficiently. A gating mechanism controls the relative contribution of TTT outputs to preserve pretraining capabilities.

4. Training regime: Pretraining for 30k steps on heterogeneous data mixtures is performed on 16 A100 GPUs, gradually scaling context length to target (up to 8K timesteps). Post-training on downstream tasks is 20k steps at 1K context. Training uses truncated backpropagation through time (TBPTT) with sequence action forcing (applying varying noise levels independently per action chunk) to stabilize training of long sequences. Task losses combine flow-matching diffusion losses with fast weight inner-loop updates.

5. Evaluation protocol: Policies are evaluated on three long-horizon assembly tasks with 20 trials each (10 for Gear Bot due to length). Metrics include normalized rubric-based task completion scores and counts of fully successful trials. Ablations vary training components (e.g. removing sequence action forcing, linear vs MLP fast models). Comparisons include baseline models: GR00T N1.7 (single-step context), GR00T N1.7 with one history frame, and GDN (a gated recurrent baseline without test-time gradient updates). Additional experiments evaluate perturbation recovery, one-shot imitation from in-context videos, and DAgger Distillation for on-the-fly improvement.

6. Reproducibility: The paper states training details and hyperparameters, with code/data not explicitly released at time of writing. The underlying GR00T N1.7 model is pretrained and publicly described but not stated as fully open-source. Dataset includes proprietary real robot and egocentric human data.

Concrete end-to-end example: For the Circuit task, RoboTTT conditions on a single human demonstration video by concatenating human video tokens and robot rollout tokens into one sequence. The fast weights update on the human video portion (without imitation loss), then generate robot actions conditioned on this updated fast state. At test time, this allows one-shot imitation of unseen configurations, successfully completing assemblies that baseline recurrent policies fail.

Technical innovations

• Integration of Test-Time Training (fast weights updated by gradient descent during inference) inside robot foundation models to compress long visuomotor contexts into parameter space.
• Scaling robot policy context length to 8,000 timesteps without increased inference latency, three orders of magnitude beyond prior art.
• Sequence action forcing: sampling noise levels independently for each action chunk during training to stabilize flow-matching objectives over long sequences.
• DAgger Distillation: meta-learning user corrections by updating fast weights on full rollout history but computing imitation loss only on corrective actions, enabling better on-the-fly recovery.

Datasets

• Pup Go Car — 8 hours of bimanual real-robot data — proprietary real-robot setup
• Circuit — 6 hours of bimanual real-robot data with 80 assembly configurations and paired human video demonstrations — proprietary
• Gear Bot — 5 hours of real-robot bimanual data, 5-minute episodes, 10-stage assembly — proprietary

Baselines vs proposed

• GR00T N1.7 single-step context: task completion score = 42% vs RoboTTT 79% (87% improvement)
• GR00T N1.7 Hist. (one-step history): task completion score = 39.5%-45.6% depending on task vs RoboTTT 71.5%-79%
• GDN (gate delta recurrent memory without test-time gradients): task completion score = 56% vs RoboTTT 79%
• One-shot imitation on Circuit: GDN = 0 successful trials / 10 vs RoboTTT = 6 /10
• Perturbation recovery on Pup Go Car roof removal: GR00T N1.7 = 10/20, GR00T Hist = 3/20, GDN = 13/20, RoboTTT = 15/20
• DAgger Distillation performance improvement: 36% boost for RoboTTT vs 13% from standard DAgger fine-tuning

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.15275.

Fig 1: RoboTTT, a long-context visuomotor policy that integrates Test-Time Training (TTT) into robot foundation

Fig 2 (page 1).

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• Proprietary real-robot datasets and pretrained models limit immediate reproducibility and benchmarking by third parties.
• No adversarial robustness evaluation or analysis of failure modes under malicious manipulations.
• Experiments are restricted to specific bimanual assembly tasks in controlled environments; generalization to diverse robot types and tasks remains untested.
• DAgger Distillation and long context training are computationally intensive, requiring large GPU clusters, which may limit practical deployment.
• The longest context trained is 8,000 timesteps (~4 minutes); behavior and scaling trends beyond this horizon remain unknown.
• Gating mechanisms to retain pretrained knowledge introduce complexity that may complicate transfer to other backbones.

Open questions / follow-ons

• How does the fast weight adaptation mechanism scale beyond 8,000 timesteps and longer task horizons?
• Can RoboTTT’s approach generalize efficiently to non-assembly robotic domains or robots with different morphology?
• What are the failure modes and robustness properties under adversarial or unexpected environmental inputs?
• How can the computational efficiency and memory requirements be further improved for edge deployment?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners interested in analyzing or defending against advanced ML policies with long-term memory, RoboTTT offers a rare example of scaling context length drastically without latency penalty via fast weight adaptation. While the domain is robotics, the core idea of compressing long histories into fast, adaptive model parameters rather than retaining explicit token buffers may inspire novel approaches to long-term behavioral pattern detection or adaptive bot fingerprinting in CAPTCHA systems. Additionally, the demonstrated capabilities like on-the-fly policy improvement and robust recovery from perturbations highlight the challenges posed by models that adapt dynamically in deployment, underscoring the need for defense techniques that can handle adaptive adversaries with long-term contextual memory. However, direct application would require adaptation to the specific visual-linguistic or interaction modalities relevant to bot behavior, as well as security-aware threat modeling absent in the paper.

Cite

@article{arxiv2607_15275,
  title={ RoboTTT: Context Scaling for Robot Policies },
  author={ Yunfan Jiang and Yevgen Chebotar and Ruijie Zheng and Fengyuan Hu and Yunhao Ge and Jimmy Wu and Tianyuan Dai and Scott Reed and Li Fei-Fei and Yuke Zhu and Linxi "Jim" Fan },
  journal={arXiv preprint arXiv:2607.15275},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.15275}
}

Read the full paper

• arXiv:2607.15275 (abstract)
• arXiv:2607.15275 (PDF)