Latent Memory Palace: Reasoning for Control as Autoregressive Variational Inference

Source: arXiv:2607.08724 · Published 2026-07-09 · By Chuning Zhu, Eva Xu, Jose Barreiros, Krishnan Srinivasan, Paarth Shah, Abhishek Gupta

TL;DR

This paper tackles the challenge of enabling adaptive, iterative reasoning within continuous robotic control policies by introducing the Latent Memory Palace (LMP) framework. Unlike prior methods that either apply discrete language tokens or fixed-length latent variables, LMP organizes internal reasoning as an autoregressive latent variable sequence, analogous to traversing locations in a mnemonic memory palace. This novel latent space formulation integrates a variable-length autoregressive model with a variational inference objective, optimized via latent-space reinforcement learning. The key advantage is a policy that dynamically allocates compute at test time, generating longer latent reasoning chains for more complex or precise decisions.

Empirically, LMP yields strong results on multiple real and simulated robot manipulation benchmarks, outperforming state-of-the-art generative policies like Diffusion Policy. The same framework also produces a variable-length action tokenizer that markedly improves downstream autoregressive policies. Analyses show that iterative latent reasoning and parsimonious compute allocation enabled by compression schedules are critical contributors to performance. This work thus presents autoregressive variational inference in latent space as a principled, practical path to bringing adaptive, interpretable reasoning to robotic control policies.

Key findings

• LMP-π achieves zero-shot success rates of up to 0.65 on DROID block-bowl and 0.70 on marker-mug tasks, outperforming Diffusion Policy (DP) baselines (0.40 and 0.25 respectively).
• After finetuning on challenging peg-hole and clean-table tasks, LMP-π obtains 0.95 success rates, compared to DP’s 0.80 and 0.55, showing superior precise control and compositional generalization.
• On LIBERO-90 multitask benchmark, LMP-π achieves 0.933 ±0.003 average success across all tasks and 0.645 ±0.023 on bottom-10 tasks, outperforming DP’s 0.909 ±0.004 and 0.463 ±0.011, indicating reduced cross-task interference.
• LMP-tok, the variable-length action tokenizer derived from LMP, significantly outperforms action tokenizers like VQ-VAE, FAST, and OAT on RoboMimic and LIBERO, with superior precision on high-fidelity tasks (e.g., RoboMimic tool-hang).
• Iterative computation enabled by autoregressive latent sequences improves performance over non-iterative latent models such as VAE Policy and VQ-BeT on D3IL and RoboMimic benchmarks.
• Stronger compression via lower decoder variance (σ_min = 0.01) improves policy success rates on LIBERO-90 from 0.870 (σ_min=0.05) to 0.933, highlighting effective parsimonious allocation of latent computation improves generalization.
• Adaptive allocation of latent steps correlates negatively with action uncertainty: fewer latent steps are used when nearest-neighbor expert actions exhibit higher variance (Pearson r = -0.518).
• Qualitative analysis shows that LMP-π dynamically uses fewer reasoning steps during gripper movement segments, reflecting irreducible timing variability in the data.

Threat model

n/a — this is a control and imitation learning paper without explicit consideration of adversaries or security threats. The focus is on probabilistic latent variable formulations to support efficient, adaptive computation in robot policies.

Methodology — deep read

1. Threat Model & Assumptions: The adversary model is not explicitly security-focused; the task is imitation learning of robotic policies from expert trajectory datasets. The model assumes access to observations (often multi-modal, including language instructions) and corresponding expert actions. The learning agent aims to replicate expert behavior with adaptive reasoning, without adversarial interference.

2. Data: Evaluation datasets come from multiple real and simulated benchmarks: DROID (real-world manipulation, multitask with language conditioning, ~20 initial conditions per task), LIBERO (diverse multi-task simulation), D3IL (multimodal simulation), and RoboMimic (high-precision simulated tasks). Dataset sizes vary per benchmark; training/test splits are task-dependent. Action spaces include continuous controls; observations include stacked histories and language instructions. Token sequences are variable-length latents.

3. Architecture & Algorithm: The policy π_θ(o) is modeled via a latent variable formulation p(a|o) = E_{z}[p(a|o,z)p(z|o)], where z is an autoregressive latent sequence z_1:T(z). Both prior p_θ(z|o) and posterior q_θ(z|o,a) are parameterized by causal transformers with cross-attention to observations; the decoder p_ϕ(a|o,z) is a bidirectional transformer producing chunks of continuous actions. An EOS token marks variable-length latent chains, enabling adaptive computation.

The decoder's output is modeled as a Gaussian with variance decreasing exponentially with latent length, imposing a length/computation penalty encouraging efficient latent usage. The overall training objective maximizes an evidence lower bound (ELBO) balancing reconstruction likelihood and KL divergence between posterior and prior.

4. Training Regime: To cope with non-differentiable latent autoregressive sampling, the authors reformulate ELBO maximization as a reinforcement learning problem with rewards based on reconstruction log-likelihood and KL penalty. PPO-style trust region optimization with clipped surrogate objectives is applied, with latent trajectory rollouts collected in buffers. Episodes end on EOS token. Training uses a fixed maximum latent length H with padding for shorter sequences.

5. Evaluation Protocol: Metrics focus on task success rates across multiple robot manipulation benchmarks. Comparisons are made to strong baselines including Diffusion Policy, VQ-BeT, VAE Policy, and various action tokenizers (VQ-VAE, FAST, OAT). Ablations examine the effects of compression strength (variance schedules), iteration versus one-step latent reasoning, and downstream performance of LMP-tok tokenization. Test-time latent truncation ablations investigate the effect of limiting reasoning steps.

6. Reproducibility: The text mentions the code and models are not explicitly released in this truncated version. Datasets used are mostly established public benchmarks (DROID, LIBERO, D3IL, RoboMimic). Hyperparameters are detailed in appendices; random seeds are averaged over three trials for simulation results.

Concrete example: Given an observation from a manipulation task (e.g., picking an object), the posterior q_θ(z|o,a) generates a variable-length latent token sequence representing an internal reasoning chain. The prior p_θ(z|o) aims to approximate this during inference. The decoder p_ϕ(a|o,z) predicts precise actions from observation and latents, allocating longer latent sequences when higher accuracy or disambiguation is needed (e.g., precise gripper timing). Training optimizes the surrogate PPO objective alternating between sampling latent rollouts and updating model parameters.

Technical innovations

• Formulating iterative and adaptive reasoning in continuous control as variational inference with a variable-length autoregressive latent distribution, enabling dynamic test-time compute allocation.
• Introducing a novel latent-space reinforcement learning approach to optimize a variational lower bound over non-differentiable autoregressive latent sequences in robotic policy learning.
• Designing a decoder with a variance schedule that exponentially decreases variance with latent sequence length, incentivizing compression and parsimonious latent use.
• Extending the framework to a variable-length sequential action tokenizer (LMP-tok) that improves downstream autoregressive policy performance by better discretizing continuous actions compared to prior tokenizers.

Datasets

• DROID — real-world visuomotor manipulation — multitask language-conditioned datasets (20 evaluative initial conditions per task)
• LIBERO — simulated diverse manipulation tasks — public multi-task benchmark
• D3IL — simulated benchmark emphasizing multimodality — public
• RoboMimic — simulated high-precision manipulation benchmark — public

Baselines vs proposed

• Diffusion Policy on DROID block-bowl zero-shot: DP = 0.40 vs LMP-π = 0.65 success rate
• Diffusion Policy on DROID peg-hole finetuned: DP = 0.80 vs LMP-π = 0.95 success rate
• Diffusion Policy on LIBERO-90 bottom-10 tasks: DP = 0.463 ±0.011 vs LMP-π = 0.645 ±0.023 average success
• VQ-VAE tokenizer on RoboMimic/Robo 0.29/0.31 vs LMP-tok 0.69/0.75 downstream autoregressive policy success rate
• VAE Policy on D3IL: VAE = 0.56 success vs LMP-π = 0.76 success
• VQ-BeT on RoboMimic: VQ-BeT = 0.57 vs LMP-π = 0.83 success

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.08724.

Fig 1: (a) Latent Memory Palace (LMP) formulates iterative and adaptive reasoning as variational

Fig 2: Training LMP-π alternates between two stages. (1) Latent rollout: the posterior generates

Fig 3 (page 2).

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 2).

Fig 4: Visualization of evaluation domains.

Fig 5: Average success rates of au-

Limitations

• Training relies on sampling-based RL techniques in latent space, causing sensitivity to hyperparameters and risk of latent distribution collapse without extensive regularization or large rollout buffers.
• The autoregressive variational inference framework requires careful tuning of variance schedules to balance compression and reconstruction error; improper settings degrade performance.
• The method presently trains latents from scratch per step rather than maintaining persistent latent memory across an episode, potentially limiting temporal consistency or efficiency.
• Evaluation is primarily in imitation learning settings; robustness under adversarial or out-of-distribution conditions is not studied.
• The approach's scalability and stability for very long or highly complex action sequences remain to be explored.
• Code and pretrained models are not presently publicly released; reproducibility depends on replication of experimental details.

Open questions / follow-ons

• Can the discrete autoregressive latent distribution be replaced with a continuous latent chain (e.g., diffusion process) to improve sampling efficiency or expressiveness?
• How can latent persistence across episode timesteps be introduced to improve temporal consistency and reduce redundant computation during policy rollouts?
• Can the latent-space RL optimization framework be extended to reinforcement learning beyond imitation, including exploration and interaction with changing environments?
• What are the limits of adaptive computation trade-offs in larger, more complex robotic tasks and how does this affect real-time control constraints?

Why it matters for bot defense

While this paper focuses on robotic control rather than CAPTCHA solving or bot detection, the core idea of representing intermediate computation as adaptive, variable-length autoregressive latent sequences via variational inference has potential analogies for captcha and bot-defense systems. Specifically, the dynamic allocation of compute based on input uncertainty could inspire CAPTCHA schemes that vary challenge complexity adaptively or leverage latent reasoning to detect automation versus human decisions. Likewise, the latent tokenizer approach suggests new ways to discretize continuous interaction signals for sequential modeling in bot detection. However, deploying this framework in CAPTCHA would require careful domain-specific adaptation and adversarial robustness evaluation. Bot-defense engineers might draw inspiration from the latent reasoning and compression mechanisms here to build more interpretable, efficient multi-step challenge-response pipelines but would need additional work on modeling malicious actors and real-world attack scenarios.

Cite

@article{arxiv2607_08724,
  title={ Latent Memory Palace: Reasoning for Control as Autoregressive Variational Inference },
  author={ Chuning Zhu and Eva Xu and Jose Barreiros and Krishnan Srinivasan and Paarth Shah and Abhishek Gupta },
  journal={arXiv preprint arXiv:2607.08724},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.08724}
}

Read the full paper

• arXiv:2607.08724 (abstract)
• arXiv:2607.08724 (PDF)