Autoregressive Boltzmann Generators

Source: arXiv:2606.27361 · Published 2026-06-25 · By Danyal Rehman, Charlie B. Tan, Yoshua Bengio, Avishek Joey Bose, Alexander Tong

TL;DR

This paper addresses the fundamental challenge of efficiently sampling molecular conformations at thermodynamic equilibrium, which is key for understanding macroscopic phenomena like protein folding. Traditional approaches such as Molecular Dynamics (MD) simulation suffer from extreme computational costs due to slow mixing over high-energy barriers. Boltzmann Generators (BGs) offer a promising generative modeling framework for drawing independent Boltzmann-weighted samples, but state-of-the-art methods predominantly rely on normalizing flows (NFs) that face theoretical and practical limitations: discrete-time flows suffer from restricted expressivity due to invertibility and topological constraints, while continuous-time flows have expensive inference costs and numerical instability.

The authors propose Autoregressive Boltzmann Generators (ArBG), a novel diffeomorphism-free autoregressive framework that factorizes the molecular density into conditional distributions over atom coordinates. This circumvents the topological constraints of flows, enables fast exact likelihood evaluation without expensive numerical integration, and benefits from innovations in large-scale autoregressive models inspired by large language models. Empirically, ArBG consistently outperforms flow-based BGs across peptide benchmarks, especially on larger systems like the 10-residue Chignolin. They also present ROBIN, a 132M parameter transferable autoregressive model that achieves strong zero-shot generalization and reduces energy error metrics by over 60% relative to prior state-of-the-art methods such as Prose.

Key findings

• ArBG outperforms flow-based BG baselines (e.g., SBG, ECNF++, RegFlow) on energy Wasserstein distance (E-W2) and torsional Wasserstein distance (T-W2) across peptides from 3 to 10 residues, with improvements especially pronounced on larger peptides like Chignolin (Table 1).
• The transferable AR model ROBIN reduces zero-shot E-W2 energy error on 8-residue peptides by over 60% compared to Prose (previous SOTA) (Table 2).
• ROBIN achieves similar or better performance than Prose and MD simulations with an order of magnitude fewer energy function evaluations on octapeptides (Figure 1, Figure A.16).
• Uniform binning parameterization for conditionals yields improved training stability and scalability compared to mixture density network approaches like MoL-PixelCNN++ and GMM-PixelCNN++.
• Twisted Sequential Monte Carlo (SMC) inference with intermediate resampling provides modest gains over self-normalized importance sampling, confirming ARBG's ability for inference-time intervention.
• Increasing the number of uniform bins in ArBG monotonically improves discrete conditional likelihoods and downstream sampling performance (Figure 5).
• Ramachandran plots and TICA visualizations demonstrate that ARBG closely matches the structural distribution and slow dynamical modes of reference MD data (Figures 3 and 4).

Threat model

The paper does not address adversarial threats in a classical security sense. Rather, the 'adversary' is the complexity of the molecular energy landscape and the need to efficiently generate unbiased samples from a highly multimodal, discontinuous distribution under computational constraints. The models cannot directly access or enumerate all high-dimensional states but rely on learned generative proposals and importance sampling corrections.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is implicit in the probabilistic sampling task—models generate samples from a proposal distribution p_θ(x) that approximates the target Boltzmann distribution μ_target(x) ∝ exp(-E(x)) for molecular conformations x ∈ ℝ^{n×3}. The goal is accurate, unbiased sampling of equilibrium conformations without requiring long MD trajectories. The adversary cannot violate model assumptions on independence or physically plausible molecular states beyond energetic constraints.

2. Data: Evaluations use several peptide systems of increasing size: alanine dipeptide (2 residues), tri-alanine (ALA3), tetrapeptide (ALA4), hexa-alanine (ALA6), and the decapeptide Chignolin (10 residues). For transfer learning, the ManyPeptidesMD dataset with 30 sequences each of 4 and 8 residues is used. Training data and splits are consistent with prior works (Tan et al. 2025a,b). Input conformations are represented by Cartesian coordinates flattened into vectors.

3. Architecture / Algorithm: ArBG models the joint density p_θ(x) autoregressively as p_θ(x) = ∏j p_θ(x_j | x{<j}), where x_j is the j-th coordinate. This removes the invertibility constraint of flows and enables exact single-pass likelihood computation. Conditionals are parameterized via novel uniform binning categorical distributions (discretizing continuous values into fixed bins), or as mixtures of logistics or Gaussians following PixelCNN++ style MDNs. The uniform bin approach offers improved training stability and aligns with autoregressive LLM training paradigms. The generation order follows peptide residue structure.

4. Training Regime: Models are trained to maximize likelihood on training sets via maximum likelihood estimation (MLE) using standard optimizers (presumably Adam-type, though exact hyperparameters are not fully specified). Multiple model sizes are explored, including the large-scale 132M parameter ROBIN model. Training curves showing FLOPs and loss are reported. Temperature tuning is applied at inference to improve sample quality.

5. Evaluation Protocol: Three Wasserstein metrics compare model samples against ground truth MD simulation data: energy Wasserstein distance (E-W2) for local physical accuracy, torsional Wasserstein distance (T-W2) to measure global structural mode coverage, and TICA-W2 to evaluate slow dynamical modes. Comparisons are made against prior flows (ECNF++, RegFlow, SBG), flow-map methods (FALCON variants), energy-based models (BoltzNCE), and other autoregressive baselines (MoL/GMM-PixelCNN++). Evaluation budgets include 10^4 and 2×10^5 energy function calls. Zero-shot transfer learning performance is assessed on unseen peptide sequences. Resampling via self-normalized importance sampling (SNIS) or twisted sequential Monte Carlo (SMC) is applied.

6. Reproducibility: The authors release code (https://github.com/danyalrehman/autobg). The ManyPeptidesMD dataset is public as from Tan et al. (2025b). Exact frozen weights and detailed hyperparameters are not explicitly detailed in the truncated text. The continuous-time NF baselines and prior BGs are publicly documented.

Concrete example: For the 10-residue Chignolin system, the ArBG model flattens all 30 atomic coordinates, discretizes them into bins, and models the joint via autoregressive prediction of categorical bin indices conditionally on prior coordinates. During inference, resampling via SNIS or twisted SMC is used with energy evaluations of partial subsequences to apply physical constraints. Metrics like E-W2 are computed by comparing energies of generated samples to MD reference samples, demonstrating significantly improved fit over flows such as SBG, including better recovery of known β-hairpin secondary structure motifs as visualized by Ramachandran plots.

Technical innovations

• Replacing normalizing flow-based Boltzmann Generators with an autoregressive framework that factorizes molecular densities without diffeomorphism constraints.
• Introducing a uniform binning categorical parameterization of continuous conditional distributions to improve training stability and enable discrete token-style training for molecular coordinates.
• Developing autoregressive twisted Sequential Monte Carlo (SMC) inference allowing substructure-level intermediate resampling and physical constraint enforcement during sample generation.
• Scaling autoregressive Boltzmann Generators to large models (e.g., 132M parameter ROBIN) leveraging architectures inspired by Large Language Models for improved zero-shot transferable molecular sampling.

Datasets

• ManyPeptidesMD — 30 sequences each of length 4 and 8 residues — public (Tan et al., 2025b)
• Single peptide systems: alanine dipeptide (ALDP), tri-alanine (AL3), tetrapeptide (AL4), hexa-alanine (AL6), Chignolin (10 residues) — sizes vary — from prior BG benchmark setups (Tan et al., 2025a)

Baselines vs proposed

• SBG: E-W2 on Chignolin = 10.819 ± 7.206 vs ArBG (tuned T): 1.723 ± 0.075
• ECNF++: E-W2 on AL4 = 5.638 ± 0.483 vs ArBG (tuned T): 0.449 ± 0.030
• GMM-PixelCNN++: E-W2 on AL3 = 0.249 ± 0.025 vs ArBG (T=1): 0.271 ± 0.113
• Prose: zero-shot E-W2 on 8AA peptides = 10.038 vs ROBIN: 4.251 under 10^4 energy evals (Table 2)
• Prose: zero-shot E-W2 on 8AA peptides = 9.360 vs ROBIN: 3.615 under 2×10^5 evaluations
• MD simulations require 3 orders of magnitude more energy evaluations to reach comparable T-W2 than ROBIN (Fig 1).

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.27361.

Fig 1 (page 15).

Fig 2 (page 15).

Fig 3 (page 24).

Limitations

• The autoregressive factorization requires an ordering of atom coordinates, which may not capture all spatial symmetries optimally compared to equivariant flows.
• Twisted SMC yields only marginal gains over SNIS for well-trained ROBIN, indicating limitations of current intermediate resampling heuristics.
• Scaling and performance were extensively demonstrated only up to 10-residue peptides; large biomolecules remain untested.
• The impact of bin size resolution introduces a discretization-induced modeling error quantified but not eliminated.
• The study lacks thorough adversarial or distributional shift evaluations beyond unseen peptides in the ManyPeptidesMD dataset.
• Details on training hyperparameters and hardware are incomplete, potentially limiting reproducibility.

Open questions / follow-ons

• How to extend ArBG to handle larger biomolecules with hundreds of residues while maintaining performance and tractability?
• Can equivariance or symmetry-aware architectures be integrated with autoregressive factorization to further improve sample quality?
• What refined intermediate resampling or control variate techniques could amplify the gains from twisted SMC methods?
• How robust are ArBG and ROBIN to novel biochemical modifications or environments outside the training distribution?

Why it matters for bot defense

Although this work is primarily focused on molecular simulation rather than bot-defense or CAPTCHA tasks, the core innovation of autoregressive likelihood-factorized generative modeling that scales effectively to complex, high-dimensional structured data has conceptual relevance. For bot-defense, the ability to explicitly factorize challenging multimodal probability distributions with exact likelihoods could inspire more expressive generative user-behavior or challenge-response models. Additionally, the newly enabled inference-time interventions and efficient resampling via twisted sequential Monte Carlo parallel techniques used in interactive security protocols where adaptive challenge generation or filtering of adversarial inputs is important. The methodology may influence future CAPTCHA schemes that require diverse but physically or logically consistent challenge generations without costly calculations typical in normalizing flow models.

Cite

@article{arxiv2606_27361,
  title={ Autoregressive Boltzmann Generators },
  author={ Danyal Rehman and Charlie B. Tan and Yoshua Bengio and Avishek Joey Bose and Alexander Tong },
  journal={arXiv preprint arXiv:2606.27361},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.27361}
}

Read the full paper

• arXiv:2606.27361 (abstract)
• arXiv:2606.27361 (PDF)