Multiplayer Interactive World Models with Representation Autoencoders

Source: arXiv:2607.05352 · Published 2026-07-06 · By Anthony Hu, Václav Volhejn, Adrien Ramanana Rahary, Chris Mulder, Aditya Makkar, Amélie Royer et al.

TL;DR

This paper introduces MIRA, the first multiplayer world model designed for highly dynamic, physically complex environments governed by multiple agents acting simultaneously. Unlike prior single-agent world models that treat other agents as part of the environment, MIRA explicitly conditions on the concurrent action streams of all four players and learns to attribute scene changes to the correct agent while maintaining temporal and multi-view coherence. The authors study this problem in the fast-paced multiplayer game Rocket League, collecting 10,000 hours of self-play matches from a state-of-the-art RL bot policy and training a 5-billion-parameter latent diffusion model. MIRA runs in real time on a single Nvidia B200 GPU, generating 20 frames per second, and produces stable rollouts far beyond the training clip lengths, with quality metrics holding steady to at least five minutes of simulated time. The model jointly generates the synchronized first-person views of all players and remains physically consistent and responsive to the agents' controls throughout long simulations.

Key design elements investigated systematically include the choice of video codec, latent prediction space, generative objective (diffusion forcing versus teacher forcing), conditioning scheme over multiple action streams, and scale in data and model size. Evaluations measure not only visual fidelity but also physical consistency, action recoverability ratio, and human preference, validating that MIRA captures the causal relationships between agent actions and environment states in a way prior single-agent models cannot. The authors release both a large-scale Rocket League dataset and their full training and inference codebase to advance research on multiplayer world models.

Key findings

• MIRA’s 5-billion-parameter latent diffusion model generates synchronized four-player Rocket League matches in real time at 20 fps on a single Nvidia B200 GPU.
• Trained only on short clips, MIRA’s rollout distribution quality remains stable out to 5 minutes (the longest measured horizon), with practical rollouts observed for hours without collapse.
• Multiplayer models conditioned explicitly on all players’ actions outperform single-player baselines in predictive accuracy and physical consistency, supporting better uncertainty reduction.
• Initializing the multiplayer model via warm-start from a pretrained single-player model yields better training and stability than training multiplayer from scratch.
• Building the latent representation on a frozen pretrained DINOv3 self-supervised feature extractor improves both reconstruction quality and temporal consistency of rollouts over training a codec from scratch.
• Diffusion forcing combined with few-step distillation leads to stable long-horizon video rollouts while enabling real-time inference speeds.
• Using a fixed-length rolling latent context window and streaming KV-cache streaming enables autoregressive simulation at 10 Hz latent prediction and 20 fps video decode.
• Scaling from 100M to 5B model parameters and from 100 to 10,000 hours of gameplay data produces emergent capabilities and characteristic failure modes, documenting the importance of scale.

Threat model

Not a security-focused work; the threat model centers around a synthetic predictive model tasked with accurately attributing environmental dynamics to simultaneous multi-agent actions and producing causally correct, temporally coherent video rollouts. No adversarial assumptions are made about manipulative or malicious actors.

Methodology — deep read

1. Threat model & assumptions: The work addresses a research setting where an agent aims to learn a predictive world model of a complex multiplayer environment. Adversarial aspects are not considered; instead the focus is on attribution of observed changes in a shared environment to correct player actions and generating causally faithful futures. The setting assumes full knowledge of all players’ low-level action streams and no access to privileged game state during generation. The adversary is thus a synthetic modeler trying to emulate the environment accurately without oracle state.

2. Data provenance, size, labels, splits: The dataset consists of approximately 10,000 hours of Rocket League 2v2 matches, resulting in 82,983 matches and 331,932 first-person video recordings, evenly split across three fixed arenas. All games were played by identical instances of the open-source RL bot Nexto, driving all four cars independently. Video was recorded at 30 fps and later downsampled to 20 fps. Actions are recorded at 15 Hz, with the underlying physics game state logged at 120 Hz for evaluation but not used in training. The data is synchronized so each player’s video, action streams, and game state align on a shared timeline. The dataset is publicly released.

3. Architecture / algorithm: The model predicts in the latent space of a video representation codec rather than pixels. The codec is a representation autoencoder where encoding is done via a frozen pretrained DINOv3 self-supervised image feature extractor mixed across layers and then downsampled spatially (2x2) and temporally (2x) via a linear bottleneck. The decoder is a spatio-temporal causal Vision Transformer that upsamples spatially and temporally to reconstruct video at 20 fps. The world model is a conditional latent video diffusion transformer with ~5B parameters that predicts the next latent frame autoregressively at 10 Hz, conditioned on past latents of all four players and all player action streams. The prediction is causal in time and conditioned on multi-agent actions simultaneously. Diffusion forcing during training introduces noise to prediction targets to improve stability. Few-step diffusion distillation compresses the sampling steps for faster inference.

4. Training regime: The authors train on 10,000 hours of gameplay data, varying model scale up to 5 billion parameters. Training uses large batch sizes and standard optimizers (not fully detailed). The multiplayer model is warm-started from a pretrained single-player model. They train for stable long-horizon prediction with losses including reconstruction L1, LPIPS, and perceptual feature losses balanced adaptively. Latent frames are predicted autoregressively with a fixed-length rolling context window. Training seeds, hardware, and hyperparameters are not fully detailed but training is computationally intensive.

5. Evaluation protocol: Multiple metrics assess visual fidelity (PSNR, LPIPS, FVD), physical consistency (probing exact ground-truth game state with learned readout heads), and action recoverability ratio (ARR) which measures how well predicted frames correspond to commanded actions. Human preferences were collected for realism. Evaluations include long-horizon rollouts (up to 5 minutes+), ablation on codec design, generative objectives (diffusion forcing vs teacher forcing), and conditioning schemes (single vs multiplayer). Baselines include single-player models, models trained from scratch, and scale variants. Metrics are tracked over training and broken out by conditions.

6. Reproducibility: The authors provide full training and inference code, a public dataset, and a live web demo. Weights and seeds are not explicitly stated. The dataset is open, but limited to Rocket League 2v2 matches with RL bot players.

Example end-to-end forward pass: At each timestep t, the encoder processes each of the four players’ past video frames into latent vectors z_t for each player. The world model transformer receives these latents plus the four players’ action vectors a_t, then predicts the next latent vector z_{t+1} for all players jointly. This predicted latent is passed through the decoder to reconstruct the four players’ future video frames at t+1. By appending this predicted latent to the rolling context window and dropping the oldest, the model auto-regressively continues rollout into the future, repeating at 10 Hz latent steps to generate synchronized multiplayer views conditioned on all players’ actions.

Technical innovations

• First explicit multiplayer latent diffusion world model conditioning jointly on multiple agents’ action streams to maintain causal and coherent multi-agent dynamics.
• Use of a representation autoencoder built on frozen pretrained DINOv3 self-supervised features to build a latent space semantically aligned for stable long-horizon video prediction.
• Integration of diffusion forcing with few-step diffusion distillation to enable stable, long-horizon autoregressive latent video rollouts at real-time speeds on a single GPU.
• A rolling fixed-length latent context window combined with streaming KV-cache and space-time causal attention enables efficient autoregressive prediction of future latent frames.
• Systematic evaluation methodology combining visual fidelity, physical consistency via privileged game state probing, and action recoverability ratio (ARR) to validate causal multi-agent dynamics.

Datasets

• Rocket League 2v2 Multiplayer Matches — 10,000 hours — publicly released at https://huggingface.co/datasets/kyutai/rocket-science

Baselines vs proposed

• Single-player world model baseline: lower physical consistency and action recoverability ratio than multiplayer MIRA model (Section 6.6).
• Multiplayer model trained from scratch: worse rollout stability and accuracy compared to multiplayer model warm-started from single-player pretrained weights (Section 6.6).
• Codec trained from scratch: similar reconstruction quality but inferior temporal consistency and rollout stability versus codec built on pretrained DINOv3 features (Section 6.3).
• Diffusion forcing objective: stable long-horizon rollouts vs teacher forcing baseline, which collapses earlier (Section 6.4).
• Model scale 100M params: less accurate rollouts and faster quality degradation vs 5B params model, showing large-scale benefits (Section 6.7).
• Training on 100 hours data: lower fidelity and emergent capabilities vs 10,000 hours training (Section 6.8).

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.05352.

Fig 1: World model imagination. Rows are the four players’ viewpoints (Players 1–4) and columns show three

Fig 2: Method overview. MIRA simulates four-player Rocket League in the latent space of a video representation

Fig 3 (page 3).

Fig 4 (page 3).

Fig 5 (page 3).

Fig 6 (page 3).

Fig 7 (page 3).

Fig 8 (page 3).

Limitations

• Behavioral diversity is limited as all players are controlled by independent instances of the same RL bot policy, potentially restricting generality.
• Dataset covers only three fixed arenas, lacking environment diversity and procedural generation found in some other settings.
• The game environment is nearly deterministic once actions of all players are known, reducing predictive uncertainty; results may not transfer to highly stochastic real-world environments.
• Reproducibility details such as exact seeds, hyperparameters, and training hardware are incomplete.
• No adversarial evaluation or robustness tests under perturbed or malicious action inputs were reported.
• The model, although stable up to 5 minutes and beyond in practice, has limited formal guarantees on rollout fidelity for hours-long simulations.

Open questions / follow-ons

• How to extend multiplayer world models to settings with heterogeneous agent behaviors and richer behavioral diversity, including human players.
• Application of these techniques to more stochastic or procedurally generated environments, testing generalization beyond fixed maps.
• Integration of such multiplayer world models into multi-agent reinforcement learning pipelines for improved policy training and coordination.
• Further improving rollout stability and visual fidelity for very long-term predictions beyond current empirical horizons.

Why it matters for bot defense

From a bot-defense or CAPTCHA perspective, MIRA demonstrates the feasibility of building detailed multi-agent world models that can correctly attribute and reproduce complex user interactions embodied as action streams, even under fast-paced, tightly coupled scenarios. Such models, conditioned on multiple simultaneous users’ inputs, could be adapted or extended to simulate realistic multi-user interactions for fraud detection or challenge generation. The ability to maintain long-horizon coherent rollouts conditioned on multiple agents’ actions also suggests utility in behavioral anomaly detection, where deviations from expected multi-user dynamics can be flagged. However, this research is primarily focused on game simulation and not adversarial bot-detection per se. Practitioners might draw inspiration from the conditioning scheme and evaluation protocols (e.g., action recoverability ratio) as metrics for bot-control fidelity in CAPTCHA or interaction monitoring systems.

Cite

@article{arxiv2607_05352,
  title={ Multiplayer Interactive World Models with Representation Autoencoders },
  author={ Anthony Hu and Václav Volhejn and Adrien Ramanana Rahary and Chris Mulder and Aditya Makkar and Amélie Royer and Manu Orsini and Alyx Liao and Adam Jelley and Eloi Alonso and Florian Laurent and Fredrik Norén and James Swingos and Jan Hünermann and Kent Rollins and Lucas Hosseini and Matthieu Le Cauchois and Maxim Peter and Pim de Witte and Tim Brown and Vincent Micheli and Moritz Böhle and Gabriel de Marmiesse and Viktoriia Sharmanska and Lucia Specia and Michael Black and Patrick Pérez },
  journal={arXiv preprint arXiv:2607.05352},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.05352}
}

Read the full paper

• arXiv:2607.05352 (abstract)
• arXiv:2607.05352 (PDF)