Hallucination in World Models is Predictable and Preventable

Source: arXiv:2606.27326 · Published 2026-06-25 · By Nicklas Hansen, Xiaolong Wang

TL;DR

This paper investigates the problem of hallucination in modern generative world models—models that predict future observations conditioned on agent actions. Although these models produce visually plausible rollouts, they often silently diverge from true environment dynamics, which can negatively impact downstream decision-making and control policies. The authors hypothesize that hallucination primarily arises from insufficient data coverage in state-action space and propose lightweight, data-centric signals to predict and mitigate hallucinations.

To experimentally validate their claims, they introduce MMBench2, a large-scale multitask dataset of 427 hours of behaviorally diverse, continuous-control visual trajectories spanning 210 tasks with ground-truth actions, rewards, and live simulators. They train a 350 million parameter Transformer-based world model on this dataset and identify three distinct hallucination modes—perceptual, action-marginalized, and scene-diverging—each corresponding to a different pipeline stage (encoder-decoder, dynamics, and rollout respectively). For each mode, they develop predictors that detect hallucinations effectively without additional labels or training. They then show how to use these signals both during training (coverage-aware data sampling) and online (hallucination-predictor-based curiosity rewards) to reduce hallucination and enable efficient fine-tuning on unseen tasks with as few as 50 real environment trajectories. The experimental results demonstrate strong correlations (~0.8 Spearman) between their signals and rollout error, reduced hallucination with coverage-aware training, and improved downstream control performance particularly with targeted data collection.

Overall, the paper argues convincingly that hallucination in modern world models is primarily a data coverage issue rather than just an architectural limitation, and that data-centric detection and mitigation strategies can significantly improve generative model reliability and generalization in complex control domains.

Key findings

• Hallucination correlates strongly with low data coverage in the state-action space across diverse tasks (Figure 4).
• Three distinct hallucination modes are identified: perceptual (encoder-decoder), action-marginalized (dynamics ignoring input actions), and scene-diverging (multi-step rollout error).
• Three no-label, runtime hallucination predictors achieve strong Spearman correlations with realized rollout error across 9,000 sequences: tokenizer round-trip residual (ρ=−0.81), flow instability (ρ=−0.79), inter-seed variance (ρ=−0.80) (Figure 5).
• Coverage-aware sampling (uniform sampling over tasks rather than frames) reduces all three hallucination signals simultaneously and improves rollout ∆PSNR by +0.88 dB, action-shuffle ratio +0.29, and lowers hallucination scores (Table 1).
• Online data collection guided by hallucination predictor curiosity rewards adapts pretrained world models to 10 unseen tasks with just 50 environment trajectories, achieving normalized MPC closed-loop performance within 90% of expert/human oracles (0.325 vs 0.362) on unseen tasks (Table 2).
• Pretraining transfers zero-shot moderately with normalized score 0.276 vs random baseline 0.118, but finetuning with hallucination-driven data outperforms and closes the gap rapidly.
• In-domain tokenizer trained on MMBench2 outperforms off-the-shelf tokenizers on seen tasks by +1.62 PSNR but underperforms on unseen tasks without finetuning, highlighting the benefits of domain-specific training (Table 3).
• Normalizing hallucination predictor signals by latent scene motion improves predictive accuracy compared to raw signals or baseline metadata such as frame counts.

Threat model

The threat model is implicit: the adversary is essentially distributional shift where the world model encounters state-action regions insufficiently covered in training data, causing silent hallucination in rollouts. The model does not consider malicious attackers, sensor noise, or adversarial inputs. The adversary cannot manipulate the training data directly or gain privileged knowledge of internal latent states; the failure is due to limited training coverage and extrapolation.

Methodology — deep read

The paper undertakes a comprehensive empirical study built on a large-scale multitask dataset and a sizable generative world model.

1. Threat model & assumptions: The adversary is implicit—hallucination is treated as a failure mode where the model silently generates plausible but incorrect rollouts due to incomplete training data coverage. The authors assume no malicious adversary, but rather natural distributional shift to unseen states and actions. They do not consider adversarial attacks or sensor noise effects.

2. Data: MMBench2 consists of 210 continuous control tasks covering locomotion, manipulation, and arcade domains from 10 sources (e.g. DMControl, Meta-World, Atari). It comprises 65,600 mixed-quality trajectories totaling 427 hours of 224×224 video at 15 fps, with 23M frames, ground-truth actions, rewards, and live simulators. 200 tasks form the pretraining corpus (~20M frames training, 3M test), with 10 held out for unseen transfer. Actions are zero-padded to 16 dimensions with validity masks. Dataset is behaviorally diverse including human play data. Split details and episodic length distributions vary by task.

3. Architecture/algorithm: The model follows Dreamer 4 architecture with 350M parameters total. The tokenizer is a 50M parameter symmetric encoder-decoder Transformer that encodes RGB frames into 64×64 latent codes with masked reconstruction loss combining pixel MSE and LPIPS. It uses patchified inputs and latent queries. The dynamics model is a 250M parameter block-causal Transformer over packed latents, conditioned on actions encoded by MLP, trained with shortcut flow-matching loss combining one-step regression and multistep consistency. The model also includes reward prediction and behavior cloning heads conditioned on CLIP-ViT/B language embeddings.

4. Training regime: Tokenizer pretrained for 300k steps (~14 GPU days on 8 NVIDIA H100s). Dynamics pretrained for 180k steps (~24 GPU days). Finetuning extended training to 380k tokenizer and 210k dynamics steps (~58 GPU days total). Mini-batches with T=24 context length. Coverage-aware sampling uniformly samples tasks rather than frames during 30k additional steps. Finetuning on unseen tasks uses targeted data collected with hallucination predictors guiding exploration.

5. Evaluation: Metrics include reconstruction PSNR (encoder-decoder), rollout ∆PSNR vs frame-repeat baseline, action shuffle ratio (one-step flow MSE sensitivity to action permutation), and downstream MPC control performance via normalized scores on held-out and unseen tasks. Correlations between hallucination predictors and rollout error are computed (Spearman ρ), AUROC against hallucination event labels, and ablations for coverage-aware training effects. Online data collection experiment compares multiple policies (expert, human, curiosity-driven, random) with finetuning results.

6. Reproducibility: The authors release full MMBench2 dataset, code, pretrained checkpoints, and an interactive browser interface for model rollouts and hallucination visualization. Detailed experimental and implementation specifics are provided in appendices.

Example: A typical experiment involves pretraining the tokenizer and dynamics on 200 tasks, then computing the tokenizer round-trip residual for predicted latent codes on held-out test sequences. A large residual indicates perceptual hallucination. They also shuffle actions in batches to test flow MSE sensitivity (detecting action marginalized hallucination), then perform multi-seed rollouts to measure inter-seed variance linked to scene-divergence hallucination. Coverage-aware data re-sampling is applied, followed by retraining and verification of reduced hallucination metrics and improved downstream control scores.

Technical innovations

• Identification and clear taxonomy of three distinct hallucination failure modes in world models mapped to separate model components: perceptual, action-marginalized, and scene-diverging hallucinations.
• Development of three no-label, lightweight hallucination predictors (tokenizer round-trip residual, flow instability, inter-seed variance) that accurately detect hallucinations at runtime without additional training.
• Introduction of MMBench2, a large-scale, behaviorally diverse multitask dataset for visual world modeling that includes ground-truth actions, rewards, and live simulators across 210 continuous control tasks.
• A coverage-aware training strategy that rebalances sampling to uniformly cover tasks, reducing hallucinations holistically across tokenizer and dynamics components without extra cost.
• Use of hallucination predictor signals as curiosity-driven reward functions for targeted, closed-loop online data collection enabling efficient finetuning on unseen tasks with minimal environment interactions.

Datasets

• MMBench2 — 427 hours (65,600 trajectories, 23M frames) — public release with live simulators, 210 continuous control tasks

Baselines vs proposed

• Base model rollout ∆PSNR = −12.44 dB vs coverage-aware training rollout ∆PSNR = −11.56 dB (improved by +0.88 dB) (Table 1)
• Base action-shuffle ratio = 1.12 vs coverage-aware training = 1.41 (more sensitive to action)
• Zero-shot pretrained model normalized task performance = 0.276 vs random baseline 0.118 (2.3× improvement)
• Finetuning with hallucination-driven curiosity data collection normalized task performance = 0.325 vs expert/human oracle = 0.362 (∼90% of oracle) (Table 2)
• Tokenizer PSNR on unseen tasks: off-the-shelf Wan 2.1 VAE = 36.62 dB vs their base tokenizer 17.34 dB unfinetuned, finetuned tokenizer = 38.04 dB (Table 3)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.27326.

Fig 1: Hallucination. We categorize hallucinations as three distinct failure modes: perceptual,

Fig 2: MMBench2 tasks. A sample of 36 of the 210 tasks in MMBench2, illustrating the visual

Fig 3: Dataset composition. Per-task frame counts across the 210 tasks in our training corpus

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 2).

Fig 7 (page 2).

Fig 8 (page 2).

Limitations

• Experiments use simulated continuous control tasks with access to ground-truth actions and rewards; results may not generalize directly to real-world robotics or noisy sensor data.
• Model scale limited to ~350M parameters; it remains an open question whether trends hold for billion-parameter world models or larger-scale video architectures.
• The hallucination predictors and mitigation strategies have been tested mostly on state coverage gaps, not on adversarial distribution shifts or sensor imperfections.
• Computational cost is substantial, requiring many GPU-days and large-scale expert data for training and finetuning.
• The coverage-aware sampling approach relies on task labels and uniformity assumptions, which may be less feasible in uncontrolled or online environments.
• While effective, there remain hallucination modes and factors not fully explained by data coverage alone (architecture or training dynamics).

Open questions / follow-ons

• How well do these hallucination predictors and mitigation strategies scale to billion-parameter or more complex video-conditioned world models?
• Can similar coverage-aware sampling and predictor-driven data collection approaches be adapted for real robot datasets with partial observability and sensor noise?
• How do architectural changes in tokenizers or dynamics influence hallucination modes independently of data coverage?
• What are effective ways to evaluate hallucination and model fidelity in real-world downstream tasks beyond quantitative rollout metrics?

Why it matters for bot defense

Although this study focuses on generative world models for control, the underlying insight that hallucination—generation of plausible yet incorrect outputs—is tightly linked to data coverage gaps extends to many generative AI domains, including CAPTCHA generation and bot-defense models. Practitioners working on visual CAPTCHA or bot detection systems that rely on predictive world models could use similar runtime hallucination predictors to identify input scenarios poorly supported by training data and proactively mitigate failure modes in model outputs. The coverage-aware data sampling and hallucination-driven curiosity data collection techniques suggest interesting avenues for improving training datasets and model robustness with guided data acquisition. Additionally, the interpretable hallucination taxonomy provides a framework to diagnose failures at perception, conditional prediction, or rollout stages, which can inform targeted defense mechanisms for adversarial or OOD input detection. However, domain transfer of these methods beyond robotics domains would require validation with appropriate data and environments.

Cite

@article{arxiv2606_27326,
  title={ Hallucination in World Models is Predictable and Preventable },
  author={ Nicklas Hansen and Xiaolong Wang },
  journal={arXiv preprint arXiv:2606.27326},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.27326}
}

Read the full paper

• arXiv:2606.27326 (abstract)
• arXiv:2606.27326 (PDF)