World Translation: Minimizing Sim-to-Real Gap with Backward Dynamics Extraction and Unpaired Domain Translation

Source: arXiv:2607.18154 · Published 2026-07-20 · By Xinchen Yao, Leixin Chang, Hua Chen

TL;DR

This paper addresses the fundamental sim-to-real gap challenge in robotic control, where policies trained in simulation perform poorly on real robots due to differences in dynamics. Prior real-to-sim methods attempt to learn dynamics models from real data conditioned on observation history, but suffer from partial observability: unobservable factors (e.g., sudden contacts) cause ambiguous transitions not recoverable from history alone. The authors propose World Translation, a novel framework that extracts latent dynamics features backward from observed transitions, instead of forward prediction from history, to capture hidden variables more reliably. Additionally, they formulate transforming these extracted features across domains (simulation and real) as an unpaired domain translation problem, using cycle-consistent adversarial training to align dynamics representations without requiring paired sim-real samples. Experiments on humanoid, quadruped, and manipulator platforms demonstrate that their approach reduces prediction error substantially compared to baselines, especially under high hidden-variable influence where prior methods fail. Real-world deployment on a Go2 quadruped confirms improved policy performance. Overall, the method exploits complementary strengths of simulators (deterministic but imperfect) and learned models (accurate but underdetermined) via latent backward extraction and domain translation to more accurately bridge sim-to-real dynamics gaps.

Key findings

• World Translation reduces single-step prediction MSE by up to 0.119 to 0.035 on G1 humanoid uneven terrain (high hidden variable, ht) task compared to Direct Prediction baseline.
• On R5 manipulator force disturbance (high-ht) task, World Translation achieves 0.401 MSE vs 0.741 for Direct Prediction.
• World Translation outperforms history-based RSSM model in high-ht tasks: e.g. Go2 payload task 0.148 vs 0.298 MSE and R5 force task 0.401 vs 0.515 MSE.
• In multi-step autoregressive rollouts on Go2 high-ht, World Translation maintains stability to horizon 30 with only ~2% divergence at 50 steps vs 20% divergence for Direct Prediction.
• Latent dynamics representation zt preserves hidden variable information with R2 up to 0.81 vs 0.73 from full transitions and retains domain characteristic info at 94% classification accuracy with auxiliary components.
• Ablation shows cycle consistency loss crucial for translation quality, increasing cycle error ~8x and prediction MSE by 69% without it.
• Real-world Go2 quadruped policy fine-tuned with World Translation improves angular velocity tracking RMSE to 0.1786 compared to 0.2475 (No Adaptation) and 0.3306 (Domain Randomization).
• Translation at latent space (zt) level succeeds, while direct observation-level translation collapses due to trivial domain cues in raw observations.

Threat model

The adversary corresponds to unobservable and unpredictable hidden variables (e.g., sudden contact events, external forces) that cause stochastic branching in observed transitions from identical observations and actions. The attacker cannot be prevented from creating these unknown state perturbations, and their exact influence is unmeasurable from observation history alone. The system assumes these hidden factors leave identifiable signatures in the complete transitions (o_t, a_t, o_{t+1}), enabling backward latent inference. There is no assumption the adversary can precisely replicate or control simulator states. Unmodeled phenomena completely absent from simulation physics are outside the scope.

Methodology — deep read

The paper formulates the sim-to-real gap problem where the system’s full state s_t is partially observable via o_t. Hidden time-varying variables h_t induce stochastic transitions even for same (o_t, a_t) pairs. The core insight is that while forward prediction methods from history fail under partial observability, the hidden variables' effects can be extracted backward from observed transitions (o_t, a_t, o_{t+1}) into a latent dynamics feature z_t = ω(h_t).

1. Threat model & assumptions: The adversary corresponds to unmodeled, unobservable dynamics factors (hidden variables h_t) that cause different outcomes from identical observation-action pairs. It assumes the simulator's physics is deterministic but imperfect; realistic dynamics differ due to hidden and domain variables. It assumes the hidden variables leave identifiable signatures in state transitions, recoverable by backward extraction.

2. Data: Transitions are collected in both source (simulation) and target (real or altered sim) domains from fixed pre-trained policies on three robotics platforms (G1 humanoid, Go2 quadruped, R5 manipulator). Transitions are split for VAE and CycleGAN training. Observations are proprioceptive only; no paired matching transitions between domains exist.

3. Architecture: The dynamics encoder (E_ϕ) is a variational autoencoder mapping transition tuples (o_t, a_t, o_{t+1}) to latent code z_t. The decoder (D_θ) reconstructs o_{t+1} conditioned on (o_t, a_t) and z_t, implemented with Feature-wise Linear Modulation layers to emphasize conditioning on z_t. A blind decoder (D_ψ) attempts to predict o_{t+1} from z_t alone, serving as adversarial regularizer to prevent z_t from encoding raw observation info. A domain classifier (C_ξ) forces z_t to contain domain (simulation vs real) information.

Unpaired domain translation is performed in latent space using CycleGAN with adversarial and cycle-consistency losses to learn mappings G_{S→R} and G_{R→S} translating latent codes between sim and real domains, preserving hidden variable semantics while modifying domain characteristics.

4. Training regime: VAE and CycleGAN training are performed jointly on batches sampled from source and target domains until convergence, using reconstruction, KL, blind decoder adversarial loss, classifier cross-entropy, adversarial GAN loss, cycle consistency, and identity losses. Domain randomization is disabled for deterministic sim transitions. Hyperparameters are fixed across platforms.

5. Evaluation: Single-step prediction uses teacher forcing with ground-truth observations to measure isolated dynamics error via MSE. Multi-step autoregressive rollouts evaluate stability and compounding error. Ablations test component necessity. Domain classification accuracy of latent codes probes learned representation quality. Real robot policy fine-tuning and deployment measure transfer improvements. Baselines include RawSim, DirectPred, Residual Dynamics, and RSSM for comparison.

6. Reproducibility: No public code or dataset release mentioned. Experiments use publicly known simulated platforms and a real Go2 quadruped robot but no pairing of sim-real transitions available.

Technical innovations

• Backward extraction of hidden dynamics features directly from observed transitions to overcome partial observability limitations of history-based methods.
• Formulation of sim-to-real hidden variable adaptation as unpaired domain translation in latent dynamics feature space using cycle-consistent adversarial training.
• Use of blind decoder adversarial regularization and domain classifier auxiliary losses to disentangle and preserve hidden variable and domain-specific information in learned latent codes.
• Deployment pipeline that overwrites simulator state transitions with translated latent dynamics features to align simulated trajectories to real-world dynamics.

Datasets

• G1 Humanoid simulated transitions — size not specified — Isaac Lab
• Go2 Quadruped simulated and real transitions — size not specified — Isaac Lab and real robot
• R5 Manipulator simulated transitions with force disturbances — size not specified — Isaac Lab

Baselines vs proposed

• Direct Prediction: MSE on G1 uneven task = 0.119 vs World Translation: 0.035
• Direct Prediction: MSE on R5 force task = 0.741 vs World Translation: 0.401
• RSSM model: MSE on Go2 payload task = 0.298 vs World Translation: 0.148
• No Adaptation policy tracking error on real Go2 = 0.2475 vs World Translation fine-tuned = 0.1786
• Domain Randomization policy tracking error on real Go2 = 0.3306 vs World Translation = 0.1786

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.18154.

Fig 1: World Translation bridges the sim-to-real gap by extracting

Fig 3: Simulation environments. Left: G1 humanoid on irregular terrain.

Fig 6: UMAP visualization of dynamics features on the R5 manipulator

Fig 8: Real-world deployment of the Go2 quadruped with the payload

Limitations

• Backward extraction assumes hidden variables create distinct, recoverable signatures in transitions; subtle continuous forces may be harder to capture.
• Unmodeled phenomena entirely absent in the simulator cannot be recovered by translation; sim must approximate relevant physics.
• Training involves complex joint objectives with adversarial components, which may exhibit stability issues or silent latent failure modes.
• Coupling between hidden variables and domain characteristics (e.g., forces with varying actuator gains) complicates disentanglement and reduces gains, as seen in R5 manipulator.
• State overwriting in deployment ignores internal simulator latent states, possibly causing physics inconsistencies over long horizons though effects appear bounded in experiments.
• No public code or datasets released currently limits reproducibility and external validation.

Open questions / follow-ons

• How can the method be extended to incorporate multi-step training and closed-loop rollout optimization to reduce error accumulation?
• Can explicit disentanglement of hidden variable effects and domain characteristics improve translation and transfer, especially under tight coupling?
• What are robust strategies to mitigate training instability and silent latent space failure modes in adversarially trained representations?
• Could combining action-level correction methods with latent state translation enhance simulator internal consistency and improve long-horizon rollouts?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this paper’s approach to bridging sim-to-real gaps via backward dynamics extraction and unpaired domain translation is a compelling example of handling partial observability and domain adaptation without requiring paired data. Analogously, CAPTCHA systems face challenges in modeling user interaction dynamics and adversarial behavior under hidden factors and domain shifts (e.g., different device types or bot vs human interactions). The latent backward extraction and domain translation framework could inspire novel defense features extracting causative hidden factors from outcomes rather than forward prediction. Additionally, the cycle-consistent unpaired translation provides a blueprint for mapping behavioral features across domains where paired ground truth is unavailable, such as between known bot distributions and novel unseen clients. However, the robotics-specific assumptions and controlled physics simulation limit direct applicability, and practical bot-defense data may have more complex or less structured hidden factors. Still, the core methodology highlights how combining simulators or heuristic models with learned corrections in a latent space can better capture hidden, domain-specific factors important for realistic modeling and detection.

Cite

@article{arxiv2607_18154,
  title={ World Translation: Minimizing Sim-to-Real Gap with Backward Dynamics Extraction and Unpaired Domain Translation },
  author={ Xinchen Yao and Leixin Chang and Hua Chen },
  journal={arXiv preprint arXiv:2607.18154},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.18154}
}

Read the full paper

• arXiv:2607.18154 (abstract)
• arXiv:2607.18154 (PDF)