Isaac Sim-to-Real: Reinforcement Learning based Locomotion for Quadrupeds
Source: arXiv:2607.18135 · Published 2026-07-20 · By Jordan Dowdy, Jean Chagas Vaz
TL;DR
This paper addresses the sim-to-real gap challenge in reinforcement learning (RL) based locomotion for quadruped robots. It presents an end-to-end RL framework trained entirely in Nvidia's Isaac Sim environment and deployed zero-shot onto the Unitree Go1 hardware, avoiding any real-world fine-tuning. Unlike many prior works relying on hierarchical or teacher-student architectures, this work produces a low-level joint position control policy capable of whole-body locomotion. Key techniques to overcome the sim-to-real gap include domain randomization of physical parameters and a learned actuator model capturing non-linear joint dynamics. Experimental validation shows the learned policy achieves velocity tracking performance comparable to Unitree’s model-based integrated controller while demonstrating superior disturbance rejection and the ability to reach linear speeds of 2.0 m/s and angular velocities of 1.8 rad/s on rough terrain and during collision events.
Key findings
- Zero-shot sim-to-real transfer achieved on Unitree Go1 quadruped with no hardware fine-tuning.
- Policy tracks linear velocities up to 2.0 m/s and angular velocities up to 1.8 rad/s with a small phase lag compared to desired commands (Fig. 3 vs Fig. 4).
- Domain randomization ranges used: foot friction (0.4–1.1), body mass scaled 0.8–1.3, joint position noise ±30%, joint velocity noise ±250% (Table I).
- Actuator-net with 3 hidden layers (64 neurons each) modeled nonlinear actuator dynamics reducing sim-to-real gap.
- Policy uses 48-dimensional observation vector including velocity, gravity projection, joint states, and prior actions (Table II).
- Policy outperforms integrated controller in disturbance rejection, successfully recovering from three incremental kicks without falling (Fig. 5).
- Training completed in 5,000 PPO iterations, about 3 hours on an Nvidia RTX 3090 GPU using 4096 parallel environments.
- Policy inference runs at 400Hz on Nvidia Jetson Nano without GPU acceleration.
Threat model
Not applicable; this paper addresses sim-to-real transfer challenges in reinforcement learning locomotion rather than security or adversarial threats.
Methodology — deep read
The paper formulates quadrupedal locomotion as a Markov Decision Process (MDP) with continuous state and action spaces, using reinforcement learning to maximize cumulative rewards. The state space includes 48 dimensions covering linear and angular velocity, projected gravity vector, user velocity commands, joint positions and velocities, and previous joint actions. No scaling was applied to observations.
The action space is a 12-dimensional joint position vector, scaled and offset around a nominal standing posture to ensure stability at initialization. Actions feed into a separately learned actuator-net, an MLP with 3 layers of 64 neurons using soft-sign activations, trained to predict joint torques from past positions and velocities. This actuator modeling accounts for nonlinear actuator dynamics in hardware, critical to bridging sim-to-real.
To train the policy, they used Nvidia Isaac Sim with domain randomization on physical parameters such as foot friction, mass, initial joint states, and velocities with ranges specified in Table I, plus a terrain curriculum with ramped terrain difficulty including random heights, bumps, and noisy terrain. Random external pushes on the robot body were applied every 10-15 seconds.
The RL policy network uses an actor-critic architecture with MLPs of three layers sized 512, 256, and 128 neurons and ELU activations. Proximal Policy Optimization (PPO) was used with tuned hyperparameters including adaptive learning rate (0.001), entropy coefficient (0.0025), target KL divergence (0.01), discount factor γ=0.99, and advantage estimator λ=0.95. Training ran for 5,000 iterations (~3 hours) on an RTX 3090 GPU, simulating 4096 environments in parallel with physics step Δt=0.002s and policy step Δt=0.02s.
The reward function incorporated terms for velocity tracking, gait synchronization, foot clearance, and penalties to ensure smooth, safe, and natural locomotion (Table III). Heavily penalized joint limit violations and foot slippage.
For evaluation, the trained policy in ONNX format ran onboard a Jetson Nano controlling the Unitree Go1 robot, tested on various rough terrains plus velocity tracking against Unitree’s default integrated controller. Disturbance rejection was tested by humans applying incremental kicks to the robot trunk, measuring recovery without falling. Velocity tracking was assessed by sinusoidal velocity commands over 35 seconds. Observations were filtered with a Kalman filter implemented in C++ for low latency.
The paper emphasizes the critical role of actuator-net modeling and domain randomization for zero-shot sim-to-real transfer without fine-tuning. Detailed ablations or statistical analyses are not reported. Code and weights availability is not stated, indicating limited reproducibility beyond described methods.
Technical innovations
- Integration of a learned MLP-based actuator model (actuator-net) to capture nonlinear joint dynamics, improving sim-to-real transfer.
- Extensive domain randomization across friction, mass, initial states, and terrain difficulty combined with random pushing events to enhance policy robustness.
- End-to-end low-level joint position control policy trained via proximal policy optimization without reliance on teacher-student architectures or trajectory generators.
- Policy and observation design that incorporates previous actions and multiple proprioceptive sensors without scaling observations to improve training stability.
Baselines vs proposed
- Unitree Go1 integrated controller: velocity tracking error shows larger phase lag and foot slippage at high frequencies than proposed RL policy (Fig. 3 vs Fig. 4).
- RL policy recovers from three consecutive pushes without falling, outperforming baseline model-based control in disturbance rejection (Fig. 5).
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.18135.
Fig 1: Unitree Go1 walking up an incline pallet using a end-to-end RL
Fig 2: Simulation of Unitree Go1 in Isaac Sim during policy inference.
Fig 3: Policy velocity tracking performance with a generated sinusoidal
Fig 4: Unitree’s model-based velocity tracking performance with a
Fig 5: Policy disturbance rejection during three separate kicking events. Each kick increased in force applied to the trunk of the quadruped. The green
Limitations
- No fine-tuning or real-world data collection reported; true robustness outside tested conditions uncertain.
- Evaluation limited to a single quadruped platform (Unitree Go1) and specific terrains.
- No adversarial or systematic stress tests beyond randomly applied pushes were conducted.
- Absence of statistical significance testing or detailed ablation studies on domain randomization parameters.
- No discussion on long-term hardware wear or policy safety under extreme conditions.
- Code and pretrained weights availability unspecified, limiting reproducibility.
Open questions / follow-ons
- How would the proposed RL policy perform on different quadruped platforms or more varied unstructured environments?
- What is the impact of removing or altering individual domain randomization components on sim-to-real robustness?
- Can the actuator-net approach be generalized to other robot types with different actuation modalities?
- How does the policy perform under longer-term deployment in terms of stability, robustness, and hardware safety?
Why it matters for bot defense
While this paper does not address bot-defense or CAPTCHA systems directly, its contributions to zero-shot sim-to-real transfer for control policies highlight general principles useful in secure robotics deployment. For CAPTCHA and bot-defense practitioners working on physical robots interacting with humans or complex environments, the demonstrated domain randomization and learned actuator modeling provide a blueprint to train robust policies in simulation before deployment, reducing reliance on vulnerable real-world fine-tuning. Moreover, the approach underlines the importance of modeling subtle hardware dynamics to bridge simulation and reality effectively, a consideration potentially relevant for designing secure autonomous agents physically embodied in adversarial settings. However, explicit adversarial robustness evaluation is lacking, which bot-defense specialists would want to consider when adapting similar techniques.
Cite
@article{arxiv2607_18135,
title={ Isaac Sim-to-Real: Reinforcement Learning based Locomotion for Quadrupeds },
author={ Jordan Dowdy and Jean Chagas Vaz },
journal={arXiv preprint arXiv:2607.18135},
year={ 2026 },
url={https://arxiv.org/abs/2607.18135}
}