Physics-enhanced reinforcement learning for real-time optimal control of dynamical systems

Source: arXiv:2607.16177 · Published 2026-07-17 · By Matteo Tomasetto, Nicolò Botteghi, Gabriele Bruni, Andrea Manzoni

TL;DR

This paper tackles the well-known sample inefficiency and scalability bottlenecks of reinforcement learning (RL) when applied to high-dimensional, complex dynamical systems governed by differential equations. Unlike classical model-free RL that treats the environment as a black box, the authors propose a novel Physics-EnhAnced Reinforcement Learning (PEARL) framework that leverages knowledge of differentiable system dynamics to compute exact policy gradients via automatic differentiation and adjoint sensitivity analysis. This actor-adjoint approach computes policy updates over short horizons while correcting the terminal adjoint state with a neural network trained to capture long-term sensitivities, thus addressing gradient instability and credit assignment challenges. Evaluated on parametric navigation control problems in challenging unsteady fluid flows, PEARL demonstrates significant improvements in sample efficiency and control performance compared to state-of-the-art model-free and differentiable environment RL baselines. It also generalizes across varying initial conditions and parametric scenarios without requiring low-dimensional embedding or multi-agent decomposition.

Key findings

• PEARL reduces environment interaction counts by leveraging physics-informed policy gradients through adjoint sensitivity, improving sample efficiency significantly over PPO, TD3, BPTT, and SHAC.
• The actor-adjoint algorithm computes gradients over short horizons to prevent vanishing/exploding gradients yet corrects terminal adjoint variables with a neural network, improving long-term credit assignment.
• In parametric leader-follower navigation tasks in double gyre flows, PEARL adapts policies effectively across multiple initial conditions and system parameters, demonstrating strong generalization.
• The method scales to high-dimensional control problems, such as distributed control over Gaussian densities, without explicit low-dimensional state representation or multi-agent schemes.
• Using the proposed TD-λ scheme for adjoint network training yields sharper physics-enhanced gradients tied to system dynamics, stabilizing policy optimization.
• Training the adjoint network with a target network and soft updates stabilizes the approximation of long-term value gradients.
• PEARL yields faster convergence compared to truncated backpropagation through time (BPTT) and SHAC by directly approximating value gradients rather than only cumulative values.
• Exact gradients computed via the adjoint state method have computational cost independent of policy parameter dimension, aiding scaling to large neural policies.

Threat model

Not a security threat model; the adversarial scenario is not applicable as the focus is on optimal control of known deterministic dynamical systems with available differentiable models.

Methodology — deep read

The threat model assumes deterministic dynamical systems governed by nonlinear ordinary or partial differential equations, where system dynamics are differentiable and known approximately. The adversary concept is n/a as this is a control optimization setting.

Data for evaluation comprises two parametric tasks involving navigation in a double gyre flow field: a leader-follower game tracking a target particle and a high-dimensional distributed control problem steering a Gaussian density. States include agent positions and scenario parameters sampled uniformly from a domain, with actions as velocity controls bounded within given limits. The datasets are generated by simulations of the dynamical system equations over a finite horizon.

The PEARL architecture combines standard feed-forward neural networks to embed states and scenario parameters, concatenated and processed through further layers to output deterministic control policies. The training employs an actor-adjoint algorithm that computes policy gradients via two components: short-horizon automatic differentiation (AD) through the differentiable environment and an adjoint network estimating the terminal adjoint variables (value gradients) at the end of the short horizon. This approximates the exact policy gradient that would be computed over the full horizon, circumventing gradient instability.

The adjoint network is trained using a temporal difference (TD-λ) scheme with targets formed by iterating backward the adjoint recursion relation, corrected by the adjoint network output itself. This leverages physics-based sensitivities rather than pure value estimation, which improves long-term credit assignment. Two copies of the adjoint network (online and target) are maintained with soft updates for training stability.

Training parameters include a discount factor γ=0.99, TD parameter λ=0.95, and a target network update rate α=0.995. Gradient-based optimization updates policy and adjoint networks online after every short horizon segment. The choice of short horizons (much less than full episode length) avoids vanishing or exploding gradients characteristic of backpropagation through time (BPTT).

Evaluation compares PEARL’s sample efficiency, convergence speed, and generalization across multiple varying initial conditions against PPO, TD3, BPTT, and SHAC. Metrics include cumulative reward, control error, and convergence time. Ablations confirm benefits of physics-informed adjoint network training versus standard value function critics.

The physical environment simulator is differentiable, crucial to enable backpropagation of gradients through system dynamics. Although code and exact hyperparameters are not explicitly released, the paper provides sufficient detail for methodological reproduction leveraging automatic differentiation frameworks.

In a concrete example, for the leader-follower navigation problem, PEARL processes state and scenario inputs through its neural policy to generate velocity commands at each control step, while the adjoint network provides corrected gradient estimates at the short horizon’s end to update policy parameters, reducing sample complexity compared to alternatives.

Technical innovations

• Integration of adjoint state method with automatic differentiation to compute exact policy gradients in differentiable dynamical systems.
• Use of a neural network-based adjoint (value-gradient) estimator trained via physics-informed TD-λ recursion to correct short-horizon gradient truncation bias.
• An actor-adjoint reinforcement learning algorithm updating policy parameters online over short horizons to mitigate gradient instability in long rollouts.
• Combining physics knowledge with RL for scalable control of high-dimensional and parametric dynamical systems without dimensionality reduction or multi-agent decomposition.

Datasets

• Leader-follower navigation in double gyre flow — simulation-generated, domain [0,2]x[0,1], parametric initial positions sampled uniformly
• Mean-field leader-follower game with distributed Gaussian density control — simulation-generated, high-dimensional state and action spaces

Baselines vs proposed

• PPO: lower sample efficiency and slower convergence compared to PEARL
• TD3: underperforms PEARL in reward accumulation and control precision
• BPTT: suffers from vanishing/exploding gradients leading to unstable policy updates, PEARL achieves more stable training
• SHAC: improves on BPTT but value function gradient approximation less accurate than PEARL’s adjoint network-based method, resulting in slower convergence and lower final performance

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.16177.

Fig 1: Graphical summary of Physics-EnhAnced Reinforcement Learning (PEARL). Starting from the current state of the

Fig 2 (page 2).

Fig 3 (page 2).

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 2).

Fig 7 (page 2).

Fig 8 (page 2).

Limitations

• Assumes differentiable environment models are available and accurately approximate system dynamics; not directly applicable to black-box or stochastic environments without modification.
• Evaluations are limited to deterministic parametric systems and simulated fluid dynamics navigation problems; real-world noisy or partially observable systems untested.
• Method relies on training an adjoint network, which may be sensitive to architecture choices and training stability in more complex or high-dimensional real problems.
• No adversarial robustness analysis or testing against model mis-specification is provided.
• Computational cost and scalability to very large-scale PDE systems beyond tested scenarios remains to be demonstrated.

Open questions / follow-ons

• How does PEARL perform when system dynamics are only partially known or noisy, e.g., incorporating model uncertainty?
• Can the adjoint network training approach be extended to stochastic and partially observable Markov decision processes?
• What are the limits of scalability in terms of state/action dimensionality and complexity of the underlying PDE models for PEARL?
• How robust is the learned policy and adjoint approximation to perturbations and distribution shifts in real-world scenarios?

Why it matters for bot defense

From a bot-defense and CAPTCHA perspective, PEARL's approach demonstrates how strong domain knowledge—here physics and differential equations—can be integrated into RL frameworks to improve sample efficiency, robustness, and generalization in complex control tasks. This philosophy aligns with advanced CAPTCHA design principles where leveraging inherent structure or known properties of interaction dynamics can enhance bot detection and resilience against adversarial manipulation. For CAPTCHA engineers, the actor-adjoint method exemplifies a technique to combine differentiable environment information with learning to reduce trial-and-error interaction costs, translating to more efficient and reliable challenge-response mechanisms. Although the paper focuses on physical dynamical control, the core idea of incorporating environment differentiability and adjoint sensitivity analyses could inspire bot-defense research that uses differentiable user-behavior models or interaction simulators to better optimize challenge designs or detect anomalous control strategies with fewer data.

Cite

@article{arxiv2607_16177,
  title={ Physics-enhanced reinforcement learning for real-time optimal control of dynamical systems },
  author={ Matteo Tomasetto and Nicolò Botteghi and Gabriele Bruni and Andrea Manzoni },
  journal={arXiv preprint arXiv:2607.16177},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.16177}
}

Read the full paper

• arXiv:2607.16177 (abstract)
• arXiv:2607.16177 (PDF)