SciPhy Reinforcement Learning for Portfolio Optimization

Source: arXiv:2607.15195 · Published 2026-07-16 · By Igor Halperin, Andrey Itkin

TL;DR

This paper addresses the classical problem of dynamic portfolio optimization for large institutional investors by introducing a Scientific Physics-Informed Reinforcement Learning (SciPhyRL) approach formulated in continuous time. Unlike traditional discrete-time RL or classical control methods, SciPhyRL leverages offline historical data to directly solve the Hamilton-Jacobi-Bellman (HJB) equation projected onto observed trajectories via a pathwise Hamilton-Jacobi equation. This is achieved using Physics-Informed Neural Networks (PINNs) in an offline, single-pass fashion rather than iterative value or policy updates. A novel microstructure-grounded quadratic price impact model accounts for realistic transaction costs in the control problem. The control variable is recast from continuous trading rates to discrete target holdings, ensuring immediate position adjustments.

Evaluated on a 14-asset ETF universe using a synthetic oracle signal, the method achieves significant out-of-sample Sharpe ratio improvements compared to static and myopic baselines. The framework effectively translates signal quality into a robust, multi-period, cost-aware strategy that tightly controls volatility and turnover. This work extends prior offline distributional RL formulations by integrating risk-sensitive objectives, microstructure-aware costs, and a rich state space including cumulative costs, prices, and holdings. The semi-analytical solver with convergence guarantees complements the PINN-based approach, and results demonstrate practical applicability for large-scale portfolio allocation under realistic market impact costs.

Key findings

• The proposed SciPhyRL framework yields substantial out-of-sample Sharpe ratio improvements over static and myopic portfolio baselines on a 14-asset ETF universe with an engineered oracle signal.
• Recasting controls from continuous trading rates to discrete target holdings enables immediate position adjustments while accommodating a quadratic price impact model consistent with market microstructure.
• Including cumulative cost as a state variable extends the Markovian state space, allowing full distributional modeling of portfolio returns and costs under a risk-sensitive utility function.
• The governing HJB equation is projected onto observed state-action trajectories as a pathwise Hamilton-Jacobi PDE and solved via a Physics-Informed Neural Network trained offline without iterative policy/value updates.
• The multi-component price impact model incorporates temporary, permanent-convex, cross-impact, and memory terms with parameters tied to observable market quantities, reducing the parameter count from 13 to 5 for tractability.
• The running cost function is quadratic in the control variable, and the asset price drift likewise depends quadratically on controls, moving beyond the classical linear-quadratic regulator framework.
• A semi-analytical fixed-point iteration solver based on operator splitting and Fast Gauss Transform is developed with established convergence criteria, serving as a complement to the PINN solver.
• Soft budget constraints implemented via a quadratic penalty preserve the Gaussian mixture structure of the optimal stochastic policy and allow flexible cash flows while enforcing near self-financing.

Threat model

The work does not explicitly model an adversary attempting to subvert or attack the system. Instead, the 'threats' arise from realistic market frictions, price impact from large trades, and the need to operate under limited offline data without environment interaction. The adversary is the challenging and partially observable financial environment with liquidity constraints and transaction costs rather than a malicious actor.

Methodology — deep read

1. Threat Model & Assumptions: The framework targets large institutional investors whose sizable trades impact market prices. The adversary is implicitly the market microstructure and liquidity constraints, modeled via price impact functions. No explicit adversarial attacks are considered; rather, the model aims to optimize trading under realistic cost dynamics. The policy is learned offline from behavioral data without environment interaction.

2. Data: The dataset consists of historical price trajectories and engineered oracle predictive signals on a 14-asset ETF universe. The data includes sequences of states: holdings, prices, and cumulative costs. Specific size and split details are not fully disclosed but the dataset is offline and fixed, representing a batch RL setting.

3. Model Architecture and Algorithm: The portfolio dynamics are modeled via stochastic differential equations with drift terms incorporating predictive signals and microstructure-based quadratic price impact effects. The extended state space includes holdings, prices, and realized cumulative cost, rendering the system Markovian.

The optimization problem is framed as a continuous-time distributional reinforcement learning task: to find a stochastic policy minimizing a risk-sensitive expected utility of cumulative cost, regularized by KL divergence from a baseline policy. This corresponds to solving a nonlinear Hamilton-Jacobi-Bellman equation reformulated as a pathwise Hamilton-Jacobi PDE projected onto observed trajectories.

The PDE solver is a Physics-Informed Neural Network (PINN) that encodes the PDE residual as a loss using automatic differentiation; training is performed offline in a single sweep over the fixed dataset. An alternative semi-analytical solver employing operator splitting and iterative fixed-point updates with a Fast Gauss Transform is also developed, accompanied by theoretical convergence guarantees.

Control variables are discretized to represent target holdings rather than continuous trading rates, enabling immediate position adjustments and more stable numerical treatment. A Gaussian mixture approximation of the optimal Gibbs policy arises naturally from the quadratic cost structure.

4. Training Regime: The PINN is trained offline using the fixed dataset of price and state trajectories, minimizing PDE residuals plus constraints. Details such as number of epochs, batch size, or hyperparameters are not fully specified. Numerical validation is performed on discretized model simulations approximating the continuous-time SDEs.

5. Evaluation Protocol: Performance is evaluated out-of-sample using Sharpe ratio improvements relative to static and myopic baseline portfolios on the ETF universe with synthetic oracle signals. Additional metrics include volatility and turnover control. Ablations investigate the impact of risk-sensitivity parameterization, budget penalties, and policy parameterization. Statistical details on significance tests or cross-validation are not detailed.

6. Reproducibility: While the codebase is not explicitly referenced as released, the paper provides comprehensive mathematical formulations, algorithmic details, and parameter settings, allowing reproduction given access to similar data. The dataset is proprietary or simulated, limiting direct replication.

Concrete Example Workflow: Given historical trajectories (St, xt) and cumulative costs Ct from an institutional portfolio, the PINN is trained offline to minimize deviations from the pathwise HJB equation by encoding its residual on collocation points. The resulting neural network approximates the value function and optimal stochastic policy as a Gibbs distribution over discrete target holdings. This policy is then deployed in simulation or live trading to generate trades achieving risk-sensitive, cost-aware allocations that improve Sharpe ratios over naive strategies.

Technical innovations

• Projection of the continuous-time HJB equation onto observed data trajectories as a pathwise Hamilton-Jacobi PDE solved directly via Physics-Informed Neural Networks without value or policy iteration.
• Recasting the portfolio control from continuous trading rates to discrete target holdings, enabling immediate enforcement of desired positions and robust handling of microstructure-aware costs.
• Development of a multi-component quadratic price impact model grounded in observable market microstructure variables, reducing complexity from 13 to 5 parameters.
• Introduction of a semi-analytical operator-splitting fixed-point iterative solver for the nonlinear PDE with proven convergence, complementing the PINN approach.
• Incorporation of cumulative realized trading costs into the extended Markovian state space, enabling distributional RL formulations over risk-sensitive cumulative cost utilities.

Datasets

• 14-asset ETF universe — size unspecified — private or simulated dataset with engineered oracle predictive signals

Baselines vs proposed

• Static portfolio baseline: out-of-sample Sharpe ratio reported significantly lower than SciPhyRL-learned policy
• Myopic baseline portfolio: Sharpe ratio lower than Gibbs policy obtained by SciPhyRL, demonstrating advantage of multi-period, cost-aware allocation
• Ablation on risk sensitivity parameter: varying quadratic utility target cost demonstrates controlled tradeoff between upside capping and risk

Limitations

• The dataset used is proprietary or simulated with engineered oracle signals limiting direct external replication and real-world verification.
• The PINN solver training details (epochs, batch sizes, hyperparameters) are not fully specified, which may affect reproducibility.
• Evaluation focuses on synthetic signals and 14 ETFs; application to noisy real signals or broader asset classes remains untested.
• No explicit adversarial robustness analysis or stress-testing under extreme market conditions provided.
• The semi-analytical solver's numerical comparison with PINNs is deferred to future work, leaving questions on relative performance open.
• The approach assumes constant volatility and other idealized model parameters that may limit fidelity in real financial markets.

Open questions / follow-ons

• How does the SciPhyRL approach perform with real, noisy predictive signals rather than an engineered oracle?
• Can the semi-analytical PDE solver outperform or complement the PINN method in large dimensions in terms of efficiency and accuracy?
• How robust is the framework under distributional shifts such as regime changes or crisis market conditions?
• How to extend the method to incorporate dynamic risk preferences or asymmetric utility functions that better capture realistic investor goals?

Why it matters for bot defense

Although this research is primarily focused on financial portfolio optimization, it illustrates a powerful example of applying scientific machine learning—specifically Physics-Informed Neural Networks—to solve high-dimensional, nonlinear control problems offline from fixed datasets. From a bot-defense and CAPTCHA perspective, the core methodology of projecting complex PDE constraints onto observed trajectories and solving them offline with PINNs could inspire analogous approaches for modeling bot behavior dynamics or optimizing defense policies without online interaction. The distributional RL formulation and risk-sensitive control elements also provide concepts transferable to designing more robust, uncertainty-aware bot defense mechanisms. Finally, the semi-analytical operators and fixed-point convergence analysis represent rigorous tools for ensuring policy stability in complex environments, which is valuable for AI-driven security systems.

Cite

@article{arxiv2607_15195,
  title={ SciPhy Reinforcement Learning for Portfolio Optimization },
  author={ Igor Halperin and Andrey Itkin },
  journal={arXiv preprint arXiv:2607.15195},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.15195}
}

Read the full paper

• arXiv:2607.15195 (abstract)
• arXiv:2607.15195 (PDF)