Explainable Reinforcement Learning via Physics-Aware Policy Distillation

Source: arXiv:2607.24672 · Published 2026-07-27 · By Shaker Al-Tamari, Waled Kadour

TL;DR

This paper addresses the critical challenge of explainability in Deep Reinforcement Learning (DRL) for continuous control tasks, which hinders deployment in safety-critical domains such as robotics and automotive systems. The authors propose a physics-aware policy distillation framework that transforms an opaque, high-performance Twin Delayed Deep Deterministic Policy Gradient (TD3) agent controlling an Inverted Pendulum into an interpretable surrogate model—a shallow decision tree. Key innovations include generating a diverse dataset via "Noisy Oracle Rollouts" that expose rare recovery behaviors and engineering a physics-aware feature called Pole Urgency to capture nonlinear system dynamics effectively in the surrogate.

The distilled decision tree matches the teacher's success rate perfectly on the 1000-step balancing benchmark while providing global and local interpretability through IF-THEN rules and decision path visualization. However, the discrete nature of the surrogate introduces Bang-Bang control effects, producing a stable bimodal limit cycle and high-frequency chattering. The authors rigorously analyze control-theoretic stability properties showing that Bounded Input Bounded Output (BIBO) stability holds despite these artifacts, providing a principled framework for certifiable interpretable control policies in real-world safety contexts.

Key findings

• The TD3 teacher agent achieved a consistent 1000 reward indicating perfect balancing on the InvertedPendulum-v4 benchmark.
• The distilled decision tree with max depth 7 achieved a 100% closed-loop success rate over 50 consecutive 1000-step episodes, matching the expert performance.
• Behavioral fidelity analysis shows strong monotonic correlation with the TD3 oracle despite control quantization into discrete leaf nodes.
• The surrogate’s discrete control induces a stable bimodal limit cycle with pole angle oscillating safely between approximately ±0.5 radians.
• BIBO stability is empirically demonstrated as the maximum pole angle deviation remains strictly within environment failure thresholds (±0.209 rad).
• The surrogate exhibits high-frequency Bang-Bang chattering in the control output, versus smooth continuous torque from the TD3 teacher.
• The physics-aware Pole Urgency feature enables the shallow tree to learn diagonal decision boundaries approximating nonlinear control logic effectively.

Threat model

N/A — the paper does not focus on security adversaries but rather on interpretability and control stability of DRL policies through policy distillation.

Methodology — deep read

1. Threat Model & Assumptions: The study assumes an environment governed by a Markov Decision Process (MDP) where the adversary is not explicitly modeled since the focus is on interpretability and stability of policy distillation rather than adversarial robustness. The environment is the classic InvertedPendulum-v4 from the Gymnasium library, widely used for continuous control benchmarks.

2. Data: A dataset was collected by running the pre-trained TD3 teacher policy in the environment with added Gaussian noise on actions during rollouts ("Noisy Oracle Rollouts"). This forced the system into rare and near-failure states to capture recovery behavior typically not present in noise-free demonstrations. The dataset mainly emphasizes initial transient states critical for stable control.

3. Architecture/Algorithm: The teacher agent is a TD3 actor-critic deep neural network with two hidden layers of 256 ReLU units each; the output layer uses Tanh appropriately scaled to environment torque range [-3, 3]. The student surrogate is a Decision Tree Regressor (using scikit-learn) with maximum depth 7 to enforce interpretability. A novel physics-aware feature named Pole Urgency (Purgency = θ + Kd * 𝜃̇) linearizes the coupled state-space into a single feature that allows diagonal splitting by the tree, overcoming the axis-aligned splitting limitation.

4. Training Regime: The TD3 agent was trained with a learning rate of 3e-4, batch size 256, and Gaussian noise exploration (σ=0.1), implemented in CleanRL framework to ensure stable convergence. The decision tree was trained on the harvested noisy rollout data via supervised regression, balancing depth for accuracy and human readability.

5. Evaluation Protocol: Multiple metrics were used for a holistic assessment: Closed-Loop Success Rate measuring percentage of episodes where the surrogate balances the pendulum for 1000 steps; Average Episodic Return averaged over 50 episodes; Behavioral Fidelity quantified by Mean Squared Error (MSE) and correlation between teacher’s continuous torques and student’s discrete outputs; Control Stability assessed by measuring steady-state angle bounds (maximum deviation), visualizing limit cycles and quantifying Bang-Bang chattering frequency.

6. Reproducibility: The implementation uses publicly available OpenAI Gym environment (InvertedPendulum-v4), CleanRL framework for TD3, and standard scikit-learn decision tree implementations in Python. The paper notes the codebase but does not explicitly confirm public release or frozen weights.

Concrete example: The TD3 teacher policy is run with injected noise to sample diverse state-action pairs covering recovery behaviors. This dataset trains a Decision Tree that uses the engineered Pole Urgency feature to optimally partition state space into discrete torque commands. Evaluating the tree in closed-loop shows the pole oscillates in a safe limit cycle while achieving perfect episode success (1000 steps balancing), despite inducing rapid discrete action switching (Bang-Bang control). Decision path visualizations provide live interpretability for the agent’s control logic in each state.

Technical innovations

• Integration of a physics-aware feature (Pole Urgency) encoding nonlinear coupling of pole angle and angular velocity enabling shallow decision trees to approximate nonlinear continuous control policies.
• Usage of Noisy Oracle Rollouts with injected Gaussian noise during dataset collection to expose rare failure and recovery behaviors essential for robust surrogate learning.
• Demonstration and rigorous evaluation of control-theoretic stability properties (BIBO stability and bimodal limit cycles) in discrete surrogates derived from continuous DRL policies.
• A framework combining policy distillation into decision trees with control theory metrics that quantify the tradeoffs between interpretability and actuation smoothness for safe autonomous systems.

Datasets

• Noisy Oracle Rollouts dataset — size not explicitly specified — generated by applying Gaussian noise to TD3 oracle policy actions during rollouts in the InvertedPendulum-v4 environment.

Baselines vs proposed

• TD3 Teacher agent: average episodic return = 1000.0 vs Distilled Decision Tree: closed-loop success rate = 100% (50 episodes, 1000 steps each)
• Behavioral fidelity (action correlation): strong monotonic correlation with quantization into discrete leaf nodes (exact MSE values not specified)
• Control effort (chattering frequency): TD3 Oracle shows smooth continuous torque vs Decision Tree exhibits high-frequency Bang-Bang switching

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.24672.

Fig 1: The InvertedPendulum-v4 environment used for testing control

Fig 2: Simplified top-level structure of the distilled Decision Tree.

Fig 3: Behavioral Fidelity. Action agreement between the continuous TD3

Fig 4: Steady-State Limit Cycle. The bimodal distribution confirms the pole

Fig 5: Control Effort over Time. A comparison of the fine-grained, continuous

Fig 6: 2D Decision Boundary Map. This visualizes the global policy

Fig 7: Live Decision Path Visualization. This real-time diagnostic dashboard

Limitations

• The inherent Bang-Bang control nature of decision tree surrogates induces high-frequency switching which could cause severe mechanical wear in physical hardware.
• Experiments are conducted only on a low-dimensional control toy environment (Inverted Pendulum), limiting direct generalization to higher-dimensional or more complex real-world tasks.
• No adversarial evaluation or robustness testing beyond Gaussian noise injection was reported.
• Details on dataset size and exact hyperparameter sensitivity for the surrogate training are sparse, limiting reproducibility assessment.
• The approach currently sacrifices the smoothness of continuous control for interpretability, suggesting a tradeoff that may be problematic in certain applications.

Open questions / follow-ons

• How can the Bang-Bang chattering of discrete surrogates be mitigated, e.g., through smoothing or hybrid continuous-discrete control strategies?
• Can the physics-aware distillation approach scale effectively to higher-dimensional continuous control tasks such as multi-joint robotic locomotion?
• What are the implications of incorporating interpretable surrogate models as safety layers in complex autonomous driving or robotics systems involving perception components?
• How robust are the distilled policies under distribution shifts, unmodeled disturbances, or adversarial input perturbations?

Why it matters for bot defense

This study provides valuable insights for bot-defense practitioners interested in designing interpretable, certifiable decision policies from complex black-box models such as neural networks. While it focuses on continuous control rather than discrete CAPTCHA challenges, the core techniques of policy distillation into transparent rule-based models and stability analysis exemplify concrete strategies to convert opaque systems into auditable, trustworthy surrogates. The physics-aware feature engineering and noisy rollout data collection highlight practical means to capture critical failure modes for robust policy replication. Additionally, the explicit trade-off characterization between interpretability and discrete control artifacts such as high-frequency switching could inspire similar caution in simplifying bot defenses or challenge-response models, where abrupt decision boundaries might induce unintended vulnerabilities or oscillatory behavior. Overall, the work advocates principled approaches to blend explainability with performance guarantees that bot-defense engineers can adapt to validate and audit complex AI-driven decision systems.

Cite

@article{arxiv2607_24672,
  title={ Explainable Reinforcement Learning via Physics-Aware Policy Distillation },
  author={ Shaker Al-Tamari and Waled Kadour },
  journal={arXiv preprint arXiv:2607.24672},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.24672}
}

Read the full paper

• arXiv:2607.24672 (abstract)
• arXiv:2607.24672 (PDF)