Deep Reinforcement Learning-Enhanced Event-Triggered Data-Driven Predictive Control for a 3D Cable-Driven Soft Robotic Arm

Source: arXiv:2606.26048 · Published 2026-06-24 · By Cheng Ouyang, Moeen Ul Islam, Kaixiang Zhang, Zhaojian Li, Xiaobo Tan, Dong Chen

TL;DR

This paper addresses the computational challenges of deploying data-enabled predictive control (DeePC) on real-time, resource-constrained soft robotic platforms, where frequent online optimization is prohibitive. The authors propose RL-ET-DeePC, a framework that couples reinforcement learning (RL) with event-triggered DeePC to adaptively decide when to update the control optimization. By learning a state-dependent policy to trigger re-optimization only when necessary, the approach aims to significantly reduce computational load without degrading tracking performance. They validate the method in simulation on a complex nonlinear 3D cable-driven soft robotic arm model and on hardware experiments, demonstrating up to 66% reduction in optimizer calls in simulation and 34% reduction in hardware, while preserving comparable accuracy to periodic DeePC and outperforming static threshold event-triggered baselines in stability and smoothness.

The novel contribution lies in integrating singular value decomposition (SVD)-based dimension reduction of the DeePC data matrices with an RL agent that observes tracking error states and decides when to invoke receding-horizon optimization. This enables rigorous data-driven predictive control on complex soft robots otherwise limited by real-time computational constraints. Results confirm the learned event-triggering policy adapts update frequency dynamically with motion state, improves efficiency, and generalizes robustly to unseen trajectories and hardware noise. Thus, RL-ET-DeePC represents a significant step toward practical real-time control for soft robotic arms.

Key findings

• RL-ET-DeePC achieves up to 66% reduction in DeePC optimization calls compared to periodic triggering in simulation while maintaining tracking RMSE within 0.230-0.288 mm versus 0.144 mm baseline (Table I).
• Among RL algorithms tested (DQN, A2C, PPO), PPO provides the best trade-off, cutting total decision time by approximately 56% at ρ=0.5 while preserving stable closed-loop control.
• The learned RL event-triggering policy concentrates optimizer calls during transient tracking deviations and reduces frequency during steady-state phases (Fig. 3, Fig. 4).
• Zero-shot sim-to-real transfer on a 3D cable-driven soft robotic arm shows a 34% reduction in optimizer calls with comparable median tracking RMSE ≈2.75 mm (Fig. 6).
• Compared to a threshold-based event-triggering baseline with similar trigger rates, RL-ET-DeePC yields smoother, context-aware updates and improved trajectory fidelity (Fig. 7).
• Singular value decomposition (SVD) reduces input-output Hankel matrix dimensions from 𝑞1×𝑞2 to rank 𝑟=600, enabling tractable real-time optimization (Sec. III.A).
• Supervisory override logic ensures forced re-optimization at most every 𝑁−1 steps (25 steps horizon), guaranteeing safety and manageable open-loop intervals (Sec. III.D).
• Robustness demonstrated through tracking of unseen multi-step sharp-bend trajectories, showing generalizability beyond training reference motions (Fig. 4).

Methodology — deep read

1. Threat Model & Assumptions: The adversary context is not security-focused. The approach targets real-time control challenges in nonlinear, time-varying soft robotic systems with unknown dynamics. The RL agent has no explicit system model and learns triggering policies solely via measured tracking errors and environment interaction.

2. Data: Offline data collection uses a ‘‘ramp-and-hold’’ excitation signal to gather 1200 input-output samples from a high-fidelity nonlinear simulator of the 3D cable-driven soft robotic arm. This dataset constructs Hankel matrices with length N=25 and initial trajectory length T_ini=20. Data is preprocessed using singular value decomposition truncating to rank r=600 to reduce dimensionality.

3. Architecture & Algorithms:

• DeePC predictive controller applies Willems’s Fundamental Lemma to model local LTI approximations from input-output data.
• The regularized SVD-DeePC optimization problem at each step minimizes weighted quadratic tracking error and control effort plus regularization penalties on slack variables and decision vectors to account for noise and improve numerical stability.
• The RL agent (policy π_θ) is a neural network with a single 128-neuron hidden layer producing discrete binary actions {0,1} that decide if DeePC optimization is triggered at each timestep.
• State input to the RL agent is the current tracking error vector and its difference from previous timestep, compactly encoding tracking quality and short-term trends.
• Reward balances squared weighted tracking error penalty against computational cost for triggering, parameterized by penalty coefficient ρ.
• Supervisory override logic forces triggers if the control input buffer is empty or if the maximum open-loop interval (N−1 steps) is exceeded.

4. Training Regime: RL agents from DQN, PPO, and A2C families were trained for 50,000 time steps in episodes of 200 steps using the Gym-compatible soft robot simulator. PPO hyperparameters include actor learning rate 1e-3, critic 1e-2, discount factor γ=0.98, clipping 0.2, and GAE λ=0.95.

5. Evaluation Protocol: Evaluation compares RL-triggered DeePC with periodic DeePC and threshold-based events on both simulation and hardware. Metrics include optimization trigger rate (%), root-mean-square tracking error (RMSE in mm), and total decision time (s). Results are reported with mean ± std over 5 random seeds for simulation and 10 hardware trials. Cross-validation on unseen multi-step trajectories tests generalization. Hardware uses an OptiTrack motion capture system for accurate position measurement.

6. Reproducibility: Hardware designs, source code, and simulation environment are publicly available. The paper includes detailed hyperparameters and optimization formulations to facilitate replication.

Example: At a given time t, the agent observes the state vector (tracking error and error change), and decides a_t=1 to trigger optimization. The SVD-DeePC quadratic program (Eq. 7) is solved to generate an optimal control sequence u*_f over horizon N=25 steps, and u*_t is applied immediately. Subsequent steps without triggering reuse buffered controls until the next event triggers a new solution. This adaptive event-triggering balances computational effort with control accuracy in closed-loop.

Technical innovations

• Integration of SVD-based dimensionality reduction with DeePC to significantly reduce optimization size and enable tractable real-time predictive control for soft robots.
• Reformulation of event-triggered data-driven predictive control as a Markov Decision Process solved by deep reinforcement learning, enabling adaptive, state-dependent triggering rather than static heuristic thresholds.
• Design of a compact tracking-error-based state representation and reward function to guide RL agents in balancing tracking accuracy and computational load.
• Implementation of a supervisory override ensuring bounded open-loop intervals and robust fallback triggering within the learned event-triggering framework.
• Demonstration of zero-shot sim-to-real transfer of RL-based event-triggering policies from a high-fidelity simulator to physical 3D cable-driven soft robotic arms.

Datasets

• Cable-driven soft robotic arm simulation data — 1200 input-output samples — collected offline via ‘‘ramp-and-hold’’ input excitations

Baselines vs proposed

• Periodic DeePC: trigger rate = 100%, RMSE = 0.144 mm, decision time = 8.94 s
• RL-ET-DeePC (PPO) at ρ=0.5: trigger rate = 44.4%, RMSE = 0.230 mm, decision time = 3.95 s
• RL-ET-DeePC (PPO) at ρ=1.0: trigger rate = 33.9%, RMSE = 0.288 mm, decision time = 3.02 s
• RL-ET-DeePC (DQN) at ρ=0.5: trigger rate = 45.1%, RMSE = 0.242 mm, decision time = 4.01 s
• RL-ET-DeePC (A2C) at ρ=0.5: trigger rate = 88.1%, RMSE = 0.161 mm, decision time = 7.79 s
• Hardware periodic DeePC: trigger rate = 100%, median RMSE ≈ 2.75 mm
• Hardware RL-ET-DeePC (PPO) at ρ=5.0: trigger rate ≈ 66%, median RMSE ≈ 2.75 mm
• Threshold-based ET-DeePC baseline matches trigger rate (~66%) but has worse and less smooth tracking accuracy than RL-ET-DeePC on hardware

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.26048.

Fig 2: 3D cable-driven soft robotic arm: (a) casting molds

Fig 1: Illustration of the proposed RL-ET-DeePC frame-

Fig 3 (page 2).

Fig 4: shows the tracking performance and corresponding

Fig 8: Tracking performance on the physical arm. The black curve is the reference circular trajectory. Compared with periodic

Limitations

• Controller and RL training validated only on a single soft robotic arm design; generalization to other soft robot morphologies untested.
• The approach relies on an accurate initial offline data collection phase; performance with limited or noisy data not analyzed.
• No formal stability or robustness proofs provided for the RL-controlled event-triggered DeePC closed-loop system; supervisory logic is a heuristic safeguard only.
• Hardware experiments employed open-loop actuation without encoder feedback, which may limit achievable control precision compared to closed-loop sensing.
• Event-triggering penalty ρ requires tuning per hardware environment, limiting plug-and-play deployment.
• RL methods tested are off-the-shelf algorithms; potential exists for improved architectures or exploration strategies tuned specifically for this event-triggering task.
• No adversarial or distribution-shift robustness testing, e.g., under unexpected disturbances or sensor faults.

Open questions / follow-ons

• Can the RL-ET-DeePC framework be extended to multi-module soft robots or other high-DOF continuum manipulators with more complex dynamics?
• What are the theoretical stability guarantees for event-triggered DeePC under learned RL policies with supervisory overrides?
• How does the framework perform with limited offline data or non-stationary dynamics varying over long deployment?
• Can continuous action spaces or combined hybrid policies improve the triggering efficiency and control accuracy trade-off?

Why it matters for bot defense

While this paper is primarily focused on soft robotic control, the concepts of adaptive event-triggered optimization informed by reinforcement learning have potential analogs in bot defense, specifically for systems requiring real-time decision-making under heavy computational constraints. Bot-defense mechanisms that employ predictive models or optimization frameworks might benefit from RL-based adaptive triggers to selectively invoke costly computations only when state changes warrant an update, reducing resource usage. The use of dimension-reduction techniques like SVD to compress high-dimensional data for faster optimization could also translate to faster signal processing or feature extraction in CAPTCHA pipelines. Additionally, the supervisory override strategy provides a practical method to balance adaptive policies with hard safety bounds—a useful design principle when tuning bot detection thresholds or challenge issuance rates. However, applying this approach directly to CAPTCHA or bot detection systems would require significant adaptation, as the problem dynamics and adversarial threat models differ substantially from robot control scenarios.

Cite

@article{arxiv2606_26048,
  title={ Deep Reinforcement Learning-Enhanced Event-Triggered Data-Driven Predictive Control for a 3D Cable-Driven Soft Robotic Arm },
  author={ Cheng Ouyang and Moeen Ul Islam and Kaixiang Zhang and Zhaojian Li and Xiaobo Tan and Dong Chen },
  journal={arXiv preprint arXiv:2606.26048},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.26048}
}

Read the full paper

• arXiv:2606.26048 (abstract)
• arXiv:2606.26048 (PDF)